Xpra: Ticket #1107: support some container formats for video

It would be nice to be able to use a container format, such as webm, for h264 encodings (specifically) for use in processing in html5 video elements. (The native video processing in the browsers don't process naked h264 packets very well.)

Mon, 01 Feb 2016 18:34:18 GMT - alas:

As a quick update... would it be possible to use webm & mp4 also, in the event that h264 doesn't work with webm?

Sat, 06 Feb 2016 07:08:04 GMT - Antoine Martin: owner, status, summary changed

The two main contenders that I can see:

both available on Fedora and Debian, but centos doesn't have either of them...

libmkv is meant to be more portable (plain C instead of C++) - its header files look more sane, but its API is file based and we only want in-memory streaming.. converting that to buffer access might be doable, if a bit risky.

webm could be used for vp8/vp9 only.. and is not supported by IE or Safari.

Sat, 12 Mar 2016 14:44:25 GMT - Antoine Martin: attachment set

makes it possible to save the raw video stream to a file on the server

Sat, 12 Mar 2016 15:23:16 GMT - Antoine Martin:

mpeg4 is the only container supported by all browsers: Removal of Ogg Vorbis and Theora from HTML5: an outrageous disaster - monopolies lying through their teeth, and we let them get away with it, again.

Unsurprisingly, the only open-source project that does what we need is ffmpeg: MOV/MP4/ISMV (Smooth Streaming) muxer: A fragmented file consists of a number of fragments, where packets and metadata about these packets are stored together. Downside is that we don't use ffmpeg for h264, which will make it harder to use the container code... or we have to make a new encoder using ffmpeg. This would need new ffmpeg builds with support for those muxers (that part is no big deal).

So, I am tempted to start with webm because that should be easier.

Tue, 15 Mar 2016 10:41:58 GMT - Antoine Martin:

Webm: so many copies of similar file-oriented code everywhere:

ffmpeg links for muxing examples:

According to the matroska faq, ogg is better suited for streaming and lossy transports (ie: UDP).

More information on the MP4 container format:

Tue, 15 Mar 2016 14:03:12 GMT - Antoine Martin:

Using the save-to-file feature merged in r12144, then adding the mpeg4 container using ffmpeg:

ffmpeg -i in.h264 -vcodec copy -f mp4 -movflags faststart out.mp4

I can see that the mpeg4 header is about 0x660 bytes long, and is followed by the raw h264 frames as "mdat". It looks like what we want is ffmpeg's frag_custom mode: mp4 options: Allow the caller to manually choose when to cut fragments, by calling av_write_frame(ctx, NULL) to write a fragment with the packets written so far so that we can fragment on each frame.

Wed, 16 Mar 2016 05:26:22 GMT - Antoine Martin: milestone changed

Fri, 18 Mar 2016 15:23:02 GMT - Antoine Martin: attachment set

stubs for 3 new video container plugins

Wed, 27 Apr 2016 02:15:03 GMT - Antoine Martin:

Interesting gstreamer discussion: https://lists.freedesktop.org/archives/gstreamer-devel/2016-April/057989.html, quote: mp4 is not a streamable format in the sense of being able to be played as it is generated.

And: HTTP Adaptive Streaming with GStreamer

Sun, 12 Jun 2016 17:06:56 GMT - Antoine Martin:

The patch above gives us AVPacket from our imagewrapper class. Maybe this will be added even without the muxer.

The next step is the muxer, using muxing. Looks like we need AVIOContext and maybe we need to buffer it in memory to support seeking: None of the function pointers in AVIOContext should be called directly, they should only be set by the client application when implementing custom I/O. Normally these are set to the function pointers specified in avio_alloc_context()

Calls to av_write_frame should give us the stream, as long as we manage to configure all the magic flags required for fragmented mpeg4 streams..

Sun, 03 Jul 2016 10:23:55 GMT - Antoine Martin:

We may want to tie this with colour profiles (#1155) since the profile can be specified in the stream and this may be used by the hardware decoders - even if browsers like chromium do not support icc profiles (Support color management transform of ICC profile tagged images to device profile on Linux)

Sun, 03 Jul 2016 15:34:57 GMT - Antoine Martin: attachment set

updated patch for b-frames (#800) and ffmpeg 3.1 (#1242)

Sun, 03 Jul 2016 17:04:52 GMT - Antoine Martin:

r12955 adds the enc_ffmpeg codec (see commit message for some important limitations). It does support b-frames (#800).

Bitstream Filtering should help with the muxing part.

This is pretty close to what we want: FFMPEG I/O output buffer

Tue, 05 Jul 2016 08:47:09 GMT - Antoine Martin: attachment set

generates mpeg4 container with h264 embedded... to file for now

Wed, 06 Jul 2016 12:06:41 GMT - Antoine Martin: attachment set

muxing work if you use pointers as unsigned long, gets mangled otherwise..

Tue, 12 Jul 2016 16:52:22 GMT - Antoine Martin: milestone changed

Milestone renamed

Tue, 19 Jul 2016 19:44:02 GMT - Antoine Martin: attachment set

updated patch against r13044

Mon, 07 Nov 2016 13:30:32 GMT - Antoine Martin: attachment set

minor updates

Tue, 15 Nov 2016 15:36:43 GMT - Antoine Martin:

merged in r14427 (big), still needs work in the html5 client as the video is lagging in Firefox and not playing at all in chrome..

Thu, 17 Nov 2016 03:22:59 GMT - Antoine Martin: status changed; resolution set

Works well enough but can only be tested with the html5 client... The streams can be saved to file with XPRA_SAVE_TO_FILE=basename. The resulting files can be inspected with https://www.bento4.com/'s mp4dump or ffmpeg's ffprobe and looks correct in both cases. It can be played with any decent media player too.

Closing as this will get tested as part of #1341.

Fri, 18 Nov 2016 06:33:07 GMT - Antoine Martin:


Fri, 04 May 2018 05:21:15 GMT - Antoine Martin:

Related links:

Sat, 09 Jun 2018 11:39:37 GMT - Antoine Martin:

Got this reply about movflags / faststart: I have succeeded in moving the moov atom.

Firstly, I set the movflags option when writing the header:

AVDictionary *options = NULL;
av_dict_set(&options, "movflags", "faststart", 0);
CHECK(avformat_write_header(oc, &options), "Error occurred when writing media header");

Secondly, because I have a custom AVIOContext on my format context, I needed to overwrite the io_open and io_close callbacks on my format context so we don't try to open a non-existent file when writing the trailer.

AVIOContext* ioContext = avio_alloc_context(buffer, BUFFER_SIZE, 1, streamPtr, &Read, &Write, &Seek);
formatContext->pb = ioContext;
formatContext->flags = AVFMT_FLAG_CUSTOM_IO;
formatContext->io_open = &Open;
formatContext->io_close = &Close;

My last step was disabling buffering on the AVIOContent I created in the Open callback. This was necessary due to my wrapping the buffer in my original AVIOcontext. The buffer would be invalid. Note that the below code is c++/cli (.net mixed mode).

static int Open(struct AVFormatContext *s, AVIOContext **pb, const char *url, int flags, AVDictionary **options)
    GCHandle streamHandle = GCHandle::FromIntPtr(IntPtr(s->pb->opaque));
    Stream ^stream = (Stream ^)streamHandle.Target;
    StreamWrapper ^wrapper = gcnew StreamWrapper(stream);
    //TODO: check allocations (may be out of memory)
    unsigned char * buffer = (unsigned char *)av_malloc( BUFFER_SIZE );
    streamHandle = GCHandle::Alloc(wrapper);
    void* streamPtr = GCHandle::ToIntPtr(streamHandle).ToPointer();
    *(pb) = avio_alloc_context(buffer, BUFFER_SIZE , 1, streamPtr, &Read, &Write, &Seek);
    (*pb)->direct = 1;
    return 0;
static void Close(struct AVFormatContext *s, AVIOContext *pb)
    GCHandle streamHandle = GCHandle::FromIntPtr(IntPtr(pb->opaque));

If you take a look movenc.c where the mov data is being placed, you'll see where it tries to reopen the file:

    /* Shift the data: the AVIO context of the output can only be used for
     * writing, so we re-open the same output, but for reading. It also avoids
     * a read/seek/write/seek back and forth. */
    ret = s->io_open(s, &read_pb, s->filename, AVIO_FLAG_READ, NULL);

It so happens that my AVIOContext is read/write (backed by a .net stream). My StreamWrapper class simply handles the read/seek/write/seek that this code tries to avoid. I went the wrote of a proxy because I did *not* want to load another stream into memory.

Implementing this buffer seeking has now been moved to #1869.

Sat, 23 Jan 2021 05:15:10 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1107