support some container formats for video

[alas]

It would be nice to be able to use a container format, such as webm, for h264 encodings (specifically) for use in processing in html5 video elements. (The native video processing in the browsers don't process naked h264 packets very well.)

[alas]

As a quick update... would it be possible to use webm & mp4 also, in the event that h264 doesn't work with webm?

[Antoine Martin]

The two main contenders that I can see:

• ​matroska.org
• ​libmkv

both available on Fedora and Debian, but centos doesn't have either of them...

libmkv is meant to be more portable (plain C instead of C++) - its header files look more sane, but its API is file based and we only want in-memory streaming.. converting that to buffer access might be doable, if a bit risky.

webm could be used for vp8/vp9 only.. and is not supported by IE or Safari.

[Antoine Martin]

makes it possible to save the raw video stream to a file on the server

[Antoine Martin]

mpeg4 is the only container supported by all browsers: ​http://www.w3schools.com/html/html5_video.asp. ​Removal of Ogg Vorbis and Theora from HTML5: an outrageous disaster - monopolies lying through their teeth, and we let them get away with it, again.

Unsurprisingly, the only open-source project that does what we need is ffmpeg: ​MOV/MP4/ISMV (Smooth Streaming) muxer: A fragmented file consists of a number of fragments, where packets and metadata about these packets are stored together.
Downside is that we don't use ffmpeg for h264, which will make it harder to use the container code... or we have to make a new encoder using ffmpeg. This would need new ffmpeg builds with support for those muxers (that part is no big deal).

So, I am tempted to start with ​ogg or ​webm because that should be easier.

[Antoine Martin]

Webm: so many copies of similar file-oriented code everywhere:

• ​https://chromium.googlesource.com/webm/libvpx/+/master/webmenc.h
• ​http://www.webmproject.org/docs/webm-sdk/example_vpxenc.html
• ​https://github.com/webmproject/libwebm/blob/master/mkvwriter.cpp

ffmpeg links for muxing examples:

• ​http://ffmpeg.org/doxygen/trunk/doc_2examples_2decoding_encoding_8c-example.html
• ​http://ffmpeg.org/doxygen/trunk/doc_2examples_2muxing_8c-example.html
• ​http://stackoverflow.com/questions/11537830/ffmpeg-api-h264-encoded-video-does-not-play-on-all-platforms
• ​http://stackoverflow.com/questions/17118340/remuxing-mp4-on-the-fly-with-ffmpeg-api
• ​http://stackoverflow.com/questions/14052457/how-can-i-mux-h264-stream-into-mp4-file-via-libavformat

• VLC links:
• ​https://fossies.org/linux/vlc/modules/mux/mp4.c
• ​https://fossies.org/dox/vlc-2.2.2/mux_2ogg_8c_source.html

According to the ​matroska faq, ogg is better suited for streaming and lossy transports (ie: UDP).

More information on the MP4 container format:

• ​http://atomicparsley.sourceforge.net/mpeg-4files.html
• ​http://fabiensanglard.net/mobile_progressive_playback/index.php
• ​http://stackoverflow.com/questions/10097645/how-to-generate-moov-before-mdat-mp4-video-files-with-high-performance-minima

[Antoine Martin]

Using the save-to-file feature merged in r12144, then adding the mpeg4 container using ffmpeg:

ffmpeg -i in.h264 -vcodec copy -f mp4 -movflags faststart out.mp4

I can see that the mpeg4 header is about 0x660 bytes long, and is followed by the raw h264 frames as "mdat".
It looks like what we want is ffmpeg's frag_custom mode: ​mp4 options: Allow the caller to manually choose when to cut fragments, by calling av_write_frame(ctx, NULL) to write a fragment with the packets written so far so that we can fragment on each frame.

[Antoine Martin]

stubs for 3 new video container plugins

[Antoine Martin]

Interesting gstreamer discussion: ​https://lists.freedesktop.org/archives/gstreamer-devel/2016-April/057989.html, quote: mp4 is not a streamable format in the sense of being able to be played as it is generated.

And: ​HTTP Adaptive Streaming with GStreamer

[Antoine Martin]

The patch above gives us ​AVFrame and ​AVPacket from our imagewrapper class.
Maybe this will be added even without the muxer.

The next step is the muxer, using ​libavformat ​muxing.
Looks like we need ​AVIOContext and maybe we need to buffer it in memory to support seeking: None of the function pointers in AVIOContext should be called directly, they should only be set by the client application when implementing custom I/O. Normally these are set to the function pointers specified in avio_alloc_context()

Calls to ​avformat_alloc_output_context2,​avformat_write_header and ​av_write_frame should give us the stream, as long as we manage to configure all the magic flags required for fragmented mpeg4 streams..

[Antoine Martin]

We may want to tie this with colour profiles (#1155) since the profile can be specified in the stream and this may be used by the hardware decoders - even if browsers like chromium do not support icc profiles (​Support color management transform of ICC profile tagged images to device profile on Linux)

[Antoine Martin]

updated patch for b-frames (#800) and ffmpeg 3.1 (#1242)

[Antoine Martin]

r12955 adds the enc_ffmpeg codec (see commit message for some important limitations). It does support b-frames (#800).

​New AVCodec API was very useful here, ​Bitstream Filtering should help with the muxing part.

This is pretty close to what we want: ​FFMPEG I/O output buffer

[Antoine Martin]

generates mpeg4 container with h264 embedded... to file for now

[Antoine Martin]

muxing work if you use pointers as unsigned long, gets mangled otherwise..

[Antoine Martin]

Milestone renamed

[Antoine Martin]

updated patch against r13044

[Antoine Martin]

minor updates

[Antoine Martin]

merged in r14427 (big), still needs work in the html5 client as the video is lagging in Firefox and not playing at all in chrome..

[Antoine Martin]

Works well enough but can only be tested with the html5 client...
The streams can be saved to file with XPRA_SAVE_TO_FILE=basename.
The resulting files can be inspected with ​https://www.bento4.com/'s mp4dump or ffmpeg's ffprobe and looks correct in both cases. It can be played with any decent media player too.

Closing as this will get tested as part of #1341.

[Antoine Martin]

PS:

• win32 needed a build fix: r14442
• py3k fix: r14489

[Antoine Martin]

Related links:

• ​How to use ffmpeg faststart flag programmatically?
• ​Preserving 'udta' atom when using -movflags +faststart
• ​movflags/faststart flag for avcodec_open2

[Antoine Martin]

Got this reply about movflags / faststart:
I have succeeded in moving the moov atom.

Firstly, I set the movflags option when writing the header:

AVDictionary *options = NULL;
av_dict_set(&options, "movflags", "faststart", 0);
CHECK(avformat_write_header(oc, &options), "Error occurred when writing media header");

Secondly, because I have a custom AVIOContext on my format context, I needed to overwrite the io_open and io_close callbacks on my format context so we don't try to open a non-existent file when writing the trailer.

AVIOContext* ioContext = avio_alloc_context(buffer, BUFFER_SIZE, 1, streamPtr, &Read, &Write, &Seek);
formatContext->pb = ioContext;
formatContext->flags = AVFMT_FLAG_CUSTOM_IO;
formatContext->io_open = &Open;
formatContext->io_close = &Close;

My last step was disabling buffering on the AVIOContent I created in the Open callback. This was necessary due to my wrapping the buffer in my original AVIOcontext. The buffer would be invalid. Note that the below code is c++/cli (.net mixed mode).

static int Open(struct AVFormatContext *s, AVIOContext **pb, const char *url, int flags, AVDictionary **options)
{
    GCHandle streamHandle = GCHandle::FromIntPtr(IntPtr(s->pb->opaque));
    Stream ^stream = (Stream ^)streamHandle.Target;
    StreamWrapper ^wrapper = gcnew StreamWrapper(stream);

    //TODO: check allocations (may be out of memory)
    unsigned char * buffer = (unsigned char *)av_malloc( BUFFER_SIZE );
    streamHandle = GCHandle::Alloc(wrapper);
    void* streamPtr = GCHandle::ToIntPtr(streamHandle).ToPointer();

    *(pb) = avio_alloc_context(buffer, BUFFER_SIZE , 1, streamPtr, &Read, &Write, &Seek);
    (*pb)->direct = 1;
    return 0;
}

static void Close(struct AVFormatContext *s, AVIOContext *pb)
{
    GCHandle streamHandle = GCHandle::FromIntPtr(IntPtr(pb->opaque));
    streamHandle.Free();
    av_free(pb->buffer);
}

If you take a look movenc.c where the mov data is being placed, you'll see where it tries to reopen the file:

    /* Shift the data: the AVIO context of the output can only be used for
     * writing, so we re-open the same output, but for reading. It also avoids
     * a read/seek/write/seek back and forth. */
    avio_flush(s->pb);
    ret = s->io_open(s, &read_pb, s->filename, AVIO_FLAG_READ, NULL);

It so happens that my AVIOContext is read/write (backed by a .net stream). My StreamWrapper class simply handles the

read/seek/write/seek that this code tries to avoid. I went the wrote of a proxy because I did *not* want to load another stream into memory.

Implementing this buffer seeking has now been moved to #1869.

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/1107