It would be nice to be able to use a container format, such as webm, for h264 encodings (specifically) for use in processing in html5 video elements. (The native video processing in the browsers don't process naked h264 packets very well.)
As a quick update... would it be possible to use webm & mp4 also, in the event that h264 doesn't work with webm?
The two main contenders that I can see:
both available on Fedora and Debian, but centos doesn't have either of them...
libmkv is meant to be more portable (plain C instead of C++) - its header files look more sane, but its API is file based and we only want in-memory streaming.. converting that to buffer access might be doable, if a bit risky.
webm could be used for vp8/vp9 only.. and is not supported by IE or Safari.
makes it possible to save the raw video stream to a file on the server
mpeg4 is the only container supported by all browsers: Removal of Ogg Vorbis and Theora from HTML5: an outrageous disaster - monopolies lying through their teeth, and we let them get away with it, again.
Unsurprisingly, the only open-source project that does what we need is ffmpeg: MOV/MP4/ISMV (Smooth Streaming) muxer: A fragmented file consists of a number of fragments, where packets and metadata about these packets are stored together. Downside is that we don't use ffmpeg for h264, which will make it harder to use the container code... or we have to make a new encoder using ffmpeg. This would need new ffmpeg builds with support for those muxers (that part is no big deal).
So, I am tempted to start with webm because that should be easier.
Webm: so many copies of similar file-oriented code everywhere:
ffmpeg links for muxing examples:
According to the matroska faq, ogg is better suited for streaming and lossy transports (ie: UDP).
More information on the MP4 container format:
Using the save-to-file feature merged in r12144, then adding the mpeg4 container using ffmpeg:
ffmpeg -i in.h264 -vcodec copy -f mp4 -movflags faststart out.mp4
I can see that the mpeg4 header is about 0x660 bytes long, and is followed by the raw h264 frames as "mdat". It looks like what we want is ffmpeg's frag_custom mode: mp4 options: Allow the caller to manually choose when to cut fragments, by calling av_write_frame(ctx, NULL) to write a fragment with the packets written so far so that we can fragment on each frame.
stubs for 3 new video container plugins
Interesting gstreamer discussion: https://lists.freedesktop.org/archives/gstreamer-devel/2016-April/057989.html, quote: mp4 is not a streamable format in the sense of being able to be played as it is generated.
And: HTTP Adaptive Streaming with GStreamer
The patch above gives us AVPacket from our imagewrapper class. Maybe this will be added even without the muxer.
The next step is the muxer, using muxing. Looks like we need AVIOContext and maybe we need to buffer it in memory to support seeking: None of the function pointers in AVIOContext should be called directly, they should only be set by the client application when implementing custom I/O. Normally these are set to the function pointers specified in avio_alloc_context()
Calls to av_write_frame should give us the stream, as long as we manage to configure all the magic flags required for fragmented mpeg4 streams..
We may want to tie this with colour profiles (#1155) since the profile can be specified in the stream and this may be used by the hardware decoders - even if browsers like chromium do not support icc profiles (Support color management transform of ICC profile tagged images to device profile on Linux)
updated patch for b-frames (#800) and ffmpeg 3.1 (#1242)
r12955 adds the enc_ffmpeg codec (see commit message for some important limitations). It does support b-frames (#800).
Bitstream Filtering should help with the muxing part.
This is pretty close to what we want: FFMPEG I/O output buffer
generates mpeg4 container with h264 embedded... to file for now
muxing work if you use pointers as unsigned long, gets mangled otherwise..
Milestone renamed
updated patch against r13044
minor updates
merged in r14427 (big), still needs work in the html5 client as the video is lagging in Firefox and not playing at all in chrome..
Works well enough but can only be tested with the html5 client...
The streams can be saved to file with XPRA_SAVE_TO_FILE=basename
.
The resulting files can be inspected with https://www.bento4.com/'s mp4dump or ffmpeg's ffprobe and looks correct in both cases. It can be played with any decent media player too.
Closing as this will get tested as part of #1341.
PS:
Related links:
Got this reply about movflags / faststart: I have succeeded in moving the moov atom.
Firstly, I set the movflags option when writing the header:
AVDictionary *options = NULL; av_dict_set(&options, "movflags", "faststart", 0); CHECK(avformat_write_header(oc, &options), "Error occurred when writing media header");
Secondly, because I have a custom AVIOContext on my format context, I needed to overwrite the io_open and io_close callbacks on my format context so we don't try to open a non-existent file when writing the trailer.
AVIOContext* ioContext = avio_alloc_context(buffer, BUFFER_SIZE, 1, streamPtr, &Read, &Write, &Seek); formatContext->pb = ioContext; formatContext->flags = AVFMT_FLAG_CUSTOM_IO; formatContext->io_open = &Open; formatContext->io_close = &Close;
My last step was disabling buffering on the AVIOContent I created in the Open callback. This was necessary due to my wrapping the buffer in my original AVIOcontext. The buffer would be invalid. Note that the below code is c++/cli (.net mixed mode).
static int Open(struct AVFormatContext *s, AVIOContext **pb, const char *url, int flags, AVDictionary **options) { GCHandle streamHandle = GCHandle::FromIntPtr(IntPtr(s->pb->opaque)); Stream ^stream = (Stream ^)streamHandle.Target; StreamWrapper ^wrapper = gcnew StreamWrapper(stream); //TODO: check allocations (may be out of memory) unsigned char * buffer = (unsigned char *)av_malloc( BUFFER_SIZE ); streamHandle = GCHandle::Alloc(wrapper); void* streamPtr = GCHandle::ToIntPtr(streamHandle).ToPointer(); *(pb) = avio_alloc_context(buffer, BUFFER_SIZE , 1, streamPtr, &Read, &Write, &Seek); (*pb)->direct = 1; return 0; } static void Close(struct AVFormatContext *s, AVIOContext *pb) { GCHandle streamHandle = GCHandle::FromIntPtr(IntPtr(pb->opaque)); streamHandle.Free(); av_free(pb->buffer); }
If you take a look movenc.c where the mov data is being placed, you'll see where it tries to reopen the file:
/* Shift the data: the AVIO context of the output can only be used for * writing, so we re-open the same output, but for reading. It also avoids * a read/seek/write/seek back and forth. */ avio_flush(s->pb); ret = s->io_open(s, &read_pb, s->filename, AVIO_FLAG_READ, NULL);
It so happens that my AVIOContext is read/write (backed by a .net stream). My StreamWrapper
class simply handles the read/seek/write/seek that this code tries to avoid. I went the wrote of a proxy because I did *not* want to load another stream into memory.
Implementing this buffer seeking has now been moved to #1869.
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1107