xpra icon
Bug tracker and wiki

Opened 3 years ago

Closed 2 years ago

Last modified 6 months ago

#1107 closed enhancement (worksforme)

support some container formats for video

Reported by: alas Owned by: Antoine Martin
Priority: major Milestone: 1.0
Component: html5 Version: trunk
Keywords: Cc:

Description

It would be nice to be able to use a container format, such as webm, for h264 encodings (specifically) for use in processing in html5 video elements. (The native video processing in the browsers don't process naked h264 packets very well.)

Attachments (7)

save-video-to-file.patch (7.3 KB) - added by Antoine Martin 3 years ago.
makes it possible to save the raw video stream to a file on the server
video-container-templates.patch (26.5 KB) - added by Antoine Martin 3 years ago.
stubs for 3 new video container plugins
enc_ffmpeg-v7.patch (29.4 KB) - added by Antoine Martin 2 years ago.
updated patch for b-frames (#800) and ffmpeg 3.1 (#1242)
mpeg4-muxer-v2.patch (19.3 KB) - added by Antoine Martin 2 years ago.
generates mpeg4 container with h264 embedded... to file for now
mpeg4-muxer-v9.patch (42.6 KB) - added by Antoine Martin 2 years ago.
muxing work if you use pointers as unsigned long, gets mangled otherwise..
mpeg4-muxer-v16.patch (25.1 KB) - added by Antoine Martin 2 years ago.
updated patch against r13044
mpeg4-muxer-v19.patch (42.5 KB) - added by Antoine Martin 2 years ago.
minor updates

Download all attachments as: .zip

Change History (23)

comment:1 Changed 3 years ago by alas

As a quick update... would it be possible to use webm & mp4 also, in the event that h264 doesn't work with webm?

comment:2 Changed 3 years ago by Antoine Martin

Owner: changed from Josh to Antoine Martin
Status: newassigned
Summary: Would like to be able to use a container format for h264 packetssupport some container formats for video

The two main contenders that I can see:

both available on Fedora and Debian, but centos doesn't have either of them...

libmkv is meant to be more portable (plain C instead of C++) - its header files look more sane, but its API is file based and we only want in-memory streaming.. converting that to buffer access might be doable, if a bit risky.

webm could be used for vp8/vp9 only.. and is not supported by IE or Safari.

Changed 3 years ago by Antoine Martin

Attachment: save-video-to-file.patch added

makes it possible to save the raw video stream to a file on the server

comment:3 Changed 3 years ago by Antoine Martin

mpeg4 is the only container supported by all browsers: http://www.w3schools.com/html/html5_video.asp. Removal of Ogg Vorbis and Theora from HTML5: an outrageous disaster - monopolies lying through their teeth, and we let them get away with it, again.

Unsurprisingly, the only open-source project that does what we need is ffmpeg: MOV/MP4/ISMV (Smooth Streaming) muxer: A fragmented file consists of a number of fragments, where packets and metadata about these packets are stored together.
Downside is that we don't use ffmpeg for h264, which will make it harder to use the container code... or we have to make a new encoder using ffmpeg. This would need new ffmpeg builds with support for those muxers (that part is no big deal).

So, I am tempted to start with ogg or webm because that should be easier.

comment:5 Changed 3 years ago by Antoine Martin

Using the save-to-file feature merged in r12144, then adding the mpeg4 container using ffmpeg:

ffmpeg -i in.h264 -vcodec copy -f mp4 -movflags faststart out.mp4

I can see that the mpeg4 header is about 0x660 bytes long, and is followed by the raw h264 frames as "mdat".
It looks like what we want is ffmpeg's frag_custom mode: mp4 options: Allow the caller to manually choose when to cut fragments, by calling av_write_frame(ctx, NULL) to write a fragment with the packets written so far so that we can fragment on each frame.

comment:6 Changed 3 years ago by Antoine Martin

Milestone: 0.170.18

Changed 3 years ago by Antoine Martin

stubs for 3 new video container plugins

comment:7 Changed 3 years ago by Antoine Martin

Interesting gstreamer discussion: https://lists.freedesktop.org/archives/gstreamer-devel/2016-April/057989.html, quote: mp4 is not a streamable format in the sense of being able to be played as it is generated.

And: HTTP Adaptive Streaming with GStreamer

Last edited 3 years ago by Antoine Martin (previous) (diff)

comment:8 Changed 3 years ago by Antoine Martin

The patch above gives us AVFrame and AVPacket from our imagewrapper class.
Maybe this will be added even without the muxer.

The next step is the muxer, using libavformat muxing.
Looks like we need AVIOContext and maybe we need to buffer it in memory to support seeking: None of the function pointers in AVIOContext should be called directly, they should only be set by the client application when implementing custom I/O. Normally these are set to the function pointers specified in avio_alloc_context()

Calls to avformat_alloc_output_context2,avformat_write_header and av_write_frame should give us the stream, as long as we manage to configure all the magic flags required for fragmented mpeg4 streams..

Last edited 3 years ago by Antoine Martin (previous) (diff)

comment:9 Changed 2 years ago by Antoine Martin

We may want to tie this with colour profiles (#1155) since the profile can be specified in the stream and this may be used by the hardware decoders - even if browsers like chromium do not support icc profiles (Support color management transform of ICC profile tagged images to device profile on Linux)

Changed 2 years ago by Antoine Martin

Attachment: enc_ffmpeg-v7.patch added

updated patch for b-frames (#800) and ffmpeg 3.1 (#1242)

comment:10 Changed 2 years ago by Antoine Martin

r12955 adds the enc_ffmpeg codec (see commit message for some important limitations). It does support b-frames (#800).

New AVCodec API was very useful here, Bitstream Filtering should help with the muxing part.

This is pretty close to what we want: FFMPEG I/O output buffer

Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by Antoine Martin

Attachment: mpeg4-muxer-v2.patch added

generates mpeg4 container with h264 embedded... to file for now

Changed 2 years ago by Antoine Martin

Attachment: mpeg4-muxer-v9.patch added

muxing work if you use pointers as unsigned long, gets mangled otherwise..

comment:11 Changed 2 years ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

Changed 2 years ago by Antoine Martin

Attachment: mpeg4-muxer-v16.patch added

updated patch against r13044

Changed 2 years ago by Antoine Martin

Attachment: mpeg4-muxer-v19.patch added

minor updates

comment:12 Changed 2 years ago by Antoine Martin

merged in r14427 (big), still needs work in the html5 client as the video is lagging in Firefox and not playing at all in chrome..

comment:13 Changed 2 years ago by Antoine Martin

Resolution: worksforme
Status: assignedclosed

Works well enough but can only be tested with the html5 client...
The streams can be saved to file with XPRA_SAVE_TO_FILE=basename.
The resulting files can be inspected with https://www.bento4.com/'s mp4dump or ffmpeg's ffprobe and looks correct in both cases. It can be played with any decent media player too.

Closing as this will get tested as part of #1341.

comment:14 Changed 2 years ago by Antoine Martin

PS:

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:16 Changed 6 months ago by Antoine Martin

Got this reply about movflags / faststart:
I have succeeded in moving the moov atom.

Firstly, I set the movflags option when writing the header:

AVDictionary *options = NULL;
av_dict_set(&options, "movflags", "faststart", 0);
CHECK(avformat_write_header(oc, &options), "Error occurred when writing media header");

Secondly, because I have a custom AVIOContext on my format context, I needed to overwrite the io_open and io_close callbacks on my format context so we don't try to open a non-existent file when writing the trailer.

AVIOContext* ioContext = avio_alloc_context(buffer, BUFFER_SIZE, 1, streamPtr, &Read, &Write, &Seek);
formatContext->pb = ioContext;
formatContext->flags = AVFMT_FLAG_CUSTOM_IO;
formatContext->io_open = &Open;
formatContext->io_close = &Close;

My last step was disabling buffering on the AVIOContent I created in the Open callback. This was necessary due to my wrapping the buffer in my original AVIOcontext. The buffer would be invalid. Note that the below code is c++/cli (.net mixed mode).

static int Open(struct AVFormatContext *s, AVIOContext **pb, const char *url, int flags, AVDictionary **options)
{
    GCHandle streamHandle = GCHandle::FromIntPtr(IntPtr(s->pb->opaque));
    Stream ^stream = (Stream ^)streamHandle.Target;
    StreamWrapper ^wrapper = gcnew StreamWrapper(stream);

    //TODO: check allocations (may be out of memory)
    unsigned char * buffer = (unsigned char *)av_malloc( BUFFER_SIZE );
    streamHandle = GCHandle::Alloc(wrapper);
    void* streamPtr = GCHandle::ToIntPtr(streamHandle).ToPointer();

    *(pb) = avio_alloc_context(buffer, BUFFER_SIZE , 1, streamPtr, &Read, &Write, &Seek);
    (*pb)->direct = 1;
    return 0;
}

static void Close(struct AVFormatContext *s, AVIOContext *pb)
{
    GCHandle streamHandle = GCHandle::FromIntPtr(IntPtr(pb->opaque));
    streamHandle.Free();
    av_free(pb->buffer);
}

If you take a look movenc.c where the mov data is being placed, you'll see where it tries to reopen the file:

    /* Shift the data: the AVIO context of the output can only be used for
     * writing, so we re-open the same output, but for reading. It also avoids
     * a read/seek/write/seek back and forth. */
    avio_flush(s->pb);
    ret = s->io_open(s, &read_pb, s->filename, AVIO_FLAG_READ, NULL);

It so happens that my AVIOContext is read/write (backed by a .net stream). My StreamWrapper class simply handles the read/seek/write/seek that this code tries to avoid. I went the wrote of a proxy because I did *not* want to load another stream into memory.

Implementing this buffer seeking has now been moved to #1869.

Last edited 6 months ago by Antoine Martin (previous) (diff)
Note: See TracTickets for help on using tickets.