xpra icon
Bug tracker and wiki

Opened 23 months ago

Closed 13 months ago

Last modified 13 months ago

#1107 closed enhancement (worksforme)

support some container formats for video

Reported by: alas Owned by: Antoine Martin
Priority: major Milestone: 1.0
Component: html5 Version: trunk
Keywords: Cc:

Description

It would be nice to be able to use a container format, such as webm, for h264 encodings (specifically) for use in processing in html5 video elements. (The native video processing in the browsers don't process naked h264 packets very well.)

Attachments (7)

save-video-to-file.patch (7.3 KB) - added by Antoine Martin 21 months ago.
makes it possible to save the raw video stream to a file on the server
video-container-templates.patch (26.5 KB) - added by Antoine Martin 21 months ago.
stubs for 3 new video container plugins
enc_ffmpeg-v7.patch (29.4 KB) - added by Antoine Martin 18 months ago.
updated patch for b-frames (#800) and ffmpeg 3.1 (#1242)
mpeg4-muxer-v2.patch (19.3 KB) - added by Antoine Martin 18 months ago.
generates mpeg4 container with h264 embedded... to file for now
mpeg4-muxer-v9.patch (42.6 KB) - added by Antoine Martin 18 months ago.
muxing work if you use pointers as unsigned long, gets mangled otherwise..
mpeg4-muxer-v16.patch (25.1 KB) - added by Antoine Martin 17 months ago.
updated patch against r13044
mpeg4-muxer-v19.patch (42.5 KB) - added by Antoine Martin 13 months ago.
minor updates

Download all attachments as: .zip

Change History (21)

comment:1 Changed 23 months ago by alas

As a quick update... would it be possible to use webm & mp4 also, in the event that h264 doesn't work with webm?

comment:2 Changed 23 months ago by Antoine Martin

Owner: changed from Josh to Antoine Martin
Status: newassigned
Summary: Would like to be able to use a container format for h264 packetssupport some container formats for video

The two main contenders that I can see:

both available on Fedora and Debian, but centos doesn't have either of them...

libmkv is meant to be more portable (plain C instead of C++) - its header files look more sane, but its API is file based and we only want in-memory streaming.. converting that to buffer access might be doable, if a bit risky.

webm could be used for vp8/vp9 only.. and is not supported by IE or Safari.

Changed 21 months ago by Antoine Martin

Attachment: save-video-to-file.patch added

makes it possible to save the raw video stream to a file on the server

comment:3 Changed 21 months ago by Antoine Martin

mpeg4 is the only container supported by all browsers: http://www.w3schools.com/html/html5_video.asp. Removal of Ogg Vorbis and Theora from HTML5: an outrageous disaster - monopolies lying through their teeth, and we let them get away with it, again.

Unsurprisingly, the only open-source project that does what we need is ffmpeg: MOV/MP4/ISMV (Smooth Streaming) muxer: A fragmented file consists of a number of fragments, where packets and metadata about these packets are stored together.
Downside is that we don't use ffmpeg for h264, which will make it harder to use the container code... or we have to make a new encoder using ffmpeg. This would need new ffmpeg builds with support for those muxers (that part is no big deal).

So, I am tempted to start with ogg or webm because that should be easier.

comment:5 Changed 21 months ago by Antoine Martin

Using the save-to-file feature merged in r12144, then adding the mpeg4 container using ffmpeg:

ffmpeg -i in.h264 -vcodec copy -f mp4 -movflags faststart out.mp4

I can see that the mpeg4 header is about 0x660 bytes long, and is followed by the raw h264 frames as "mdat".
It looks like what we want is ffmpeg's frag_custom mode: mp4 options: Allow the caller to manually choose when to cut fragments, by calling av_write_frame(ctx, NULL) to write a fragment with the packets written so far so that we can fragment on each frame.

comment:6 Changed 21 months ago by Antoine Martin

Milestone: 0.170.18

Changed 21 months ago by Antoine Martin

stubs for 3 new video container plugins

comment:7 Changed 20 months ago by Antoine Martin

Interesting gstreamer discussion: https://lists.freedesktop.org/archives/gstreamer-devel/2016-April/057989.html, quote: mp4 is not a streamable format in the sense of being able to be played as it is generated.

And: HTTP Adaptive Streaming with GStreamer

Last edited 19 months ago by Antoine Martin (previous) (diff)

comment:8 Changed 18 months ago by Antoine Martin

The patch above gives us AVFrame and AVPacket from our imagewrapper class.
Maybe this will be added even without the muxer.

The next step is the muxer, using libavformat muxing.
Looks like we need AVIOContext and maybe we need to buffer it in memory to support seeking: None of the function pointers in AVIOContext should be called directly, they should only be set by the client application when implementing custom I/O. Normally these are set to the function pointers specified in avio_alloc_context()

Calls to avformat_alloc_output_context2,avformat_write_header and av_write_frame should give us the stream, as long as we manage to configure all the magic flags required for fragmented mpeg4 streams..

Last edited 18 months ago by Antoine Martin (previous) (diff)

comment:9 Changed 18 months ago by Antoine Martin

We may want to tie this with colour profiles (#1155) since the profile can be specified in the stream and this may be used by the hardware decoders - even if browsers like chromium do not support icc profiles (Support color management transform of ICC profile tagged images to device profile on Linux)

Changed 18 months ago by Antoine Martin

Attachment: enc_ffmpeg-v7.patch added

updated patch for b-frames (#800) and ffmpeg 3.1 (#1242)

comment:10 Changed 18 months ago by Antoine Martin

r12955 adds the enc_ffmpeg codec (see commit message for some important limitations). It does support b-frames (#800).

New AVCodec API was very useful here, Bitstream Filtering should help with the muxing part.

This is pretty close to what we want: FFMPEG I/O output buffer

Last edited 18 months ago by Antoine Martin (previous) (diff)

Changed 18 months ago by Antoine Martin

Attachment: mpeg4-muxer-v2.patch added

generates mpeg4 container with h264 embedded... to file for now

Changed 18 months ago by Antoine Martin

Attachment: mpeg4-muxer-v9.patch added

muxing work if you use pointers as unsigned long, gets mangled otherwise..

comment:11 Changed 17 months ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

Changed 17 months ago by Antoine Martin

Attachment: mpeg4-muxer-v16.patch added

updated patch against r13044

Changed 13 months ago by Antoine Martin

Attachment: mpeg4-muxer-v19.patch added

minor updates

comment:12 Changed 13 months ago by Antoine Martin

merged in r14427 (big), still needs work in the html5 client as the video is lagging in Firefox and not playing at all in chrome..

comment:13 Changed 13 months ago by Antoine Martin

Resolution: worksforme
Status: assignedclosed

Works well enough but can only be tested with the html5 client...
The streams can be saved to file with XPRA_SAVE_TO_FILE=basename.
The resulting files can be inspected with https://www.bento4.com/'s mp4dump or ffmpeg's ffprobe and looks correct in both cases. It can be played with any decent media player too.

Closing as this will get tested as part of #1341.

comment:14 Changed 13 months ago by Antoine Martin

PS:

Last edited 13 months ago by Antoine Martin (previous) (diff)
Note: See TracTickets for help on using tickets.