Xpra: Ticket #415: ffmpeg 2.0 support

Split from #398

There are API changes in ffmpeg which cause compatibility problems with how we use the buffer API. (see #398 for details)

See also the problems with libav in #421



Thu, 17 Oct 2013 07:38:21 GMT - Antoine Martin: owner, status, description, milestone changed


Wed, 06 Nov 2013 11:09:18 GMT - Antoine Martin: attachment set

untested patch


Tue, 12 Nov 2013 08:37:08 GMT - Antoine Martin:

The solution is quite simple it seems: Re: get_buffer vs get_buffer2, how do I manage/release buffers?

We can just drop all the context lookup code for v2!


Thu, 14 Nov 2013 13:58:06 GMT - Antoine Martin: attachment set

patch using refcounted_frames - not working, crashes in av_frame_unref dereferencing a NULL pointer for side_data...


Fri, 15 Nov 2013 14:00:20 GMT - Antoine Martin: priority, milestone changed

Fedora 20 is going to be out soon, this needs to be dealt with.


Sat, 30 Nov 2013 10:10:08 GMT - Antoine Martin: attachment set

working ffmpeg v2 patch to apply on top of r4816


Sat, 30 Nov 2013 10:45:37 GMT - Antoine Martin: owner, status changed

Patch above merged as a new codec (too many changes to manage with IFDEFs alone) in r4821 + r4835. The RPM spec file adds this for Fedora 20 and 21:

--without-dec_avcodec --with-dec_avcodec2

Notes:

afarr: can you please compare clients ffmpeg v1 builds (like Fedora 19) and ffmpeg v2 builds (like Fedora 20) and see if there are any regressions or any major differences in client CPU usage when under load (high FPS with big windows) - if so, we'll need to implement the complicated code before we can move win32 and OSX to ffmpeg v2.


Sun, 01 Dec 2013 12:49:46 GMT - Antoine Martin:

Backported to v0.10.x in r4827 + r4829 + r4837 + r4838 + r4844


Sat, 04 Jan 2014 06:28:22 GMT - Antoine Martin:

Works fine on win32, as of r5111 simply:

Notes:

to avoid this error: xpra/codecs/dec_avcodec2/decoder.so: version node not found for symbol av_fast_malloc@LIBAVCODEC_55


Fri, 10 Jan 2014 04:06:55 GMT - Antoine Martin:

Some clarifications for testing:

The application used for generating video frames should produce exactly the same video stream each time. I usually use glxgears, but something like mplayer playing the same local video will also do. A browser is not a good test since advertising may vary, network speed may vary, etc..

As usual, it is important to bear in mind that different runs may give different results in terms of number of frames, quality and speed settings, etc. To truly compare the two libraries, let them run freely and record the settings they settle on (xpra info), and also do test runs with fixed settings. Bonus points for using pretty wiki tables to present the results.


Fri, 10 Jan 2014 21:37:07 GMT - J. Max Mena:

I tested on version 0.11.0 r5158 with and without ffmpeg2. I sat and watched the same twitch.tv stream for a good half hour. I had to make a second attempt for the version without ffmpeg2 because my network connection fussed up and the connection dropped. Anyways, I logged my CPU and memory usage watching the stream every 5 minutes and here they are:

without ffmpeg2:

Time Memory (in Megabytes) CPU % usage
Start 107.9mb 6-8%
5 min. 112.3mb same
10 min. 113.8mb same
15 min. 116.1mb same
20 min. 113.1mb higher 9-10%
25 min. 116.1mb slightly higher 9-11%
30 min. 117.0mb down to 8-10%

with ffmpeg2:

Time Memory (in Megabytes) CPU % usage
Start 120.0mb 6-8%
5 min. 121.0mb same
10 min. 124.1mb same
15 min. 125.7mb same
20 min. 126.5mb same
25 min. 125.9mb same
30 min. 126.1mb same

After 30 minutes on both versions, I closed the stream and tried out a couple of websites, like reddit, gmail, along with youtube. The performance was about the same, if not better on ffmpeg2. I'm running Win7 64-bit here on my laptop, and it was running pretty stable the entire time. From my experience the performance and CPU cost is the same with and without ffmpeg2. All in all, it only broke 10% a couple of times while watching the stream, and when not doing anything graphics intensive it never went above 2% usage. With the exception of the network, it ran stable the whole time.

Laptop specs for the curious:


Sat, 11 Jan 2014 00:30:26 GMT - Antoine Martin: owner changed

Please see comment:7 where I specifically said that a browser was *not* a good way of testing. Also, since the number of frames is not reported, it is impossible to know if the decoding effort was equivalent (one cannot visually discern changes in dropped frames at ~25fps)

Both result tables are named without ffmpeg2 - I assume this is a mistake? (it would be easier to read if each version had a distinct column(s) in the same table)


Sat, 11 Jan 2014 00:59:16 GMT - J. Max Mena:

Oops that was a typo. I updated it. Also, I must have missed your comment when I started testing.


Tue, 14 Jan 2014 00:29:26 GMT - J. Max Mena:

I re-ran the test using a local video file in mplayer. I used clive http://clive.sourceforge.net/ to download the file from Youtube. The video I used was:

http://www.youtube.com/watch?v=KaOC9danxNo

It's 5:30 long and has some good high frame rate footage. I watched it and took down the CPU and memory usage every 2 minutes, with the last part taken down at ~5 minutes into the video for both tables.

Here is the relevant data:

With ffmpeg1
Time Memory (in Megabytes) CPU % usage
Start 113.4mb 9-15%
2 min. 118.4mb same
4 min. 119.4mb same
5 min. 119.58mb same
With ffmpeg2
Time Memory (in Megabytes) CPU % usage
Start 109.9mb 9-15%
2 min. 112.6mb same
4 min. 114.8mb same
5 min. 115.7mb same

Tue, 14 Jan 2014 14:22:21 GMT - Antoine Martin: owner, status changed

Here's how I tested:

Started just after connecting with --no-speaker (to prevent the sound from interfering).

(the window we are interested in is the mplayer window - the "second" one)


elapsed time memory usage (MB) client latency avg/max actual batch delay client decode speedqualityspeedtotal frames
ffmpeg1ffmpeg2ffmpeg1ffmpeg2ffmpeg1ffmpeg2ffmpeg1ffmpeg2ffmpeg1ffmpeg2ffmpeg1ffmpeg2ffmpeg1ffmpeg2
0 (*)555541 / 27627 / 1994547506772627759n/an/a
10m747124 / 9524 / 9251497045606161618215 h264 + 8214 rgb248133 h264 + 8132 rgb24
20m888825 / 10422 / 186515865546161606016389 h264 + 16384 rgb2416504 h264 + 16503 rgb24

(*) the first line of data is not particularly relevant for most values as things haven't had time to settle down


This raises more questions than it answers:


Thu, 16 Jan 2014 11:13:01 GMT - Antoine Martin:

The rgb24 frames are due to the xterm which has a size of:

window[1].size=(499, 316)

And it is encoded with x264, which does not support odd dimensions, so we send the 1 pixel edge as rgb24. (not a bug)

There was a small bug in the rgb processing though, fixed in r5204, which prevented proper accounting of rgb frame decoding (as they were acked as failed).


Fri, 17 Jan 2014 09:42:56 GMT - Antoine Martin: status changed; resolution set

Works well enough, now used with Fedora 20+ builds, static builds (r5194), win32 (r5188) and osx (r5193)

Memleak will be tracked in #457


Sat, 23 Jan 2021 04:54:46 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/415