Xpra: Ticket #1257: better x264 tuning for type of content

Related to #1107, #1232 and #800.

We (almost) already know if the content we are encoding using the video encoders is really real video or just fast moving. This is what we use to enable b-frames (#800), we should also use this to tune x264 better.



Fri, 15 Jul 2016 15:13:44 GMT - Antoine Martin: status changed

For details on tune, see: Explanation of x264 tune.

ie:

XPRA_B_FRAMES=0 \
XPRA_FORCE_CSC_MODE=YUV420P \
XPRA_X264_PROFILE=high10 \
XPRA_X264_TUNE=film \
    xpra --bind-tcp=0.0.0.0:10000 start --speed=100 -d x264

Will use YUV420P csc mode, high10 profile, film tune with ultrafast preset (speed). In this case, using "film" without b-frames is not particularly useful!

The main question here is to figure out what is the best setting to use, both for video and non-video content. There are going to be many quality vs cpu vs bandwidth vs latency trade offs to choose from, so maybe using x264 directly from the command line would be quicker.


Wed, 20 Jul 2016 15:44:20 GMT - Antoine Martin:

As of r13051, we are also more likely to use colour subsampling for video regions, and less likely to use it for other areas (the rest of the window).


Wed, 20 Jul 2016 17:10:52 GMT - Antoine Martin:

As of r13052, we will use "film" tuning (intended for high-bitrate/high-quality movie content. Lower deblocking is used here) for video regions, and "zerolatency" (optimization for fast encoding and low latency streaming) for everything else.

Here's the "-d x264" output for a video region:

using default profile=high444
x264 context=0x7fabf40497c0,    BGRX  300x300  preset=faster, profile=high444, tune=film
 me=HEX, me_range=16, mv_range=-1, opencl=False, b-frames=1, max delayed frames=1
 vfr-input=0, lookahead=0, sync-lookahead=-1, mb-tree=1, bframe-adaptive=FAST
 threads=auto, sliced-threads=True
x264 encode 1 delayed frames after 0
x264 encode frame 0 as  IDR slice with 7 nals, total   15737 bytes, keyframe=True , delayed=1
x264 encode frame 1 as    P slice with 4 nals, total   12211 bytes, keyframe=False, delayed=1
x264 encode frame 2 as    B slice with 4 nals, total    7849 bytes, keyframe=False, delayed=1
x264 encode frame 3 as    P slice with 4 nals, total   12390 bytes, keyframe=False, delayed=1
x264 encode frame 4 as    B slice with 4 nals, total    7695 bytes, keyframe=False, delayed=1

And for non-video (no b-frames):

using default profile=high444
x264 context=0x7efc480151c0,    BGRX  498x316  quality=99, speed=74, source=unknown
 preset=faster, profile=high444, tune=zerolatency
 me=DIA, me_range=16, mv_range=-1, opencl=False, b-frames=0, max delayed frames=0
 vfr-input=0, lookahead=-1, sync-lookahead=0, mb-tree=0, bframe-adaptive=FAST
 threads=auto, sliced-threads=True
x264 encode frame     0 as  IDR slice with 7 nals, total  124041 bytes, keyframe=True , delayed=0
x264 encode frame     1 as    P slice with 4 nals, total     154 bytes, keyframe=False, delayed=0
x264 encode frame     2 as    P slice with 4 nals, total     278 bytes, keyframe=False, delayed=0
x264 encode frame     3 as    P slice with 4 nals, total     109 bytes, keyframe=False, delayed=0

This should be tested together with #800 as there is a lot of overlap.


Wed, 20 Jul 2016 19:39:24 GMT - Antoine Martin:

r13056 allows us to detect the whole window as a video region, so that we will correctly use colour subsampling and more aggressive compression settings for full screen video (or anything full screen with high fps, like glxgears).

If this makes us wrongly detect the whole window as video area, we may need to tweak the video region detection code #967 (#410 - see also #1135, #1060)

r13058 also exposes the tune option via xpra info:

$ xpra info | grep tune
window.2.encoder.tune=film

Thu, 21 Jul 2016 12:23:18 GMT - Antoine Martin: owner, status changed

Ready for testing, including b-frames from #800.

We want to:

Note: if we end up with a blurry window and delayed paints because the whole window is mistakenly identified as a video region then this needs to go in #967, not here.


Mon, 25 Jul 2016 17:32:09 GMT - J. Max Mena: owner changed

Using a Fedora 23 trunk r13086 built from source server and client:


Note: if we end up with a blurry window and delayed paints because the whole window is mistakenly identified as a video region then this needs to go in #967, not here.


I do see this, will leave a comment in #967.



ideally, measure the bandwidth savings and cpu costs of those new settings - do we need to adjust the speed (or even quality) setting to compensate?


I'll talk about this with Smo and see what he thinks. Last I heard he was still setting up his testing environment. We also got a new hire and she should be able to do this once she's all set up.


Tue, 26 Jul 2016 08:27:23 GMT - Antoine Martin: owner changed

However, typing quickly into a text box (like Trac tickets...) does trigger an h264 region, also using film, not zerolatency.


Anything that is mistakenly detected as a video region will end up using film. You must have had a video region detected at that point, you can verify with xpra info | grep region. So unless this is not the case, it does not belong here (x264 tuning) but in #967 (video region detection).



Last I heard he was still setting up his testing environment.


Note: you will need to run one or more sets of tests with extra environment variables to compare with and without b-frames and tuning.


Tue, 23 Aug 2016 13:16:02 GMT - Antoine Martin:

r13436 further lowers the quality of "true" video (more than 10 frames per second, and at least 20 frames have been processed using the given video region) As of r13439, we also only enable "b-frames" and the "video" encoder tuning for "true" video. See also #1232: scrolling should now be detected and compressed better. There is now more information in

$ xpra info | grep subregion
$ xpra info | grep -e "window\.[0-9]*\.encoder.*"

Fri, 09 Sep 2016 11:24:01 GMT - Antoine Martin:

See ticket:1232#comment:16 which now exposes the "video-mode" flag for "true" video. (>10fps, etc..)

As of r13775, we can switch to and from "true" video on the fly, see ticket:800#comment:25


Fri, 28 Oct 2016 18:59:33 GMT - J. Max Mena: owner changed

Alright, so I think it's time to make a decision on this one.

It looks like we (afarr and myself) aren't gonna get someone who can write the tests, and aren't going to be given the time to write them ourselves.

So, @antoine - what would you like to do with this ticket? The x264 tuning works well in my experience, and I haven't had any issues with it. We can hand it to Smo, in the hopes that he will have time to include it in his tests? Probably not the best idea as he's already busy.


Sat, 29 Oct 2016 04:45:49 GMT - Antoine Martin: status changed; resolution set

Closing and hoping for the best :/


Sat, 23 Jan 2021 05:19:19 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1257