xpra icon
Bug tracker and wiki

Opened 17 months ago

Closed 6 months ago

#1014 closed enhancement (fixed)

take video size into account for scoring codecs

Reported by: Antoine Martin Owned by: alas
Priority: major Milestone: 1.0
Component: encodings Version: 0.15.x
Keywords: Cc:

Description (last modified by Antoine Martin)

The codecs are rated according to their speed and quality performance, but it may make sense to change which codec is used based on how big the video is: h265 and vp9 become more competitive at higher resolution.
See ticket:832#comment:19.

Also means removing changing the "default" encoding to a new "auto" mode so that we can mix vp8/vp9 and h264 in the same window.

Attachments (4)

ticket1014_screenshot-with-overall-green-vp8_many-orange-rgb-regions_lossy-areas-not-refreshed-for-15-minutes-and-change.png (1.0 MB) - added by alas 7 months ago.
screenshot with paint boxes of vp8 encoded window not losslessly refreshed after 15+ minutes, with rgb & h264 regions that have been static as well
ticket1014_vp8-on-re-size-slow-to-lossless-refresh6.txt (133.7 KB) - added by alas 7 months ago.
xpra info to go with screenshot, taken very shortly after page settled (with boxes indicated by screenshot)
ticket1014_vp8-on-re-size-slow-to-lossless-refresh_even-after-15-minutes7.txt (133.2 KB) - added by alas 7 months ago.
another xpra info after some 15-20 minutes, window still painted the same (maybe video paused?, but no lossless refresh)
ticket1014_vp8-slow-to-lossless-refresh2.txt (137.6 KB) - added by alas 7 months ago.
xpra info from early tests inducing vp8 on 4K display by scrolling, lossless refreshes taking a long half-second to trigger (expected with such large windows?)

Download all attachments as: .zip

Change History (11)

comment:1 Changed 17 months ago by Antoine Martin

Description: modified (diff)
Status: newassigned

comment:2 Changed 13 months ago by Antoine Martin

Milestone: 0.170.18

Re-scheduling.

comment:3 Changed 9 months ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

comment:4 Changed 7 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas
Status: assignednew
  • r13662 adds "size efficiency" attribute for codec scoring
  • r13665 adds an "auto" encoding mode when the client does not specify anything, we then select the most appropriate video encoding as needed, taking into account the size of the video region (and many other heuristics as before).

Testing shows that x264 is still used for "small" dimensions, vp8 starts to win at higher resolutions.
I'm not switching to vp9 because although it compresses better and decompresses faster, it is still slower than vp8 on the fastest setting.
Note: vp8/vp9 doesn't have b-frames. (#800)


@afarr: you should be seeing vp8 encoding being used at higher video resolutions.

comment:5 Changed 7 months ago by alas

Owner: changed from alas to Antoine Martin

Tested with 1.0 r13673 fedora 23 server against 1.0 r13637 osx and win32 clients (hoping new client bits aren't essential).

With a 1440p display as a secondary monitor for the osx client, it was almost impossible to trigger the vp8 encoding, even playing videos full screen... though I was able to trigger by resizing windows near the full display size. I assume 1440p is still "smaller", and that this behavior was expected.

I did notice, though, that portions of the window never got a lossless refresh once the vp8 was triggered (I'll attach a screenshot, and leave it relatively large so some details can be discerned... looks like the video portion is encoding with h264, there are several orange boxed regions being encoded in rgb, but there are several other regions of the window, which seems to be largely a green-boxed vp8 region, which are still displaying lossily even after many minutes... will also attach an xpra info when I initially spotted, and a second taken some 15-20 minutes later with the same region still not refreshed).

With the win32 client on a 4 display, it was considerably easier to trigger the vp8 encoding. Some checking on size just to see if it's behaving about as expected.

  • window.2.size=(2617, 1499) -> very occasional vp8 (only when scrolling so that the entire window area detects as video), nearly always h264.
  • window.2.size=(2713, 1578) -> vp8 much more often, still mostly when scrolling.
  • window.2.size=(2978, 1649) -> vp8 more often than not, though still more when scrolling than video regions (theater mode on youtube isn't big enough to trigger, scrolling the browser page creates a large enough area of detected motion to trigger nearly always).
  • Fullscreen 4K is definitely being rendered with vp8.

Scrolling in the window sizes needed to trigger the vp8, I noticed that (even when the scrolling region ends up being encoded with h264 or jpeg) I'm seeing a fair amount of delay before any lossless refreshes are triggered. On further consideration though, I think I'll just attach one with the vp8 on the 4k display here, and the h264 and jpeg lossy xpra infos in #967.

If there doesn't seem to be anything suspicious in the slow refresh info outputs (or if there's something that should be broken out to another ticket), then I think this can be closed... vp8 is definitely being seen more often than previously and more often the larger the video region detected.

Changed 7 months ago by alas

screenshot with paint boxes of vp8 encoded window not losslessly refreshed after 15+ minutes, with rgb & h264 regions that have been static as well

Changed 7 months ago by alas

xpra info to go with screenshot, taken very shortly after page settled (with boxes indicated by screenshot)

Changed 7 months ago by alas

another xpra info after some 15-20 minutes, window still painted the same (maybe video paused?, but no lossless refresh)

Changed 7 months ago by alas

xpra info from early tests inducing vp8 on 4K display by scrolling, lossless refreshes taking a long half-second to trigger (expected with such large windows?)

comment:6 Changed 7 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas

I assume 1440p is still "smaller", and that this behavior was expected.


The heuristics take various things into account, 1440p is near the limit in most cases. (x264 still scores higher for quality and speed).

I think we can close this ticket.
FYI: as of r13684, the "auto" encoding option is now visible in the tray menu, the command line help, etc..


Scrolling should be using the new "scroll" encoding: #1232 and not video.

Refresh issues are unlikely to be caused by the changes in this ticket: the refresh logic does not differentiate between vp8 and h264 video encodings.
Please include the "-d regionrefresh" for video refresh issues only, "-d refresh" for refresh issues in other areas of the page. (in a separate ticket unless going back to a pre-r13662 version fixes the issue).

Last edited 7 months ago by Antoine Martin (previous) (diff)

comment:7 Changed 6 months ago by alas

Resolution: fixed
Status: newclosed

Both details noted (and hopefully will be remembered when the time comes.

I'll take the liberty of closing this.

Note: See TracTickets for help on using tickets.