Xpra: Ticket #419: better/faster encoding selection

The problem: we need lossless encoding (small regions, auto-refresh), but:



Wed, 21 Aug 2013 10:03:47 GMT - Antoine Martin: owner, status, description changed


Fri, 15 Nov 2013 14:13:59 GMT - Antoine Martin: milestone changed

too late for 0.11 and fixing webp (#491) will have to be done first


Sun, 09 Feb 2014 08:25:21 GMT - Antoine Martin: milestone changed

re-scheduling all webp stuff


Sun, 30 Mar 2014 11:28:55 GMT - Antoine Martin: attachment set

stub for a cython version of the webp encoder


Mon, 31 Mar 2014 11:22:42 GMT - Antoine Martin:

The webp encoder in Cython was added in r5958, used ahead of the old python-webm encoder (which is still used as fallback for pixel modes it cannot handle).

r5959 adds some tests that make it easier to compare webp / webm, png and jpeg encoders. Some preliminary results:

Conclusions:

Some performance data using some sample images, showing both speed (in mega pixels per second - higher is better, and compression ratio - lower is better):

Mode Desktop 2560x1600Browser 1920x1080Diagram 640x800Small Alpha 64x48
EncodingCompressorPresetQualitySpeedMPixels/scompMPixels/scompMPixels/scompMPixels/scomp
pngPIL (old)optimized 0 1.6 8.1
pngPIL (old) 100 6.5 8.5
pngPIL 100 11.6 10.0 16.1 3.1 14.6 5.0 6.0 30.1
png/PPIL 100 21.8 4.7 17.2 9.0 23.9 3.9 2.2 14.0
png/LPIL 100 24.3 3.8 31.1 1.4 29.7 2.6 7.6 4.7
webpPIL 0 5.5 0.5 6.3 0.5 5.9 0.7 0.4 13.4
webpPIL 50 5.1 2.1 6.0 1.1 5.6 1.6 0.4 16.7
webpPIL 100 4.3 6.2 5.9 3.1 5.3 4.3 0.4 28.8
jpegPILoptimized 0 71.0 0.5 68.5 0.4 80.6 0.5 16.6 3.2
jpegPIL 50 86.0 3.1 68.1 1.9 68.2 2.5 12.7 8.8
jpegPILoptimized 100 48.0 10.8 56.8 5.7 56.0 7.7 9.5 26.1
jpegPIL 100 72.0 13.0 58.4 5.7 56.5 7.7 8.4 26.1
rgblz4 100 224.0 15.0 497.0 4.3 526.0 6.9 76.7 56.0
rgbzlib 50 21.0 9.1 30.2 2.5 27.8 3.1 11.3 39.4
webppython-webm 0 6.0 0.8 6.7 0.5 6.2 0.7 0.5 13.4
webppython-webm 50 5.3 2.1 6.5 1.1 5.7 1.6 0.6 16.7
webppython-webm 100 0.3 4.8 1.0 0.7 0.7 1.6 0.1 25.0
webpCythonTEXT 0 100 18.6 0.6 19.4 0.4 18.9 0.6 1.5 2.9
webpCythonTEXT 50 100 15.5 2.7 17.7 1.5 16.6 2.2 1.2 11.9
webpCythonTEXT 99 100 12.6 6.7 15.2 3.3 13.9 4.6 0.8 29.7
webpCythonTEXT + lossless 100 0 0.1 4.9 0.6 0.7 0.3 1.6 0.0 24.7
webpCythonTEXT + lossless 100 100 2.3 5.9 27.8 1.3 27.2 1.9 1.5 26.9

Notes and caveats:


Tue, 01 Apr 2014 03:45:36 GMT - Antoine Martin:

After figuring out that libwebp wants us to use the quality attribute for storing the speed when in lossless mode... I kid you not! applied in r5962. The results are a lot better on "high speed" (s=100%), but still much slower than png... (table above updated)


Tue, 01 Apr 2014 07:47:11 GMT - Antoine Martin:

Digging further shows that it is possible to make png faster instead. r5966 tunes png and jpeg better.

I have also tested to see if changing the png compression_type attribute can give us better compression or performance (no: default is best):

Desktop 2560x1600Browser 1920x1080Diagram 640x800Small Alpha 64x48
Compression TypeLevelMPixels/scompMPixels/scompMPixels/scompMPixels/scomp
HUFFMAN_ONLY n/a 9.8 19.0 11.4 15.8 10.8 17.3 6.6 29.9
FIXED 1 12.1 11.7 16.7 3.6 14.2 5.8 5.7 35.1
FIXED 3 10.7 10.7 16.6 3.5 13.8 5.5 5.0 33.0
RLE 1 12.1 11.5 16.6 4.6 15.2 6.5 7.1 27.3
RLE 3 10.8 10.7 16.6 4.6 14.7 6.5 6.8 27.3
FILTERED 1 12.1 10.0 16.7 3.1 14.1 5.0 5.8 30.1
FILTERED 3 10.7 9.4 16.4 3.0 12.0 4.8 5.0 29.1
DEFAULT (old code) unset 6.8 8.4 12.5 2.7 10.3 4.9 3.4 26.2

Notes:


Tue, 01 Apr 2014 14:00:14 GMT - Antoine Martin: summary changed

(updating bug title to reflect the greater scope of this ticket)


Wed, 02 Apr 2014 14:33:07 GMT - Antoine Martin: attachment set

updates the encoding heuristics based on the new data


Thu, 03 Apr 2014 05:08:22 GMT - Antoine Martin: summary changed

Many improvements:


Sat, 17 May 2014 05:30:37 GMT - Antoine Martin: owner, status changed

smo: please run this latest code on your test system so we can compare with previous runs and ensure the performance is at least as good as before for most tests


Mon, 26 May 2014 15:32:25 GMT - Antoine Martin: priority changed

Important regression discovered and fixed in r6568 (see "Trouble with xpra 0.13" mailing list posts for the bug report), please test and verify that PNG, RGB, png/L and png/P never end up using lossy encodings, no matter what command line options or previous encodings were used. And no matter how bad the refresh rate gets when under pressure.

Raising to blocker as this should definitely go in 0.13.1. Backported in r6571. Quite a few fixes have gone in, none of them quite as critical as the one above (transparency, opengl toggle, etc): r6577, r6578, r6583


Mon, 21 Jul 2014 22:09:00 GMT - alas:

Testing with win client 0.14.0 r6910 against fedora 20 0.14.0 r6910 server.

With a client-side --encoding=png the encoding seems to stick to png, no matter how bad it gets.

2014-07-21 12:39:45,496 update_speed() info={'damage_latency.abs_factor': 0, 'damage_latency.avg': 6, 'decoding_latency.target': 8000000, 'damage_latency.target': 35, 'min_speed': 0, 'frame_delay': 8, 'decoding_latency.factor': 109$
2014-07-21 12:39:45,496 reconfigure(False) csc_encoder=None, video_encoder=None
2014-07-21 12:39:47,190 damage(WindowModel(0xa00022 - "jimador@zapopan:~"), 155, 301, 6, 13, {})
2014-07-21 12:39:47,190 damage(155, 301, 6, 13, {}) wid=1, sending now with sequence 59
2014-07-21 12:39:47,191 process_damage_regions: wid=1, adding png pixel data to queue, elapsed time: 1.5 ms, request rgb time: 0.6 ms
2014-07-21 12:39:47,192 make_data_packet: image=XShmImageWrapper(BGRX: 155, 301, 6, 13), damage data: (1, 155, 301, 6, 13, 'png')
2014-07-21 12:39:47,195 update_speed() info={'damage_latency.abs_factor': 0, 'damage_latency.avg': 7, 'decoding_latency.target': 8000000, 'damage_latency.target': 35, 'min_speed': 0, 'frame_delay': 4, 'decoding_latency.factor': $
2014-07-21 12:39:47,195 sending 6x13 BGRX as png, mode=RGB, options={'compress_level': 1}
2014-07-21 12:39:47,195 reconfigure(False) csc_encoder=None, video_encoder=None

Likewise with RGB & png/L.

With --encoding=png/P however, the encoder seems to use png/P for "existing delayed regions" (2014-07-21 14:32:10,285 damage(86, 427, 640, 54, {}) wid=3, using existing delayed png/P regions created 0.0ms ago), but seems to bounce around from rgb to webp to png/P overall.

Here's a bit of a clip of server-side logs (I'll include a full long log for this):

2014-07-21 14:31:32,441 update_speed() info={'damage_latency.abs_factor': 0, 'damage_latency.avg': 8, 'decoding_latency.target': 8000000, 'damage_latency.target': 35, 'min_speed': 0, 'frame_delay': 24, 'decoding_latency.factor':$
2014-07-21 14:31:32,442 damage(143, 2, 6, 13, {}) wid=1, sending now with sequence 2
2014-07-21 14:31:32,442 reconfigure(False) csc_encoder=None, video_encoder=None
2014-07-21 14:31:32,444 damage(WindowModel(0xc00022 - "jimador@zapopan:~"), 143, 2, 6, 13, {})
2014-07-21 14:31:32,444 damage(143, 2, 6, 13, {}) wid=2, scheduling batching expiry for sequence 3 in 24.0 ms
2014-07-21 14:31:32,446 process_damage_regions: wid=1, adding rgb24 pixel data to queue, elapsed time: 5.5 ms, request rgb time: 1.1 ms
2014-07-21 14:31:32,446 make_data_packet: image=XShmImageWrapper(BGRX: 143, 2, 6, 13), damage data: (1, 143, 2, 6, 13, 'rgb24')
...
2014-07-21 14:32:22,577 damage(86, 325, 640, 102, {}) wid=3, using existing delayed png/P regions created 0.0ms ago
2014-07-21 14:32:22,577 damage(WindowModel(0xe00001 - "John Oliver Is Psyched for the World Cup - Late Night with Seth Meyers - YouTube - Google Chrome"), 86, 427, 640, 54, {})
2014-07-21 14:32:22,577 damage(86, 427, 640, 54, {}) wid=3, using existing delayed png/P regions created 0.0ms ago
2014-07-21 14:32:22,618 send_delayed for wid 3, delaying again because of backlog: 3 packets, batch delay is 42.3135884506, elapsed time is 43.0 ms
2014-07-21 14:32:22,664 damage(WindowModel(0xe00001 - "John Oliver Is Psyched for the World Cup - Late Night with Seth Meyers - YouTube - Google Chrome"), 86, 481, 512, 6, {})
2014-07-21 14:32:22,664 damage(86, 481, 512, 6, {}) wid=3, using existing delayed png/P regions created 0.1ms ago
2014-07-21 14:32:22,704 send_delayed_regions: bytes_cost=238592, bytes_threshold=698084, pixel_count=233472
2014-07-21 14:32:22,707 process_damage_regions: wid=3, adding webp pixel data to queue, elapsed time: 132.2 ms, request rgb time: 2.5 ms
2014-07-21 14:32:22,707 make_data_packet: image=XShmImageWrapper(BGRX: 86, 121, 640, 366), damage data: (3, 86, 121, 640, 366, 'webp')

If I'm not mistaken, though, none of those are "lossy encodings" ... so that may be expected behavior.

Smo will try to arrange those tests for performance as soon as he can.


Tue, 22 Jul 2014 01:45:01 GMT - alas: attachment set

ticket 419 (overlarge) server-side png/P -d encode log


Wed, 20 Aug 2014 02:20:47 GMT - Antoine Martin: owner, status changed

I guess this should have been re-assigned to me - taking it back.

I thought that this was all working as expected until #645

I will try to clarify the logging:

And so the encoding actually may well be different from the one that is set, especially when:

etc.


Sun, 24 Aug 2014 06:08:03 GMT - Antoine Martin: status changed; resolution set

Despite some fairly big bugs that went unnoticed until after the release (#645 mentioned above and also #651), I think this is now working as expected and also much faster than before (see #620).

r7424 removes 'rgb' from the logging in process_damage_regions.

I have added information to the wiki on debugging encodings here: wiki/Encodings. Closing this ticket, some of these issues will be followed up in #410.


Sat, 23 Jan 2021 04:54:53 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/419