xpra icon
Bug tracker and wiki

Version 51 (modified by Antoine Martin, 5 years ago) (diff)

--

http://xpra.org/icons/encoding.png

Picture Encodings

This page is strongly related to WindowRefresh, the Network and client rendering are also relevant.
For video encodings (vpx and h264), colorspace conversion is also required reading.


Introduction

Xpra supports a number of picture encodings, provided you have the required libraries installed. Even then, the features of each encoding may vary based on the version of the libraries and other dependencies, both client and server side.
Here is the list as of v0.13

  • lossless encodings:
    • rgb: rgb24 for regular windows, rgb32 for transparency support (compressed with zlib or lz4, just like packet compression)
    • png (24/32-bit colour), png/L (8-bit grayscale) and png/P (8-bit colour palette)
    • webp (lossless mode)
  • lossy encodings:
    • jpeg
    • webp (lossy mode)
    • vpx (VP8 and VP9)
    • h264: either using x264 or nvenc
    • h265 using x265 (not usable, far too slow)


Note: for backwards compatibility, on versions older than 0.10.10 you have to specify the encoding as x264 to get h264 and vpx to get vp8. Newer versions use h264 and vp8, but x264 and vpx will remain for command-line backwards compatibility.

Choosing an Encoding

The best thing to do is to try them all and choose the one that provides the best results.

Here are some rough guidelines:

  • on LANs with 100MBit/s or higher: rgb + zlib/lz4 should give you the best latency possible (whilst consuming quite a lot of bandwidth in the process..)
  • if you have the required hardware, use NVENC
  • to save a little bit of bandwidth at the expense of framerate and latency, use lossless encodings: png or webp
  • otherwise, choose h264 and tune the speed/quality to suit your needs (see below)

The other encodings are somewhat less useful:

  • vpx : VP8 is similar to h264 but it does not support all speed and quality tuning, it also subsamples colours which leads to visual degradation. VP9 is only usable with version 0.15 onwards.
  • h265 via x265 is far too slow to encode
  • jpeg gives lower size/quality than other lossy encodings, use this if the video encodings are not available and you need better compression than png or webp can give you
  • webp (lossy mode) is a single image subset of vpx, and therefore lacks intra-frame compression - but latency is decent

Colourspace conversion step - CSC

Before passing the pixels to the video encoder, we may or may not include a colourspace conversion step, (some newer versions of x264 support BGRA pixels as input directly) it is required when downscaling the video.

See: Colourspace conversion step - CSC

Tuning

The best way to choose the right options is through wiki/Testing. Though you may find some good illustrations online to give you an idea of the trade offs. (ie: fps vs noise, fps vs size) Be aware that the lossless auto-refresh will trigger a lossless frame encoded using png or rgb. If the delay is too low, it may negate the benefits of using efficient compression.

Notes

When comparing performance, make sure that you use the right metrics... The number of updates per second is not always a good one (if there are more small regions, this can be a good or a bad thing), more examples here: Misleading Statistics

Statistics

As part of encoding improvements (mostly in #419), we have collected encoding performance statistics for 4 still test pictures using the current crop of codec code (pre-release version 0.13). These encodings are also used when a video encoding is selected as primary encoding, this is done to optimize bandwidth and CPU usage: video encodings require a full frame everytime, so we use still picture encodings for smaller regions and for automatic lossless refresh.

Mode Desktop 2560x1600Browser 1920x1080Diagram 640x800Small Alpha 64x48
EncodingCompressorPresetQualitySpeedMPixels/scompMPixels/scompMPixels/scompMPixels/scomp
pngPIL (old)optimized 0 1.6 8.1
pngPIL (old) 100 6.5 8.5
pngPIL 100 11.6 10.0 16.1 3.1 14.6 5.0 6.0 30.1
png/PPIL 100 21.8 4.7 17.2 9.0 23.9 3.9 2.2 14.0
png/LPIL 100 24.3 3.8 31.1 1.4 29.7 2.6 7.6 4.7
webpPIL 0 5.5 0.5 6.3 0.5 5.9 0.7 0.4 13.4
webpPIL 50 5.1 2.1 6.0 1.1 5.6 1.6 0.4 16.7
webpPIL 100 4.3 6.2 5.9 3.1 5.3 4.3 0.4 28.8
jpegPILoptimized 0 71.0 0.5 68.5 0.4 80.6 0.5 16.6 3.2
jpegPIL 50 86.0 3.1 68.1 1.9 68.2 2.5 12.7 8.8
jpegPILoptimized 100 48.0 10.8 56.8 5.7 56.0 7.7 9.5 26.1
jpegPIL 100 72.0 13.0 58.4 5.7 56.5 7.7 8.4 26.1
rgblz4 100 224.0 15.0 497.0 4.3 526.0 6.9 76.7 56.0
rgbzlib 50 21.0 9.1 30.2 2.5 27.8 3.1 11.3 39.4
webppython-webm 0 6.0 0.8 6.7 0.5 6.2 0.7 0.5 13.4
webppython-webm 50 5.3 2.1 6.5 1.1 5.7 1.6 0.6 16.7
webppython-webm 100 0.3 4.8 1.0 0.7 0.7 1.6 0.1 25.0
webpCythonTEXT 0 100 18.6 0.6 19.4 0.4 18.9 0.6 1.5 2.9
webpCythonTEXT 50 100 15.5 2.7 17.7 1.5 16.6 2.2 1.2 11.9
webpCythonTEXT 99 100 12.6 6.7 15.2 3.3 13.9 4.6 0.8 29.7
webpCythonTEXT + lossless 100 0 0.1 4.9 0.6 0.7 0.3 1.6 0.0 24.7
webpCythonTEXT + lossless 100 100 2.3 5.9 27.8 1.3 27.2 1.9 1.5 26.9

Notes:

  • all tests were performed on an AMD FX-8150 (octa-core 3GHz AMD CPU)
  • video encodings are not included, and should generally be preferred to the single picture encodings shown here
  • this shows both speed in mega pixels per second (higher is better), and compression ratio (lower is better)
  • Low quality at low speed doesn't really make much sense, so some values have been omitted
  • changing the webp preset does not affect speed or output size much, it probably does affect the perceived picture quality (seems best to stick with TEXT to ensure that text remains readable)
  • The new webp encoder seems to perform about the same with speeds >50%, only low speed is really slow, and lossless is unbearably slow (it can take more than 30 seconds to encode a single frame using lossless + low speed!)
  • quality is not the same for each encoder, some have lossless modes others not, etc..
  • for data showing the differences between the various png compression types (HUFFMAN_ONLY, FIXED, RLE, FILTERED, DEFAULT), see ticket:419#comment:6


You can find a much more detailed analysis (but which is limited to lossy formats in YUV420 colourspace mode...) here: Lossy Compressed Image Formats Study

Future Encoding Support

We should eventually add support for:

  • #617 f265 (h265)
  • #616 HEVC (single frame)
  • #454 daala - unlikely
  • #455 and #464: vp9 and libvpx improvements

Debugging

The first thing to check is the codec availability and version. On MS Windows, run the Encoding_Info.exe utility, or the bug report tool. On other platforms, run the xpra/codecs/loader.py script, and for debugging video the xpra/codecs/video_helper.py script. (both support a --verbose option).

When using non-video encodings, the encoding used for sending the pixels to the client will be the one which has been selected. The only exception to this rule is when the number of pixels is so small that trying to compress them would be pointless, and they are then usually sent as plain {{rgb}}} or png / webp.

With video encodings, things are more complicated: if there is a video region, the non video areas will use other encodings. Even the video region (which may be the whole window) will get automatically refreshed with a lossless encoding when it stops refreshing rapidly.

To debug encoding selection:

  • the actual encoding used is logged with -d compress in the form:
    make_data_packet: image=XShmImageWrapper(BGRX: 149, 2, 6, 13), damage data: (1, 149, 2, 6, 13, 'rgb24')
    compress:   0.9ms for  499x316  pixels using  mmap with ratio   0.0% (  615KB to     0KB), delta=-1, client_options={'rgb_format': 'BGRX'}
    
  • the current encoding set is best seen with: xpra info | grep encoding=
  • to see statistics about which encodings are actually used: xpra info | egrep "last_used|total_frames|total_pixels"
  • video region detection can be seen with xpra info | grep region