xpra icon
Bug tracker and wiki

Opened 12 months ago

Closed 8 months ago

Last modified 2 months ago

#1699 closed enhancement (fixed)

pass hints to the encoder

Reported by: Antoine Martin Owned by: Antoine Martin
Priority: major Milestone: 2.3
Component: encodings Version: trunk
Keywords: Cc:

Description (last modified by Antoine Martin)

See ticket:1694#comment:4 and ticket:999#comment:24

We can add some hints to the window-model to try to help the picture encoders make better decisions.
For example:

  • "xterm" / "term" or "terminal" window classes should trigger "text" encoding, with higher quality, and avoid video
  • a media player (not sure about the best way to detect those), should trigger video

etc

The webp encoder (#1694) can use this hint, we can also use this hint for selecting video mode.

Attachments (8)

dowebp.sh (713 bytes) - added by J. Max Mena 8 months ago.
webp compression script
Webp Compression Data no charts.csv (3.0 KB) - added by J. Max Mena 8 months ago.
Webp Compression Data.xlsx (14.2 KB) - added by J. Max Mena 8 months ago.
1699 Video Chart.png (40.0 KB) - added by J. Max Mena 8 months ago.
Chart of data from Video.png
1699 Video Large Text Chart.png (34.1 KB) - added by J. Max Mena 8 months ago.
Chart of data from Video Large Text.png
Image and Text Chart.png (39.2 KB) - added by J. Max Mena 8 months ago.
Chart of data from Image and Text.png
Video.png (2.0 MB) - added by J. Max Mena 8 months ago.
The original screenshot of the video
dowebplossless.sh (697 bytes) - added by J. Max Mena 8 months ago.
The same as the other shell script, but lossless and I fixed a typo

Change History (21)

comment:1 Changed 12 months ago by Antoine Martin

Description: modified (diff)
Status: newassigned

comment:2 Changed 11 months ago by Antoine Martin

Owner: changed from Antoine Martin to J. Max Mena
Status: assignednew

Added in r17665, see commit message for details.
An xterm shows up as:

$ xpra info | grep content-type
window.1.content-type=text

A Firefox window correctly shows up as:

$ xpra info | grep content-type
window.1.content-type=browser

And the webp encoder now shows details on "preset" and "image hint":

webp.compress(XImageWrapper(BGRX: 0, 1317, 1513, 13), 1513, 13, False, browser) buf=0x7f072c8624f4
webp.compress config: lossless=0    , quality= 40, method=0, alpha=  0,  0,  0, preset=text    , image hint=graph

@maxmylyn: this is not finished yet, but already a good time to hand this over to you:

  • to make sure that your application identifiers are added so that we detect it as the correct "content-type" (ie: "browser" for most of your windows)
  • for you to run compression tests using webp on a variety of sample window screenshots to see which ones offer the best results (in terms of efficiency, compression, quality, etc..) - so that we can then tune the webp encoder accordingly

The coarse webp tunings using "preset" and "image hint" may not be enough for our needs - in which case we'll need to enhance the webp encoder logic, see libwebp/enc/config.c for what those preset values do.

Last edited 11 months ago by Antoine Martin (previous) (diff)

comment:3 Changed 11 months ago by J. Max Mena

Okay part one of comment:2 is done - as far as I can tell most major applications I've tried show up correctly (even a couple Electron applications). I'll write a script to test the webp encoder today or tomorrow depending on when I get to it.

comment:4 Changed 8 months ago by J. Max Mena

Okay, I've finally gotten around to the second part of comment:2 after much reading (mostly learning Bash, to be honest) I cobbled together a not very pretty bash script that will loop through all the presets and quality from 10 to 100 (inclusive) in increments of 10. It will save the outputted file and logs from said compression into subfolders with the name of the preset. If anyone would like to reproduce my metrics, I'll attach the .sh file. You can replace the cwebp......."image.png" lines with whatever image inputs you'd like - just make sure to change the log file name accordingly. It's not pretty nor is it configurable, but it's for a fairly single-use case. Yes it could have been prettier, yes it could have accepted inputs, but at this point I don't care. I'm sorry in advance.

(There's also a second bash script that just does lossless that I'll also attach)

I created three test images - two that are stills from the Documentary "Inside Job" - I chose this one because I watched it again last week on a whim, had the 1080p version available on my Emby Server, and it's also available free on archive.org or Vimeo - one is just a picture of some scenery from Iceland, and the other is the title text. The third image is a Wikipedia page with a picture on the top right. So we got a nice well-rounded three images to work with. They're titled "Video.png", "Image and Text.png", and "Video Large Text.png"

Also of use: https://developers.google.com/speed/webp/docs/cwebp and https://developers.google.com/speed/webp/docs/using

NOTE: -v adds more verbose logging of the encoding itself - including encoding time!

There's a lot more flags we can use - but I think implementing the presets and quality level is a great first step.

Anyways, I'll throw together a quick spreadsheet with relevant info.

Changed 8 months ago by J. Max Mena

Attachment: dowebp.sh added

webp compression script

comment:5 Changed 8 months ago by Antoine Martin

r18670 normalizes the webp quality lower.
Please use a wiki table for the data, and maybe also attach the sample pictures so we can reproduce the data later, and again after webp library updates. (the current version in Fedora 27 is 0.6.1)

comment:6 Changed 8 months ago by J. Max Mena

Okay I managed to comb through the data and put together a nice spreadsheet. Interestingly, the photo preset consistently has a notably higher encoding time than the others.

I'll attach a .csv of the data, and a .xlsx with charts (made it in Google Drive). Yes I know proprietary formats are bad, but the charts make it pretty obvious what is going on at a glance.

I'll write up a Wiki page shortly. I'll put it in wiki/WebpEncodingData.

Last edited 8 months ago by Antoine Martin (previous) (diff)

Changed 8 months ago by J. Max Mena

Changed 8 months ago by J. Max Mena

Attachment: Webp Compression Data.xlsx added

comment:7 Changed 8 months ago by J. Max Mena

Actually, it turns out you can export Google Sheet charts as a .png, so I'll post those instead. Also the original images as you asked.

Changed 8 months ago by J. Max Mena

Attachment: 1699 Video Chart.png added

Chart of data from Video.png

Changed 8 months ago by J. Max Mena

Chart of data from Video Large Text.png

Changed 8 months ago by J. Max Mena

Attachment: Image and Text Chart.png added

Chart of data from Image and Text.png

Changed 8 months ago by J. Max Mena

Attachment: Video.png added

The original screenshot of the video

Changed 8 months ago by J. Max Mena

Attachment: dowebplossless.sh added

The same as the other shell script, but lossless and I fixed a typo

comment:8 Changed 8 months ago by Antoine Martin

The outlier is the "photo" preset, but even then it doesn't take that much more time to encode (~10%).
So, which presets are useful for which types of content? Do they make a noticeable difference?
I also expect the lossless results to look quite different, judging from the API usage.

comment:9 Changed 8 months ago by J. Max Mena

Owner: changed from J. Max Mena to Antoine Martin

Okay I finally got around to visually inspecting the images. I'd attach them here, but all together they're ~40mb even compressed with gzip. It'd be better if you recreated them on your own using the bash script I attached.

Before I get into the subjective, I'll point out that the lossless usually performs similar to 50-70% quality for each preset. I'm honestly not sure if that's a good or a bad thing, but it's worth noting that lossless will perform better than higher quality lossy encodings.

On to the subjective:

From what I can see, looking at the worst cases, icon performs the worst in most cases. It makes text readable with no issue even on the lowest quality, so there's that.

For text-only content, text, unsurprisingly, looks the best. However for regular images, text looks subjectively worse than any of the other encodings - it has a hard time with color gradients - in particular trees look rather dreadful on the lower qualities.

Other than that - I can't really tell that much of a difference between the others - at a glance they all kind of look the same to me. I noticed that picture tens to get the colors right most of the time, so there's that - the others can have the lossy color chunks, but picture tends to get the colors more spot on while ignoring the lines.

From my subjective look - I'd say go with text for text only situations, and then with whatever performs the best for anything else.

comment:10 Changed 8 months ago by Antoine Martin

Owner: changed from Antoine Martin to J. Max Mena

Before I get into the subjective, I'll point out that the lossless usually performs similar to 50-70% quality for each preset.

"performance" - I assume that you mean compression latency only here? Do you have the resulting file size data?
We care about both compression latency and compression ratio.
The lossless mode has completely different performance characteristics: the "quality" setting is then used to drive compression effort, at least when using the API. (more or less what we usually call "speed" in xpra).
A lossless picture is worth spending a bit more time to compress since we can then skip the whole auto-refresh logic and the eventual lossless refresh that this may trigger later. But not if the resulting compressed data is too much bigger.
We currently switch to lossless mode when the compression request has a quality value higher than XPRA_WEBP_LOSSLESS_THRESHOLD which defaults to 75.

icon performs the worst in most cases

We only use the icon as default preset for very small rectangles, I've made this even stricter in r18926.

For text-only content, text, unsurprisingly, looks the best. However for regular images, text looks subjectively worse than any of the other encodings - it has a hard time with color gradients - in particular trees look rather dreadful on the lower qualities.

That's annoying, as I've made "text" the default. It's not as bad as it sounds because the lossless auto refresh should always eventually fix that (and video is handled separately) - should we change it? To what?

I'd say go with text for text only situations, and then with whatever performs the best for anything else.

Most of the time, unless we detect the application or if it supplies us with a hint, we don't know what we're dealing with so "text" seems like the conservative choice: making text unreadable is worse than color changes I would think?

Minor related fix in r18927.

comment:11 Changed 8 months ago by J. Max Mena

Owner: changed from J. Max Mena to Antoine Martin

"performance" - I assume that you mean compression latency only here? Do you have the resulting file size data?

We care about both compression latency and compression ratio.

Right I should specify, my bad.

By performance there I meant compression time - lossless compresses fairly quickly.

As for file size, they're all very similar - here's the table for the default encoding of the image "Image and Text.png":

Quality Filesize in Bytes
original 851739
10 144804
20 173878
30 198934
40 221090
50 237386
60 255202
70 271838
80 314068
90 397692
100 556888
lossless 281438

For all the presets, I see a very similar trend - a notable increase in file-size as quality increases. However, Lossless seems to hover around the 50-70 quality. Which mirrors the encoding times as reported in the spreadsheets I attached.


That's annoying, as I've made "text" the default. It's not as bad as it sounds because the lossless auto refresh should always eventually fix that (and video is handled separately) - should we change it? To what?

picture performs similarly and has an acceptable level of quality for text even at the lowest levels of quality. But,

Most of the time, unless we detect the application or if it supplies us with a hint, we don't know what we're dealing with so "text" seems like the conservative choice: making text unreadable is worse than color changes I would think?

I concur with you - leaving text as the default is a good conservative choice, at least as long as we aren't receiving hints as to the content type for a particular application.

comment:12 Changed 8 months ago by Antoine Martin

Resolution: fixed
Status: newclosed

OK then I think we're good, we're not using compression levels above 75%.
If I had more time, I would benchmark the lossless compression with different "speed" values, but this will do for now.

comment:13 Changed 2 months ago by Antoine Martin

Follow up: #1950

Note: See TracTickets for help on using tickets.