xpra icon
Bug tracker and wiki

Opened 5 years ago

Closed 5 years ago

#596 closed task (fixed)

better auto refresh: handle video regions, don't refresh unnecessarily

Reported by: Antoine Martin Owned by: J. Max Mena
Priority: major Milestone: 0.14
Component: encodings Version: 0.12.x
Keywords: Cc:

Description

Split from #592.

The problems:

  • the video region updates a lot by definition (many fps), so we want to exclude it from auto-refresh (done in r6728) but only for as long as it is updating rapidly: after that it should get refreshed (not done). Maybe we can keep an eye on the video region and trigger a manual partial refresh when it becomes inactive.
  • when we have a video region (say youtube), some other things on the page will also refresh (annoying animated gifs, mouseover, etc): we don't want those to cause a full page lossless refresh. The refresh would be costly (at 1080p in PNG, this can take 200ms and cause the video to stutter, causing problems with the heuristics too). We should keep track of the regions which have been damaged with lossy updates, and update those only rather than the full page. (we can probably just re-use the existing code which decides if a full window update is better than multiple regions, using the same codepath)
  • the quality and speed settings are mostly relevant to the video region (most of the statistics will come from there), when we use a non-video encoding for other regions we end up using a lower quality setting than we should, causing the problems above where we then have to refresh those regions. Effectively painting them twice, costing more bandwidth, and causing visual degradation. Potential solutions:
    • normalize the quality setting? (so that jpeg at 50% is visually closer to h264 at 50%)
    • increase the quality of other updates when a video region exists?

Attachments (2)

596minquality0quality1.png (352.9 KB) - added by J. Max Mena 5 years ago.
596minquality30quality0.png (395.1 KB) - added by J. Max Mena 5 years ago.

Download all attachments as: .zip

Change History (8)

comment:1 Changed 5 years ago by Antoine Martin

Much improved in:

  • r6837 and r6838: keep track of regions needing the refresh (and more)
  • r6840: dynamic lossless threshold

Still TODO:

  • losslessly refresh the video region when it stops updating (visible when switching tabs or scrolling)
  • profiling
Last edited 5 years ago by Antoine Martin (previous) (diff)

comment:2 Changed 5 years ago by Antoine Martin

Owner: changed from Antoine Martin to alas
  • fix for video regions not auto-lossless-refreshing when they stop updating in r6848
  • boost in quality and speed for non-video areas in r6849

A good way to see this in action, which also helps to understand what it does is to start with min-quality at 0 and quality very very low. From the command line or:

xpra control :10 min-quality 0
xpra control :10 quality 1

Then later you can go back to normal with:

xpra control :10 quality 0
xpra control :10 min-quality 30
Last edited 5 years ago by Antoine Martin (previous) (diff)

comment:3 Changed 5 years ago by J. Max Mena

Retested with 14.0 r6853 Windows 7 64-bit client against a 14.0 r6853 Fedora 20 server:

  • Setting min-quality to 0, and quality 1:
    • Caused the whole webpage to become blurry, and stay blurry
  • Setting min-quality to 30, and quality to 0:
    • Again, caused the whole webpage to become blurry and stay blurry

As I mentioned in #410, with the video running, the whole webpage becomes blurry as well and stays that way. It appears that the auto-refresh for the rest of the webpage is not kicking in, and staying in a low-quality state, which could go into an even worse state with the min-quality lowered.

I'll attach some screenshots.

Changed 5 years ago by J. Max Mena

Attachment: 596minquality0quality1.png added

Changed 5 years ago by J. Max Mena

Attachment: 596minquality30quality0.png added

comment:4 Changed 5 years ago by alas

  • Oddly, testing with 0.14.0 r6853 Windows client against r6853 fedora 19 server... using google-chrome with min-quality 0 and quality 6, the whole page generally remains blurry, but with firefox or lazarus the non-video section (mostly) remains clear while the video section becomes blurry as expected.
  • Setting quality 0 and min-quality 30 makes youtube sharp with firefox or lazarus, but chrome is still very blurry.
  • With -d refresh and a firefox window - while video is playing there are an expected flurry of h264 refreshes; with occasional rgb24 refresh checks (unless are changes, such as rolling the mouse over the window).
2014-06-20 16:46:27,823 auto refresh: rgb24 screen update (quality=100), nothing to do (region=rectangle[50, 479, 443, 20], refresh regions=[])
2014-06-20 16:46:27,868 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:27,880 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:27,919 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:27,963 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,002 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,028 auto refresh: rgb24 screen update (quality=100), nothing to do (region=rectangle[50, 479, 444, 39], refresh regions=[])
2014-06-20 16:46:28,050 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,085 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,130 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,163 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,199 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,229 auto refresh: rgb24 screen update (quality=100), nothing to do (region=rectangle[50, 479, 444, 20], refresh regions=[])
2014-06-20 16:46:28,250 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,285 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,326 auto refresh: h264 screen update (quality= 70), list of refresh regions unchanged (region=rectangle[50, 131, 640, 390], refresh regions=[])
2014-06-20 16:46:28,331 auto refresh: rgb24 screen update (quality=100), nothing to do (region=rectangle[688, 484, 2, 4], refresh regions=[])
  • However, with a -d refresh and a chrome window - while video is playing there seem to be a mix of rgb, jpeg and h264 refresh screen updates and png auto refresh packets being sent:
2014-06-20 16:17:37,311 schedule_auto_refresh: elapsed time 379 with target=297, refreshing now
2014-06-20 16:17:37,311 timer_full_refresh() after 0ms, regions=[R[0, 0, 1450, 894]]
2014-06-20 16:17:37,459 auto-refresh png packet sent
2014-06-20 16:17:39,523 auto refresh: jpeg screen update (quality= 34), scheduling refresh (region=rectangle[0, 0, 213, 26], refresh regions=[R[0, 0, 213, 26]])
2014-06-20 16:17:39,622 auto refresh: jpeg screen update (quality= 34), list of refresh regions unchanged (region=rectangle[0, 0, 213, 26], refresh regions=[R[0, 0, 213, 26]])
2014-06-20 16:17:39,726 auto refresh: jpeg screen update (quality= 35), list of refresh regions unchanged (region=rectangle[0, 0, 213, 26], refresh regions=[R[0, 0, 213, 26]])
2014-06-20 16:17:39,816 auto refresh: h264 screen update (quality= 37), keeping existing timer (region=rectangle[0, 0, 1450, 894], refresh regions=[R[0, 0, 1450, 894]])
2014-06-20 16:17:39,818 auto refresh: rgb24 screen update (quality=100), removed rectangle from regions (region=rectangle[1450, 0, 1, 895], refresh regions=[R[0, 0, 1450, 894]])
2014-06-20 16:17:39,818 auto refresh: rgb24 screen update (quality=100), removed rectangle from regions (region=rectangle[0, 894, 1451, 1], refresh regions=[R[0, 0, 1450, 894]])
2014-06-20 16:17:39,875 schedule_auto_refresh: elapsed time 352 with target=297, refreshing now
2014-06-20 16:17:39,876 timer_full_refresh() after 0ms, regions=[R[0, 0, 1450, 894]]
2014-06-20 16:17:40,025 auto-refresh png packet sent
2014-06-20 16:17:40,025 auto refresh: jpeg screen update (quality= 35), scheduling refresh (region=rectangle[0, 0, 213, 26], refresh regions=[R[0, 0, 213, 26]])
2014-06-20 16:17:40,056 auto refresh: jpeg screen update (quality= 35), list of refresh regions unchanged (region=rectangle[0, 0, 213, 26], refresh regions=[R[0, 0, 213, 26]])
2014-06-20 16:17:40,222 auto refresh: jpeg screen update (quality= 35), list of refresh regions unchanged (region=rectangle[0, 0, 213, 26], refresh regions=[R[0, 0, 213, 26]])
2014-06-20 16:17:40,289 schedule_auto_refresh: elapsed time 263 with target=50, refreshing now
2014-06-20 16:17:40,289 timer_full_refresh() after 0ms, regions=[R[0, 0, 213, 26]]
2014-06-20 16:17:40,292 auto-refresh png packet sent
2014-06-20 16:17:40,310 auto refresh: jpeg screen update (quality= 35), scheduling refresh (region=rectangle[0, 0, 213, 26], refresh regions=[R[0, 0, 213, 26]])
2014-06-20 16:17:40,410 auto refresh: jpeg screen update (quality= 35), list of refresh regions unchanged (region=rectangle[0, 0, 213, 26], refresh regions=[R[0, 0, 213, 26]])
2014-06-20 16:17:40,510 auto refresh: jpeg screen update (quality= 35), list of refresh regions unchanged (region=rectangle[0, 0, 213, 26], refresh regions=[R[0, 0, 213, 26]])
2014-06-20 16:17:40,572 schedule_auto_refresh: elapsed time 262 with target=50, refreshing now
2014-06-20 16:17:40,573 timer_full_refresh() after 0ms, regions=[R[0, 0, 213, 26]]
2014-06-20 16:17:40,575 auto-refresh png packet sent
2014-06-20 16:17:48,633 auto refresh: h264 screen update (quality= 37), scheduling refresh (region=rectangle[0, 0, 1450, 894], refresh regions=[R[0, 0, 1450, 894]])

comment:5 Changed 5 years ago by Antoine Martin

Owner: changed from alas to J. Max Mena
  • firefox detects the video correctly: we get many h264 updates for the video subregion (443x20 at 50x479). The other areas update as lossless (rgb24 shown in log). What we want to know is if the video region gets a lossless refresh soon after the video stops playing (pause the video or change tab). And if the non-video regions get lossless updates whilst the video is playing.
  • chrome does not seem to detect the video (this belongs in #410).


This ticket is about the quality setting when refreshing the non-video regions. It is only relevant when video regions are detected. Also, note that setting the quality very low is useful for inspecting visually what is happening, to see when things get a lossless refresh if we do end up sending a low quality update, but unless it also happens with the correct settings, it is not necessarily a bug.

comment:6 Changed 5 years ago by alas

Resolution: fixed
Status: newclosed

With the issues seeming to be related to #410 rather than this ticket, everything else seems to be working as expected. Adjusting quality &/or min-quality primarily affects the video regions when there is video playing, and the pages overall if there's no region requiring especial attention.

Closing.

Note: See TracTickets for help on using tickets.