xpra icon
Bug tracker and wiki

Opened 5 months ago

Closed 10 days ago

Last modified 3 days ago

#2231 closed enhancement (fixed)

automated tests should be able to test html5 client

Reported by: Antoine Martin Owned by: Antoine Martin
Priority: critical Milestone: 3.0
Component: tests Version: 2.5.x
Keywords: Cc:

Description

And record the client type so we can compare python vs html5.

Attachments (4)

comparebrowser.tar.gz (141.4 KB) - added by Smo 2 weeks ago.
Compare firefox and chrome with charts
compareclients.tar.gz (142.2 KB) - added by Smo 11 days ago.
compares all clients
avg-batch-delay.png (28.5 KB) - added by Antoine Martin 10 days ago.
average batch delay
regions-per-second.png (29.2 KB) - added by Antoine Martin 10 days ago.
regions per second

Download all attachments as: .zip

Change History (17)

comment:1 Changed 5 months ago by Antoine Martin

Priority: majorcritical
Status: newassigned

Blocker for #2232.

comment:2 Changed 5 months ago by Antoine Martin

Blocked by #2251

comment:3 Changed 4 months ago by Antoine Martin

Owner: changed from Antoine Martin to Jonathan Anthony
Status: assignednew

Working OK now that #2251 is fixed + minor fixes from r22517.

With one important caveat: the browser session must not be resumed from an earlier one as this would interfere with the testing, so one has to use a profile specifically setup for this testing.
Either a brand new user, or forcing a new profile:

  • firefox now uses firefox -P Test: so you must create a "Test" profile
  • google-chrome now uses google-chrome --user-data-dir=~/Downloads/TEMP, so you must run mkdir -p ~/Downloads/TEMP once.

Notes:

  • more browsers could be added, @encodedEntropy: do you have a standalone one I could use for testing?
  • we could also add a setting for testing via https vs http, but again this would require manual steps for accepting the certificates - probably not worth the hassle.
  • when running the tests with a browser as client, it doesn't make sense to have XPRA_TEST_ENCODINGS and many of the other options, so r shortcuts that out. Which means that there are fewer browser tests than python client tests sets. Adjust the test options accordingly.
  • do we also want to set the window size so that the test setup is more reproducible? Firefox has an option, google-chrome does not. (so it would require an xdotool hack)
  • I have seen firefox occasionally failing to startup and getting an error page instead of the html5 client - not sure why that is yet. It means the corresponding test data is garbage.. Another time, it came up with a "do you want to refresh" dialog... PITA

@encodedEntropy: I think there's enough there to hand over to you, let's get some data and iterate.

comment:4 Changed 3 weeks ago by Smo

Owner: changed from Jonathan Anthony to Smo

comment:5 Changed 2 weeks ago by Smo

Browsers are working great with automated tests. The only feedback I have is what you have said already is that we may want to set the window size but in my tests my window always starts the same size.

Firefox window seems to be a good size but the chrome one sometimes cuts off the window of the application being tested.

Not sure if this matters all that much since it is the same size every time.

comment:6 Changed 2 weeks ago by Smo

Owner: changed from Smo to Antoine Martin

comment:7 Changed 2 weeks ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

Can you post some data?

comment:8 Changed 2 weeks ago by Smo

Owner: changed from Smo to Antoine Martin

I'm attaching some perf tests comparing firefox and chrome. I wanted to compare them with the python2 client as well but the data doesn't seem to line up.

Changed 2 weeks ago by Smo

Attachment: comparebrowser.tar.gz added

Compare firefox and chrome with charts

comment:9 Changed 2 weeks ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

Interesting data points:

  • memscroller is killing the latency (#2382): maximum goes up to 7 seconds with chrome!
  • memscroller aside, Firefox is doing better than chrome in almost all key metrics: batch delay, damage latency, quality, decoding speed, regions and pixels per second, but it does use quite a bit more cpu client side

Can you post some comparison of python vs html5? On auto, so we can see the gap in performance.
(then I think we can close this ticket)

Last edited 2 weeks ago by Antoine Martin (previous) (diff)

Changed 11 days ago by Smo

Attachment: compareclients.tar.gz added

compares all clients

comment:10 Changed 11 days ago by Smo

Attached is a comparison of all the 3 clients. I had the data before but I had to fix the script for the charts to compare them.

Also changed the css a bit to be able to read things.

comment:11 Changed 11 days ago by Smo

Owner: changed from Smo to Antoine Martin

Changed 10 days ago by Antoine Martin

Attachment: avg-batch-delay.png added

average batch delay

Changed 10 days ago by Antoine Martin

Attachment: regions-per-second.png added

regions per second

comment:12 Changed 10 days ago by Antoine Martin

Resolution: fixed
Status: newclosed

Also changed the css a bit to be able to read things.

Nice, thanks!

First thing to mention is that the decoding-pixels-per-second values for the html5 client are wrong: javascript is not decoding 50GPixels per second! (nothing can - I will try to fix it)

As expected, the python client is way ahead of the html5 client:

  • best network send speed of 20MB/s vs 3MB/s
  • max damage latency: ~50ms vs 7000! (because of memscroller: #2382)
  • max batch delay at 22ms, vs over 300.. (memscroller again - but overall quite bad with Chrome)
  • the key metrics: more pixels, much better batch delay, more regions-per-second:

average batch delay
regions per second

That said, the html5 client does better than I thought it would do. (it is sacrificing framerate and quality in the pure-video glxgears test - but managing OK)
Also interesting to see that Firefox does quite a bit better than Chrome overall.

comment:13 Changed 3 days ago by Antoine Martin

Fix for the decoding latency reported by the html5 client: r23551.

Note: See TracTickets for help on using tickets.