Xpra: Ticket #1655: Initial Xorg vs Xvfb testing data

Some important caveats:

I will also post the config and modified test_measure_perf.py files I'm running with, so that others can test this as well.

After posting this, I will copy all the config files to our "dedicated" test box in-office and re-run the test suite with a larger repetition.

Fri, 06 Oct 2017 17:23:05 GMT - J. Max Mena: description changed

Fri, 06 Oct 2017 17:24:59 GMT - J. Max Mena: attachment set

My config file

Fri, 06 Oct 2017 17:25:58 GMT - J. Max Mena: attachment set

modified test_measure_perf

Fri, 06 Oct 2017 17:30:07 GMT - J. Max Mena: attachment set

Xorg data

Fri, 06 Oct 2017 17:31:03 GMT - J. Max Mena: attachment set

Xorg log file

Fri, 06 Oct 2017 17:33:00 GMT - J. Max Mena: attachment set

Xvfb test data

Fri, 06 Oct 2017 17:34:06 GMT - J. Max Mena: attachment set

Xvfb log file

Sat, 07 Oct 2017 03:48:30 GMT - Antoine Martin: owner changed; cc deleted

Can you generate some graphs so we can visually inspect? Was there anything suspicious?

Mon, 09 Oct 2017 16:54:18 GMT - J. Max Mena:

I'll work on generating some graphs today.

In the meantime, here are the longer tests I ran on our test box.

Mon, 09 Oct 2017 16:54:48 GMT - J. Max Mena: attachment set

Sorry about the typo

Mon, 09 Oct 2017 16:55:02 GMT - J. Max Mena: attachment set

Xorg test data

Mon, 09 Oct 2017 16:55:36 GMT - J. Max Mena: attachment set

xorg logs

Mon, 09 Oct 2017 16:55:57 GMT - J. Max Mena: attachment set

xvfb logs

Mon, 09 Oct 2017 21:21:58 GMT - J. Max Mena:

I've now posted the html files that the test_measure_perf_charts.py outputs. I didn't get any errors, so it looks to be okay, but I'm trying to deal with getting JS added so I can actually view them.

Please note this is the first time that I've ran this utility. In the process of running it today I noticed that I had the automated tests outputting files with the wrong naming convention.

Of note:

The naming convention should be:

prefix_id_rep, but test_measure_perf.py takes the args rep id...they're reversed.

I also learned that feeding the test_measure_perf.py a different integer for rep doesn't actually run the tests that many times. So, I've updated my daily test script to reflect this so it'll actually run the tests twice now.

Of course, immediately after posting this, I realized the JS files are located in the same folder where the test_measure_perf_charts.py originally was. I copied the folder to the folder where I put the .html files, and reloaded the file and immediately got an error:

Error: Invalid dimensions for plot, width = 1220, height = 0

Well...back to editing the file some more.

Mon, 09 Oct 2017 22:48:34 GMT - J. Max Mena:

Okay I fixed the charts and they seem to work now. (removed rgb24 from the list of encodings)

Mon, 09 Oct 2017 22:49:15 GMT - J. Max Mena: attachment set

*Should* work now

Mon, 09 Oct 2017 22:55:02 GMT - J. Max Mena: attachment set

Fixed it so the headers actually say xvfb in this file

Mon, 09 Oct 2017 22:57:19 GMT - J. Max Mena:

Sorry for the email spam.

Anyways, I've uploaded the now working chart files. You'll need the "js" folder from src/tests/xpra/ folder. Oddly enough, my machine will only display the charts if I open the files in that folder where I've downloaded it on my machine. If I copy the .html and the js folder to somewhere else the charts don't show up and I get the error mentioned comment:4.

No idea why, web development is not my area of expertise.

Tue, 10 Oct 2017 05:31:55 GMT - Antoine Martin:

I'm not sure those ~1MB log do much good in this ticket, if not then please delete them.

The argument to test_measure_perf.py is just a reference which can be used for differentiating test runs. Especially useful if you're changing environment variables, or anything that is not recorded in the test result data.

Please try to combine xvfb and xdummy in the graphs so we can see them side by side.

Wed, 11 Oct 2017 16:56:02 GMT - J. Max Mena: attachment set

Updated charts that have the two data points next to each other

Wed, 11 Oct 2017 16:59:03 GMT - J. Max Mena:

I reworked the charts and now the Xdummy and Xvfb data points are overlaid next to each other. It looks like Xvfb is within 5-10% of Xdummy consistently.

I'm going to set up another test run on our test box with two runs of each test to get some more averaged data.

Wed, 18 Oct 2017 17:23:33 GMT - J. Max Mena: attachment set

I accidentally left the tests running for a couple more days, so we have about 5 runs worth of data.

Wed, 18 Oct 2017 17:26:56 GMT - J. Max Mena: owner changed

So I forgot to turn off the cron job that ran these tests for a couple days. So we now have approximately 5 runs worth of data that can be averaged over. I re-ran the charting file and got a new set of test data - this time with 5 points of data the charts average over to give us much better information about the performance.

I'll pass this to you to give the latest charts file a lookover. I guess close it when you're done. If there's any more work done on the Xvfb server, I have everything in place to run these tests some more.

Thu, 19 Oct 2017 06:03:37 GMT - Antoine Martin: owner changed

Having 5 runs worth of data is actually very useful, can you try to see what the variation is for each test then we can use this as reference in the future. Ideally using some form of Box Plot.

Wed, 25 Oct 2017 18:21:54 GMT - J. Max Mena: owner changed

I talked to Nick and he asked me to re-write this into a Feature Ticket and to give that to him. So, what would you like to do with this ticket?

Wed, 25 Oct 2017 18:23:44 GMT - Antoine Martin: status changed

So, what would you like to do with this ticket?

I'll take it and try to publish the charts somewhere.

Fri, 27 Oct 2017 12:19:26 GMT - Antoine Martin: owner, status changed

I finally moved the charts in place: http://xpra.org/stats/charts/Xdummy_vs_Xvfb.html.

There are non-negligible differences between the two options, in particular the "pixels/s sent" is an important metric and is markedly lower with Xvfb. The "min batch delay" is also consistently lower, the values are already very high in both cases to begin with.

There are definitely some things to investigate, we can't use any of the test data if it doesn't make sense overall:

Both of these issues could manifest themselves as a slow or unresponsive server. Is it an artifact of the test setup or a real issue?

The packet accounting is also missing. It would be better to have it, or just remove it from the charts.

Sun, 22 Apr 2018 06:37:54 GMT - Antoine Martin: priority changed

Sat, 09 Feb 2019 03:40:52 GMT - Antoine Martin: owner changed

Thu, 01 Aug 2019 11:59:12 GMT - Smo: owner changed

Tue, 13 Aug 2019 23:54:42 GMT - Smo: attachment set

New set of data comparing xorg vs xvfb

Tue, 13 Aug 2019 23:55:47 GMT - Smo: owner changed

Trying to get more data on this one. Here is some test data and the charts to go with it.

There is a good set of tests there with rgb and auto encoding with the python2 client.

Wed, 14 Aug 2019 06:02:08 GMT - Antoine Martin: owner changed

The gtkperf data seems to be missing for Xorg - did it crash? There are some outliers in there, ie: memscroller - which could be explained by test variance, more test data might smooth it out. The only surprise is how Xvfb does better at regions-per-second and encoding-pixels-per-second.

Mon, 19 Aug 2019 21:07:43 GMT - Smo: owner changed

Yup problems with gtkperf will try and figure out what is going on with it.

Do we need some much longer test runs or more iterations to try and smooth out the data?

Tue, 20 Aug 2019 05:43:18 GMT - Antoine Martin: owner changed

Do we need some much longer test runs or more iterations to try and smooth out the data?

Longer test runs are not usually helpful. The problem with more iterations is when one of the iteration failed and you don't notice, then the average is skewed. Maybe we should exclude the 2 problematic tests, and deal with those separately?

Thu, 12 Dec 2019 08:07:58 GMT - Antoine Martin: status changed; resolution set

Not heard back.

Sat, 23 Jan 2021 05:30:13 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1655