xpra icon
Bug tracker and wiki

Opened 13 days ago

Last modified 12 hours ago

#1655 new task

Initial Xorg vs Xvfb testing data

Reported by: J. Max Mena Owned by: J. Max Mena
Priority: major Milestone: 2.2
Component: tests Version: trunk
Keywords: Cc:

Description (last modified by J. Max Mena)

Some important caveats:

  • Only 2 runs through each test suite
  • Machine in question is very low end - quad core Atom + some Nvidia Ion "GPU"
  • VirtualGL disabled - said GPU doesn't play nicely with VirtualGL (performance is abysmal for OpenGL apps)
  • simulate_console_user.py is unpatched and as such failed its test

I will also post the config and modified test_measure_perf.py files I'm running with, so that others can test this as well.


After posting this, I will copy all the config files to our "dedicated" test box in-office and re-run the test suite with a larger repetition.

Attachments (14)

config_xvfb.py (12.0 KB) - added by J. Max Mena 13 days ago.
My config file
test_measure_perf_xvfb.py (46.7 KB) - added by J. Max Mena 13 days ago.
modified test_measure_perf
xorg_2_2_201710042230.csv (1.4 KB) - added by J. Max Mena 13 days ago.
Xorg data
xorg_2_2_201710042230.log (84.4 KB) - added by J. Max Mena 13 days ago.
Xorg log file
xvfb_2_2_201710041534.csv (12.3 KB) - added by J. Max Mena 13 days ago.
Xvfb test data
xvfb_2_2_201710041534.log (108.6 KB) - added by J. Max Mena 13 days ago.
Xvfb log file
ixvfb_201710061100_4_2.csv (46.5 KB) - added by J. Max Mena 10 days ago.
Sorry about the typo
xorg_201710061100_4_2.csv (46.5 KB) - added by J. Max Mena 10 days ago.
Xorg test data
xorg_201710061100_4_2.log (1.0 MB) - added by J. Max Mena 10 days ago.
xorg logs
xvfb_201710061100_4_2.log (994.3 KB) - added by J. Max Mena 10 days ago.
xvfb logs
charts_xorg.html (128.9 KB) - added by J. Max Mena 10 days ago.
*Should* work now
charts_xvfb.html (128.9 KB) - added by J. Max Mena 10 days ago.
Fixed it so the headers actually say xvfb in this file
charts.html (198.2 KB) - added by J. Max Mena 8 days ago.
Updated charts that have the two data points next to each other
charts_5.html (198.2 KB) - added by J. Max Mena 25 hours ago.
I accidentally left the tests running for a couple more days, so we have about 5 runs worth of data.

Change History (24)

comment:1 Changed 13 days ago by J. Max Mena

Description: modified (diff)

Changed 13 days ago by J. Max Mena

Attachment: config_xvfb.py added

My config file

Changed 13 days ago by J. Max Mena

Attachment: test_measure_perf_xvfb.py added

modified test_measure_perf

Changed 13 days ago by J. Max Mena

Attachment: xorg_2_2_201710042230.csv added

Xorg data

Changed 13 days ago by J. Max Mena

Attachment: xorg_2_2_201710042230.log added

Xorg log file

Changed 13 days ago by J. Max Mena

Attachment: xvfb_2_2_201710041534.csv added

Xvfb test data

Changed 13 days ago by J. Max Mena

Attachment: xvfb_2_2_201710041534.log added

Xvfb log file

comment:2 Changed 13 days ago by Antoine Martin

Cc: smo@… sbennett@… afarr@… removed
Owner: changed from Antoine Martin to J. Max Mena

Can you generate some graphs so we can visually inspect? Was there anything suspicious?

comment:3 Changed 10 days ago by J. Max Mena

I'll work on generating some graphs today.

In the meantime, here are the longer tests I ran on our test box.

Changed 10 days ago by J. Max Mena

Attachment: ixvfb_201710061100_4_2.csv added

Sorry about the typo

Changed 10 days ago by J. Max Mena

Attachment: xorg_201710061100_4_2.csv added

Xorg test data

Changed 10 days ago by J. Max Mena

Attachment: xorg_201710061100_4_2.log added

xorg logs

Changed 10 days ago by J. Max Mena

Attachment: xvfb_201710061100_4_2.log added

xvfb logs

comment:4 Changed 10 days ago by J. Max Mena

I've now posted the html files that the test_measure_perf_charts.py outputs. I didn't get any errors, so it looks to be okay, but I'm trying to deal with getting JS added so I can actually view them.

Please note this is the first time that I've ran this utility. In the process of running it today I noticed that I had the automated tests outputting files with the wrong naming convention.

Of note:

The naming convention should be:

prefix_id_rep, but test_measure_perf.py takes the args rep id...they're reversed.

I also learned that feeding the test_measure_perf.py a different integer for rep doesn't actually run the tests that many times. So, I've updated my daily test script to reflect this so it'll actually run the tests twice now.


Of course, immediately after posting this, I realized the JS files are located in the same folder where the test_measure_perf_charts.py originally was. I copied the folder to the folder where I put the .html files, and reloaded the file and immediately got an error:

Error: Invalid dimensions for plot, width = 1220, height = 0

Well...back to editing the file some more.

Last edited 10 days ago by J. Max Mena (previous) (diff)

comment:5 Changed 10 days ago by J. Max Mena

Okay I fixed the charts and they seem to work now. (removed rgb24 from the list of encodings)

Changed 10 days ago by J. Max Mena

Attachment: charts_xorg.html added

*Should* work now

Changed 10 days ago by J. Max Mena

Attachment: charts_xvfb.html added

Fixed it so the headers actually say xvfb in this file

comment:6 Changed 10 days ago by J. Max Mena

Sorry for the email spam.

Anyways, I've uploaded the now working chart files. You'll need the "js" folder from src/tests/xpra/ folder. Oddly enough, my machine will only display the charts if I open the files in that folder where I've downloaded it on my machine. If I copy the .html and the js folder to somewhere else the charts don't show up and I get the error mentioned comment:4.

No idea why, web development is not my area of expertise.

comment:7 Changed 10 days ago by Antoine Martin

I'm not sure those ~1MB log do much good in this ticket, if not then please delete them.

The argument to test_measure_perf.py is just a reference which can be used for differentiating test runs. Especially useful if you're changing environment variables, or anything that is not recorded in the test result data.

Please try to combine xvfb and xdummy in the graphs so we can see them side by side.

Changed 8 days ago by J. Max Mena

Attachment: charts.html added

Updated charts that have the two data points next to each other

comment:8 Changed 8 days ago by J. Max Mena

I reworked the charts and now the Xdummy and Xvfb data points are overlaid next to each other. It looks like Xvfb is within 5-10% of Xdummy consistently.

I'm going to set up another test run on our test box with two runs of each test to get some more averaged data.

Changed 25 hours ago by J. Max Mena

Attachment: charts_5.html added

I accidentally left the tests running for a couple more days, so we have about 5 runs worth of data.

comment:9 Changed 25 hours ago by J. Max Mena

Owner: changed from J. Max Mena to Antoine Martin

So I forgot to turn off the cron job that ran these tests for a couple days. So we now have approximately 5 runs worth of data that can be averaged over. I re-ran the charting file and got a new set of test data - this time with 5 points of data the charts average over to give us much better information about the performance.

I'll pass this to you to give the latest charts file a lookover. I guess close it when you're done. If there's any more work done on the Xvfb server, I have everything in place to run these tests some more.

comment:10 Changed 12 hours ago by Antoine Martin

Owner: changed from Antoine Martin to J. Max Mena

Having 5 runs worth of data is actually very useful, can you try to see what the variation is for each test then we can use this as reference in the future.
Ideally using some form of Box Plot.

Note: See TracTickets for help on using tickets.