Changes between Version 18 and Version 19 of Testing

Timestamp:

04/23/15 18:36:33 (6 years ago)

Author:

Nick Centanni

Comment:

--

Testing

@@ -36 +36 @@
 {{{#!div class="box"
 == Automated performance and regression testing ==
-The [http://xpra.org/trac/browser/xpra/trunk/src/tests/xpra/test_measure_perf.py xpra.test_measure_perf] script can be used to run a variety of client applications within an xpra session (or optionally vnc) and get statistics on how well the encoding performed. The data is printed out at the end in CSV format which you can then import into any tool to compare results - you can find some examples generated using [http://www.sofastatistics.com/home.php sofastats] [http://xpra.org/stats/ here] and [/ticket/147#comment:11 here].
+The [http://xpra.org/trac/browser/xpra/trunk/src/tests/xpra/test_measure_perf.py xpra.test_measure_perf] script can be used to run a variety of client applications within an xpra session (or optionally vnc) and get statistics on how well the encoding performed. The data is printed out at the end in CSV format which you can then import into any tool to compare results - you can find some examples generated using [http://www.sofastatistics.com/home.php sofastats] [http://xpra.org/stats/ here] and [/ticket/147#comment:11 here]. There is a facility for generating charts directly from the CSV data, using a script that we now provide, described below.
+
 It can also be useful to redirect the test's output to a log file to verify that none of the tests failed with any exceptions/errors (looking for exception messages in the log afterwards).
-At the moment it does not have a command line interface, and all the options have to be edited directly in the source file.
-[[BR]]
+
+At the moment it does not have a command line interface, and all the options have to be edited directly. However, we have improved that process by splitting out the configuration data into a separate file: [http://xpra.org/trac/browser/xpra/trunk/src/tests/xpra/perf_config_default.py xpra.perf_config_default.py].
+
 Note: to take advantage of iptables packet accounting (mostly for comparing with VNC which does not provide this metric), follow the error message and setup iptables rules to match the port being used in the tests, ie: by default:
 {{{
@@ -49 +51 @@
 * disable cron jobs or any other scheduled work (systemd makes this a little harder)
 * etc..
+
+To create multiple output files which can be used to generate charts, using [http://xpra.org/trac/browser/xpra/trunk/src/tests/xpra/test_measure_perf_charts.py xpra.test_measure_perf_charts]:
+                                                                                        * Build a config class by taking a copy from [http://xpra.org/trac/browser/xpra/trunk/src/tests/xpra/perf_config_default.py xpra.perf_config_default.py], then making changes as necessary.
+* Determine the values of the following variables: prefix: (a string to identify the data set), id: (a string to identify the variable that the data set is testing, for example '14' because we're testing xpra v14 in this data set), repetitions: the number of times you want to run the tests.                                                                                                                                                                                                    
+* The data file names you will produce will then be in the format: prefix_id_rep.csv.                                                                                                                                                                                                                                              
+* With this information in hand you can now create a script that will run the tests.
+
+For example:                                                                                                                                                                                                                                                      
+                                                                                                                                                                                                                                                                   
+{{{
+./test_measure_perf.py all_tests_40 ./data/all_tests_40_14_1.csv 1 14 > ./data//all_tests_40_14_1.log                                                                                                                                                             
+./test_measure_perf.py all_tests_40 ./data/all_tests_40_14_2.csv 2 14 > ./data//all_tests_40_14_2.log                                                                                                                                                             
+}}}
+                                                                                                                                                                                                                                                                   
+In the above example, test_measure_perf is run twice, using a config class named "all_tests_40.py", and outputting the results to data files using the prefix "all_tests_40", for version 14.
+                                                                                                                                                                         
+The additional arguments "1 14", "2 14" are custom paramaters which will be written to the "Custom Params" column in the corresponding data files.                                                                                                                                                                                                                                  
+                                                                                                                                                                                                                                                                   
+The "1", "2" in the file names or in the parameters, refer to the corresponding repetition of the tests.
+                                                                                                                                                
+
 Please also see:
 }}}