Debugging

Some areas of the system may have dedicated pages which may be useful for debugging, in combination with the information contained here:

• Sound
• Keyboard
• Window Refresh

First Things

Always try to narrow it down as much as possible, by turning off as many features as possible (clipboard, etc) and trying various encodings.

Debug Builds

If a debug build is available, please use it. It may provide more useful diagnostic messages. You can generate debug builds by calling setup.py with the --with-debug flag (or by setting DEBUG=1 in the MS Windows BAT file)

---

Via Logging

Sometimes the problem is so obvious that you will simply get the error directly from the command line or in the server log file. You may also want to run the server with the --no-daemon switch to more easily keep an eye on its log file and status.

Otherwise, a good first step is to start xpra from the command line and add "-d all" to it. The amount of data logged can be overwhelming, so make sure you log it or redirect it to a file that you can then grep to find and extract the data you are looking for.

Some modules do not log by default, either to reduce the amount of logging or to avoid the overhead of logging in critical paths. Logging can generally be enabled using environment variables, you can find the list of such runtime configuration options with:

egrep -re "XPRA_.*DEBUG|XPRA_.*LOG" src/xpra | grep 'os.environ'
egrep -re "XPRA_.*DEBUG|XPRA_.*LOG" src/wimpiggy | grep 'os.environ'

At time of writing this shows the following options:

• XPRA_KEYBOARD_DEBUG
• XPRA_DAMAGE_DEBUG (damage events show which regions have been repainted)
• XPRA_DEBUG_SOUND
• XPRA_X11_LOG (enables logging for X11 events)
• XPRA_X11_DEBUG (logs all X11 events through going through our loop - this is very verbose)

Other Environment Variables

There are other environment variables which can be used to tune the behaviour of the system and may be used for debugging or testing, you can obtain the full list with:

grep 'os.environ' src/xpra
grep 'os.environ' src/wimpiggy

Notably:

• some x264 attributes and thresholds: XPRA_X264_*_PROFILE, XPRA_X264_*_MIN_QUALITY, XPRA_X264_*_QUALITY
• lossless damage threshold values (number of pixels): XPRA_MAX_NONVIDEO_PIXELS and MAX_NONVIDEO_OR_INITIAL_PIXELS
• sound test (use fake sound source on/off): XPRA_SOUND_TEST
• use cython maths (on/off): XPRA_CYTHON_MATH
• dump latency debug information to log file (on/off): XPRA_DEBUG_LATENCY
• try to schedule other threads in network and damage loops (on/off - see #181): XPRA_YIELD
• check that we obey our network call threading rules (on/off): XPRA_CHECK_THREAD
• use PIL for parsing png and jpeg packets (on/off): XPRA_USE_PIL

Please refer to the actual code for details.

---

Network Traffic

Seeing what is being exchanged between the client and server can be useful at times, an easy way to achieve this is to use tcp mode (ie: --bind-tcp=0.0.0.0:10000 option) without packet compression (-z 0 option) then the packets can be looked at with your favourite packet inspection tool, ie:

ngrep -p 10000

The packet type is the first thing in each packet and it is a simple string which makes it easy to observe traffic. With version 0.9 onwards, it is best to set XPRA_USE_ALIASES=0 to ensure that the packets will use plain-text headers rather than numeric aliases.

xtrace

(X11 traffic)

​xtrace allows us to monitor the traffic between the xpra server and the Xvfb (X11 server), or between the client and its display server (Posix only).

• server setup: because the server normally launches its own xvfb, we need to tell it to use an existing one and we interpose xtrace in between:
  • start your xvfb:

    Xvfb ... :10
    

  • start xtrace forwarding from :10 to :11:

    xtrace -k -d :10 -D :11
    

  • start an xpra server on the existing :11 display:

    xpra start --use-display :11
    

• client setup: simply run the client via xtrace:

  xtrace xpra attach ..
  

---

GDB

When dealing with crashes ("core dumped"), the best way to debug is to fire gdb.

Getting a Backtrace

xpra start ...
# or xpra attach ...
ps -ef | grep xpra
gdb python $PID_OF_XPRA_PROCESS_TO_DEBUG
# wait for it to load all the debug symbols
(gdb) continue

Then once you get a crash, gdb should show you its prompt again and you can extract the python stacktrace with py-bt and the full stacktrace with bt. Having both is useful. Note: installing the required "debug" symbol packages for your distribution is out of scope, please refer to your vendor's package manager for details (ie: ​debian and ​yum debuginfo-install).

X11 errors

Occasionally you may encounter an X11 crash (BadDrawable, BadWindow, etc). These are more tricky. You may need to set a breakpoint:

(gdb) break gdk_x_error

(or _XError if gdk_x_error is not found) And get a backtrace from there when gdb hits it.

Another useful environment variable is XPRA_X11_DEBUG_EVENTS, it allows you to specify a CSV list of X11 events to log specifically, ie:

XPRA_X11_DEBUG_EVENTS="PropertyNotify,MotionNotify" xpra ...

The full list of event names can be obtained by using an invalid value, which will trigger a warning message, or by looking at the source.

Unfortunately, because of the asynchronous nature of X11 calls, the error may generate this crash some time after the event that caused it. In this case, we may want to force all X11 calls to be synchronized (which will hurt performance and may even hide the bugs - beware of heisenbugs!), trace all X11 calls, etc. this modified error.py allows you to do just that (see the constants at the top).

---

Venerable Print Statements

When all else fails, or just when appropriate, sprinkling some print statements around the critical sections of code is often the best way to get a clearer picture of what is really going on..

---

Memory Leaks

In C code

If the memory leak is one of the Cython parts or in a C library, use regular tools, like ​valgrind. When using valgrind, the connection timeout may be exceeded because the client will be slowed down. Increasing this timeout will be necessary to be able to use the client with Valgrind. With r4861 onwards, you can do this using the XPRA_SOCKET_TIMEOUT env var:

XPRA_SOCKET_TIMEOUT=30 xpra attach ...

In python code

Needs documenting better..

• ​objgraph is a good place to start
• ​simple leak code
• ​Debug with garbage collection recipe

---

Debugging in Windows

Using Microsoft Visual Studio Express

You will need to create a new, empty project (File->New->Project), select "General" and "Empty Project". Visual Studio will require a name for the project and will ask for a place to save it. This project will contain nothing but basic information to start Xpra with the debugger, so save it somewhere for later use.

You will be placed in the tree, containing the "Solution" and the project you've created. Right click on the project in the tree and select "properties".

Go to "Debugging", in fill in the fields to attach your Xpra client - example below.

Then you can do F5 to start the application with the debugger.

Alternatively, if you set "Attach" to yes, the debugger will ask you to what running process you want to attach upon pressing F5.

---

OpenGL

Whenever you encounter problems with client-side OpenGL rendering, for example with OpenGL error messages, crashes, or visual artifacts, there are some specific steps to take to investigate. The *Apitrace* tool will get a dump of the OpenGL command stream, allowing it to be sent to the developers for replaying and investigation.

Installation of Apitrace

On Linux

Download ​Apitrace, build it according to the documentation: ​https://github.com/apitrace/apitrace/blob/master/INSTALL.markdown

On Windows

You can use pre-built binaries found at ​http://people.freedesktop.org/~jrfonseca/apitrace/ Try to always use the latest version and fallback to an older one if it doesn't work.

Trace generation

This trace will enable you to replay the actions of the client without having to re-run it; giving you additional error tracing, ability to send the trace to another person, and ability to replay the exact same steps every time.

Use as follows:

$ apitrace trace xpra attach <your regular commandline>

• Do whatever is needed to reproduce the issue you are experiencing. Beware: Apitrace is keeping a copy of everything you do in RAM, it's going to be huge, so don't run the session for too long, and focus on reproducing one issue at a time.
• Exit Xpra when done

  $ qapitrace
  

File -> load the trace Then go to Trace -> Replay and check both boxes

Apitrace will replay your trace and report in the bottom panel any errors it found, with additional explanation as compared to what the GL driver typically gives.

Be ready to send the .trace file to the developers for further investigation.