xpra icon
Bug tracker and wiki

Version 34 (modified by Antoine Martin, 6 years ago) (diff)

--

Debugging


Some areas of the system may have dedicated pages which may be useful for debugging, in combination with the information contained here:


First Things

Always try to narrow it down as much as possible, by turning off as many features as possible (clipboard, etc) and trying various encodings. Try another operating system, try a different version, etc

Debug Builds

If a debug build is available, please try to use it. It may provide more useful diagnostic messages. You can generate debug builds by calling setup.py with the --with-debug flag (or by setting DEBUG=1 in the MS Windows BAT file)

Via Logging

Sometimes the problem is so obvious that you will simply get the error directly from the command line or in the server log file.

Otherwise, a good first step is to start xpra from the command line and add "-d CATEGORY" to it. The list of categories can be found with xpra -d help. The amount of data logged can be overwhelming, especially with the pseudo-category all, so make sure you log it or redirect it to a file that you can then grep to find and extract the data you are looking for.


Each category can also be enabled using environment variable, which can be useful if you cannot modify the command line, or if the logging should happen very early on, or if you aren't calling the code from its normal wrappers. Use: XPRA_CATEGORY_DEBUG=1 xpra ... to enable debug logging for your chosen CATEGORY.


It is also possible to enable and disable debug logging at runtime using:

xpra control :DISPLAY debug enable CATEGORY

and

xpra control :DISPLAY debug enable CATEGORY

And you can get the current list of debug loggers which:

xpra control :DISPLAY debug status

Other Environment Variables

There are other environment variables which can be used to tune the behaviour of the system and may be used for debugging or testing, you can obtain the full list with:

grep 'os.environ' src/xpra

Some examples:

  • some x264 attributes and thresholds: XPRA_X264_*_PROFILE, XPRA_X264_*_MIN_QUALITY, XPRA_X264_*_QUALITY
  • lossless damage threshold values (number of pixels): XPRA_MAX_NONVIDEO_PIXELS
  • sound test (use fake sound source on/off): XPRA_SOUND_TEST
  • use cython maths (on/off): XPRA_CYTHON_MATH
  • try to schedule other threads in network loop (on/off - see #181): XPRA_YIELD
  • use PIL for parsing png and jpeg packets (on/off): XPRA_USE_PIL

Please refer to the actual code for details.

Network Traffic

Seeing what is being exchanged between the client and server can be useful at times, an easy way to achieve this is to use tcp mode (ie: --bind-tcp=0.0.0.0:10000 option) without packet compression (-z 0 option) then the packets can be looked at with your favourite packet inspection tool, ie:

ngrep -p 10000

The packet type is the first thing in each packet and it is a simple string which makes it easy to observe traffic. With version 0.9 onwards, it is best to set XPRA_USE_ALIASES=0 to ensure that the packets will use plain-text headers rather than numeric aliases.

xtrace

(X11 traffic)

xtrace allows us to monitor the traffic between the xpra server and the Xvfb (X11 server), or between the client and its display server (Posix only).

  • server setup: because the server normally launches its own xvfb, we need to tell it to use an existing one and we interpose xtrace in between:
    • start your xvfb:
      Xvfb ... :10
      
    • start xtrace forwarding from :10 to :11:
      xtrace -k -d :10 -D :11
      
    • start an xpra server on the existing :11 display:
      xpra start --use-display :11
      


  • client setup: simply run the client via xtrace:
    xtrace xpra attach ..
    

GDB

When dealing with crashes ("core dumped"), the best way to debug is to fire gdb.

Attaching to an existing xpra process

Find the pid of the xpra process:

ps -ef | grep xpra

Then:

gdb python $PID_OF_XPRA_PROCESS_TO_DEBUG
# wait for it to load all the debug symbols
(gdb) continue

Starting xpra in gdb

gdb python $PID_OF_XPRA_PROCESS_TO_DEBUG
run /usr/bin/xpra start ...

Getting the backtrace

Then once you get a crash, gdb should show you its prompt again and you can extract the python stacktrace with py-bt and the full stacktrace with bt. Having both is useful. Note: installing the required "debug" symbol packages for your distribution is out of scope, please refer to your vendor's package manager for details (ie: debian and yum debuginfo-install).

X11 errors

Occasionally you may encounter an X11 crash (BadDrawable, BadWindow, etc). These are more tricky. You may need to set a breakpoint:

(gdb) break gdk_x_error

(or _XError if gdk_x_error is not found) And get a backtrace from there when gdb hits it.

Another useful environment variable is XPRA_X11_DEBUG_EVENTS, it allows you to specify a CSV list of X11 events to log specifically, ie:

XPRA_X11_DEBUG_EVENTS="PropertyNotify,MotionNotify" xpra ...

The full list of event names can be obtained by using an invalid value, which will trigger a warning message, or by looking at the source.

Unfortunately, because of the asynchronous nature of X11 calls, the error may generate this crash some time after the event that caused it. In this case, we may want to force all X11 calls to be synchronized (which will hurt performance and may even hide the bugs - beware of heisenbugs!), trace all X11 calls, etc. this modified error.py allows you to do just that (see the constants at the top).

Venerable Print Statements

When all else fails, or just when appropriate, sprinkling some print statements around the critical sections of code is often the best way to get a clearer picture of what is really going on..

Memory Leaks

In C code

If the memory leak is one of the Cython parts or in a C library, use regular tools, like valgrind. When using valgrind, the connection timeout may be exceeded because the client will be slowed down. Increasing this timeout will be necessary to be able to use the client with Valgrind. With r4861 onwards, you can do this using the XPRA_SOCKET_TIMEOUT env var:

XPRA_SOCKET_TIMEOUT=30 xpra attach ...

In python code

Needs documenting better..

Attachments (1)

  • error.py (6.7 KB) - added by Antoine Martin 7 years ago. debug version of error.py which allows us to force sync calls, add logging, etc

Download all attachments as: .zip