Xpra: Ticket #231: queuing packets for sending is very expensive in UI thread

We currently require all calls to send() to be done in the main thread, but this can be quite expensive - see this pycallgraph diagram: https://www.xpra.org/trac/raw-attachment/ticket/231/pycallgraph-expensivetosend3.png

which was obtained with a server running:

xpra start :10 --start-child="vglrun glxgears"

And the client running

XPRA_OPENGL=0 ./tests/scripts/pycallgraph -i '*' -e ALL -d 2 -r 20 -- \
    attach :10 --no-mmap --encoding=x264

Most of the time used in the UI thread is to do with this network stuff, all the other things (decoding, I/O are already in separate threads and therefore do not affect latency much).

Potential solutions:



Tue, 08 Jan 2013 05:36:22 GMT - Antoine Martin: attachment set

shows how expensive it is to be queuing packets from the main thread


Tue, 08 Jan 2013 05:36:58 GMT - Antoine Martin: status, description changed; owner set


Tue, 08 Jan 2013 06:00:40 GMT - Antoine Martin: description changed


Wed, 09 Jan 2013 10:14:52 GMT - Antoine Martin: attachment set

re-generate the graph with better / more reproducible instructions (and replace old one)


Wed, 09 Jan 2013 10:16:21 GMT - Antoine Martin: attachment set

re-generate the graph with better / more reproducible instructions


Wed, 09 Jan 2013 10:19:37 GMT - Antoine Martin: description changed


Wed, 09 Jan 2013 10:24:18 GMT - Antoine Martin: attachment set

re-generate the graph with better / more reproducible instructions (trying again - not sure why trac doesn't let us see it / preview it)


Wed, 09 Jan 2013 10:25:05 GMT - Antoine Martin: description changed


Wed, 09 Jan 2013 10:26:38 GMT - Antoine Martin: attachment set

updated graph with r2492 showing a much improved codepath and timing


Wed, 09 Jan 2013 10:38:25 GMT - Antoine Martin: status changed; resolution set

r2492 fixes this (lots of details in the commit message)


Profiling wiki received some import updates for correct profiling (some tricky and misleading issues)


What is immediately obvious by looking at the new callgraph below is how we skip numerous indirection steps by simplifying the code structure:


https://www.xpra.org/trac/raw-attachment/ticket/231/pycallgraph-fixed-r2492.png


Wed, 09 Jan 2013 10:40:06 GMT - Antoine Martin: attachment set

updated graph with r2493 showing more improvements


Wed, 09 Jan 2013 10:41:30 GMT - Antoine Martin:

Finally, with r2493, we're down to barebones, probably as fast as it is going to be in python:
https://www.xpra.org/trac/raw-attachment/ticket/231/pycallgraph-fixed-r2493.png


Sat, 23 Jan 2021 04:49:05 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/231