Xpra: Ticket #1075: make it possible to use gdppay and gdpdepay instead of ogg

Currently, the muxers are hard-coded for any given codec combination.

It would be nice if the client and server could negotiate which muxers they want to use so that we can use gdppay / gdpdepay instead of ogg. Ogg is heavier and adds latency. Especially for opus (#1074). This builds on the work done in #849.



Tue, 05 Jan 2016 16:30:15 GMT - Antoine Martin: owner, description changed

Negotiating the muxers would have made the code very very hard to maintain, and there is a much easier solution implemented in r11594 + r11596: we add some new virtual codecs named "flac+gdp", "opus+gdp" and "speex+gdp".

I now get on Fedora 23:

./xpra/sound/gstreamer_util.py
(...)
GStreamer version: 1.6.2.0
PyGStreamer version: 3.18.2
encoders supported: vorbis, opus+gdp, opus, flac+gdp, mp3, wav, wavpack, speex+gdp, speex
decoders supported: vorbis, opus+gdp, opus, flac+gdp, mp3, wav, wavpack, speex+gdp, speex
source plugins: pulsesrc, autoaudiosrc, alsasrc, osssrc, oss4src, audiotestsrc
sink plugins: autoaudiosink, pulsesink, alsasink, osssink, oss4sink
default sink: pulsesink

With this in place, we now prefer the "gdp" muxer variant for "opus", "flac" and "speex" if both ends support it. And if we end up making opus the default (#1074), we will just have "opus+gdp" first in the list.


Having some hard data to verify that gdp is really better than ogg would be useful too, though that's going to be hard to measure (everything else would need to be turned off: no screen updates, just sound), I would expect:

If possible, it is better to completely remove xpra from the equation for making measurements. For example we can run the gstreamer pipelines standalone. After running the server with -d sound we can see in the log output that the pipeline used for "opus+gdp" is:

sound source pipeline=pulsesrc device=alsa_output.pci-0000_00_14.2.analog-stereo.monitor ! \
    queue name=queue min-threshold-time=0 max-size-buffers=0 max-size-bytes=0 max-size-time=50000000 leaky=2 silent=1 ! \
    volume name=volume volume=1.0 ! opusenc cbr=0 complexity=0 ! \
    gdppay crc-payload=0 crc-header=0 ! \
    appsink name=sink emit-signals=true max-buffers=10 drop=true sync=false async=false qos=false

Then we can replace the source with a test "audiotestsrc" and the pulsesink with a "fakesink". Script the whole thing to run for 10 seconds and measure CPU usage with time:

time gst-launch-1.0 audiotestsrc ! \
    queue name=queue min-threshold-time=0 max-size-buffers=0 max-size-bytes=0 max-size-time=50000000 leaky=2 silent=1 ! \
    volume name=volume volume=1.0 ! opusenc cbr=0 complexity=0 ! \
    gdppay crc-payload=0 crc-header=0 ! fakesink &
sleep 10;killall gst-launch-1.0

Then we can repeat this process for all the different codecs... both client and server. (and if we wanted to, we could even test gstreamer 0.10 vs 1.x by replacing gst-launch-1.0 by gst-launch-0.10).

If we wanted to be really thorough, we could also change the audiotestsrc to produce different types of test sounds, ie: to get silence: audiotestsrc wave=4 (more info here: http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audiotestsrc.html). Or even feeding a specific music sampled as input using filesrc.

For more information on the output of time, see Real, User and Sys process time statistics. Ideally we would get more detailed statistics using a tool like perf, but this will do for now.

So far I got:

Client or Server Codec Real CPU User CPU Sys CPU
Server opus+gdp 10.0 19.9 ~0.04 +- 0.02
Server opus 10.0 19.9 0.09 +- 0.02

So that's a very small win for gdp, which is spending less time in kernel mode, probably because it does not rely on timers for chunking the data.


@afarr: please test that you can select the new "+gdp" codec options, unless you have lots of spare time and feel like compiling lots of interesting numbers as I started doing above...


Wed, 06 Jan 2016 01:29:40 GMT - alas: owner changed

Testing with fedora 23 0.17.0 r11580, I'm getting the same ./xpra/sound/gstreamer_util.py output as you posted, and the server runs with any of the "+gdp" codec options.

I don't have any clients handy that will also support those speaker-codecs, however, so I haven't tested that those codecs actually work (at least not yet).

I'll assign this back to you for now... but I'll test with a fedora client soon (which will presumably also support the codecs) to see if they actually output sound. I'll see what I can do about making the time to run some of those benchmark tests too... hopefully also in the near future. Not sure if you want to close or hold out for my making time (I could also fill out chart in closed ticket... if you'd prefer, until it's wiki-ready).


Wed, 06 Jan 2016 03:09:38 GMT - Antoine Martin: owner changed

I think we want to make sure that each new combination gets tested before the 0.17.0 release. No rush.

I've also added "vorbis+ogg" in r11599, which may help with the html5 client. r11628 adjusts the UI to make it fit on session info.

Please forward this to David.


Tue, 12 Jan 2016 00:50:26 GMT - alas:

Testing the "vorbis+ogg" with 0.17.0 r11640 osx client against 0.17.0 r11649 fedora 23 server... I'm stumbling across a client side error that's triggering a speaker stop.

2016-01-11 16:39:38,482 sound output pipeline error: gst-stream-error-quark: Could not decode stream. (7)
2016-01-11 16:39:38,482 sound output  gstvorbisdec.c(519)
2016-01-11 16:39:38,482 sound output  vorbis_handle_data_packet ()
2016-01-11 16:39:38,482 sound output  /GstPipeline:pipeline0/GstVorbisDec:vorbisdec0:
no header sent yet
2016-01-11 16:39:38,482 stopping speaker because of error: gst-stream-error-quark: Could not decode stream. (7)

I'll also attach full client and server side -d sound connection and error logs, in case there's something there of interest.

(Now that I have clients that will support codecs, I'll test the rest as well... and post any further issues.)


Tue, 12 Jan 2016 00:51:08 GMT - alas: attachment set

r11640 0.17 client output with -d sound


Tue, 12 Jan 2016 00:51:43 GMT - alas: attachment set

r11649 0.17 fedora 23 server outpur with -d sound


Tue, 12 Jan 2016 04:21:37 GMT - Antoine Martin:

r11654 removes vorbis+ogg and adds a note in the code, I had it working before, no idea why this doesn't work anymore (gstreamer 1.x?) and not going to care.

If all the other encodings work, please close.


Wed, 13 Jan 2016 03:03:58 GMT - alas: owner changed

Hmmm... quick summation (I'll throw in some tracebacks and such below... good news?).

Windows 0.17.0 r11653 (so listed from xpra.org/beta .. but indicating itself to be r11649 (1 change) )

2016-01-12 18:29:23,867 sound output Warning: bencode import failed: No module named 'xpra.net.bencode'
2016-01-12 18:31:10,473 the speex sound sink has stopped
2016-01-12 17:15:34,515 sound output Warning: bencode import failed: No module named 'xpra.net.bencode'
2016-01-12 17:15:49,492 the speex+gdp sound sink has stopped
2016-01-12 17:11:27,784 sound output pipeline error: gst-stream-error-quark: Internal data flow error. (1)
2016-01-12 17:11:27,784 sound output  gstbasesrc.c(2943)
2016-01-12 17:11:27,784 sound output  gst_base_src_loop ()
2016-01-12 17:11:27,784 sound output  /GstPipeline:pipeline0/GstAppSrc:src:
streaming task paused, reason not-negotiated (-4)
2016-01-12 17:11:27,784 sound output push-buffer error: <enum GST_FLOW_FLUSHING of type GstFlowReturn>
2016-01-12 17:11:27,784 sound output push-buffer error: <enum GST_FLOW_FLUSHING of type GstFlowReturn>
2016-01-12 17:11:27,783 stopping speaker because of error: push-buffer error: <enum GST_FLOW_FLUSHING of type GstFlowReturn>
2016-01-12 17:11:27,784 sound output push-buffer error: <enum GST_FLOW_FLUSHING of type GstFlowReturn>
2016-01-12 17:11:27,784 sound output stopping
2016-01-12 17:11:28,269 sound output using audio codec: flac+gdp

with the irregular bonus of:

2016-01-12 18:39:47,490 received console event CTRL_C
2016-01-12 18:39:47,490 error closing None
Traceback (most recent call last):
  File "xpra\net\protocol.pyc", line 942, in close
AttributeError: 'NoneType' object has no attribute 'input_bytecount'

OSX 0.17.0 r11653

2016-01-12 18:48:27,330 Warning: invalid value for speaker-codec: opus+gdp
2016-01-12 18:48:27,330  valid options: vorbis, vorbis+ogg, flac+gdp, mp3, wav, wavpack, speex+gdp, speex

(And the same message for opus and flac, with appropriate change to first line.)


I suppose I can try to test with gstreamer 0.10 as best I can... but it looks like vorbis+ogg isn't the only codec misbehaving.

I'll also see about taking a stab at that automated testing command you provided, see what I can get out of it...


Wed, 13 Jan 2016 06:29:13 GMT - Antoine Martin: owner changed

Windows 0.17.0 r11653 (so listed from xpra.org/beta .. but indicating itself to be r11649 (1 change) )


Well spotted, I'll look into it.


Causing client system warnings as the sound crashes


Those are caused by the switch to gstreamer 1.x and will be shown (until I remove them) whether the sound works or not, I've mentioned them in ticket:1041#comment:3. Just ignore them for now.
Doesn't speex / speex+gdp work? Works fine here.


Throwing same error as vorbis+ogg — flac+gdp:



Failing "elegantly" — flac, opus, opus+gdp


They're not really "failing", just not available. In the case of flac, that's expected (see just above), but opus should be available. It is on my systems and we even want to make it the default codec. Problems related to opus belong in #1074.


Wed, 13 Jan 2016 11:42:40 GMT - Antoine Martin:

I refused to give up and came up with a way to get flac to work with gstreamer 1.x, done in r11686. (an ugly fix..)

I also attempted (and failed) with those combinations: vorbis+ogg, flac (which is flac+ogg), opus+rtp, opus+webm..

For the record: some codec names are short (ie: "flac") because they were introduced before we had a choice of muxers to use... we probably should support both short and long names (ie: "flac" == "flac+ogg").

also:


Fri, 15 Apr 2016 14:47:34 GMT - Antoine Martin: priority changed

Raising for the imminent 0.17 release.


Tue, 17 May 2016 23:16:41 GMT - alas: owner changed

Well... not so sure this late is better than never... but here goes.

Re-tested with 0.17.0 r12465 osx client against 0.17.2 r12453 (unknown changes) fedora 23 server (I think I'll have to add something to those build scripts to be sure it deletes old revision numbers).

Checking the list of available codecs from /usr/lib64/python2.7/site-packages/xpra/sound/gstreamer_util.py, it looks like all the listed codec are working rather nicely except VORBIS_MKA, which I'm presuming is largely intended for the html5 client (?).

As I was going through the codecs though, I started noticing that, with a 1440p display and using the BBC HD sync test page from youtube, I was occasionally getting some significantly different av-sync results.

All work though. I suspect closing this and addressing av-sync fine tuning in another ticket is the best plan... but let me know if you'd feel better to have me testing with a variety of clients before closing.


Wed, 10 Aug 2016 02:32:47 GMT - Antoine Martin: status changed; resolution set

all the listed codec are working rather nicely except VORBIS_MKA


This should be recorded in #1090


I was occasionally getting some significantly different av-sync results


This should go in #1164.

Closing, even though some codecs don't seem to work on osx: #1276.


Wed, 24 Aug 2016 05:37:00 GMT - Antoine Martin: status changed; resolution deleted

gdp muxers were broken, see #1276.


Wed, 24 Aug 2016 05:37:52 GMT - Antoine Martin: status changed; resolution set

It would have been good to catch this during the 0.17 release cycle :(

See ticket:1276#comment:13. gdp is no more.


Sat, 23 Jan 2021 05:14:15 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1075