Currently, the muxers are hard-coded for any given codec combination.
It would be nice if the client and server could negotiate which muxers they want to use so that we can use gdppay / gdpdepay instead of ogg. Ogg is heavier and adds latency. Especially for opus (#1074). This builds on the work done in #849.
Negotiating the muxers would have made the code very very hard to maintain, and there is a much easier solution implemented in r11594 + r11596: we add some new virtual codecs named "flac+gdp", "opus+gdp" and "speex+gdp".
I now get on Fedora 23:
./xpra/sound/gstreamer_util.py (...) GStreamer version: 1.6.2.0 PyGStreamer version: 3.18.2 encoders supported: vorbis, opus+gdp, opus, flac+gdp, mp3, wav, wavpack, speex+gdp, speex decoders supported: vorbis, opus+gdp, opus, flac+gdp, mp3, wav, wavpack, speex+gdp, speex source plugins: pulsesrc, autoaudiosrc, alsasrc, osssrc, oss4src, audiotestsrc sink plugins: autoaudiosink, pulsesink, alsasink, osssink, oss4sink default sink: pulsesink
With this in place, we now prefer the "gdp" muxer variant for "opus", "flac" and "speex" if both ends support it. And if we end up making opus the default (#1074), we will just have "opus+gdp" first in the list.
Having some hard data to verify that gdp is really better than ogg would be useful too, though that's going to be hard to measure (everything else would need to be turned off: no screen updates, just sound), I would expect:
If possible, it is better to completely remove xpra from the equation for making measurements. For example we can run the gstreamer pipelines standalone.
After running the server with -d sound
we can see in the log output that the pipeline used for "opus+gdp" is:
sound source pipeline=pulsesrc device=alsa_output.pci-0000_00_14.2.analog-stereo.monitor ! \ queue name=queue min-threshold-time=0 max-size-buffers=0 max-size-bytes=0 max-size-time=50000000 leaky=2 silent=1 ! \ volume name=volume volume=1.0 ! opusenc cbr=0 complexity=0 ! \ gdppay crc-payload=0 crc-header=0 ! \ appsink name=sink emit-signals=true max-buffers=10 drop=true sync=false async=false qos=false
Then we can replace the source with a test "audiotestsrc" and the pulsesink with a "fakesink". Script the whole thing to run for 10 seconds and measure CPU usage with time:
time gst-launch-1.0 audiotestsrc ! \ queue name=queue min-threshold-time=0 max-size-buffers=0 max-size-bytes=0 max-size-time=50000000 leaky=2 silent=1 ! \ volume name=volume volume=1.0 ! opusenc cbr=0 complexity=0 ! \ gdppay crc-payload=0 crc-header=0 ! fakesink & sleep 10;killall gst-launch-1.0
Then we can repeat this process for all the different codecs... both client and server. (and if we wanted to, we could even test gstreamer 0.10 vs 1.x by replacing gst-launch-1.0 by gst-launch-0.10).
If we wanted to be really thorough, we could also change the audiotestsrc to produce different types of test sounds, ie: to get silence: audiotestsrc wave=4
(more info here: http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audiotestsrc.html). Or even feeding a specific music sampled as input using filesrc.
For more information on the output of time, see Real, User and Sys process time statistics. Ideally we would get more detailed statistics using a tool like perf, but this will do for now.
So far I got:
Client or Server | Codec | Real CPU | User CPU | Sys CPU |
---|---|---|---|---|
Server | opus+gdp | 10.0 | 19.9 | ~0.04 +- 0.02 |
Server | opus | 10.0 | 19.9 | 0.09 +- 0.02 |
So that's a very small win for gdp, which is spending less time in kernel mode, probably because it does not rely on timers for chunking the data.
@afarr: please test that you can select the new "+gdp" codec options, unless you have lots of spare time and feel like compiling lots of interesting numbers as I started doing above...
Testing with fedora 23 0.17.0 r11580, I'm getting the same ./xpra/sound/gstreamer_util.py
output as you posted, and the server runs with any of the "+gdp" codec options.
I don't have any clients handy that will also support those speaker-codecs, however, so I haven't tested that those codecs actually work (at least not yet).
I'll assign this back to you for now... but I'll test with a fedora client soon (which will presumably also support the codecs) to see if they actually output sound. I'll see what I can do about making the time to run some of those benchmark tests too... hopefully also in the near future. Not sure if you want to close or hold out for my making time (I could also fill out chart in closed ticket... if you'd prefer, until it's wiki-ready).
I think we want to make sure that each new combination gets tested before the 0.17.0 release. No rush.
I've also added "vorbis+ogg" in r11599, which may help with the html5 client. r11628 adjusts the UI to make it fit on session info.
Please forward this to David.
Testing the "vorbis+ogg" with 0.17.0 r11640 osx client against 0.17.0 r11649 fedora 23 server... I'm stumbling across a client side error that's triggering a speaker stop.
2016-01-11 16:39:38,482 sound output pipeline error: gst-stream-error-quark: Could not decode stream. (7) 2016-01-11 16:39:38,482 sound output gstvorbisdec.c(519) 2016-01-11 16:39:38,482 sound output vorbis_handle_data_packet () 2016-01-11 16:39:38,482 sound output /GstPipeline:pipeline0/GstVorbisDec:vorbisdec0: no header sent yet 2016-01-11 16:39:38,482 stopping speaker because of error: gst-stream-error-quark: Could not decode stream. (7)
I'll also attach full client and server side -d sound
connection and error logs, in case there's something there of interest.
(Now that I have clients that will support codecs, I'll test the rest as well... and post any further issues.)
r11640 0.17 client output with -d sound
r11649 0.17 fedora 23 server outpur with -d sound
r11654 removes vorbis+ogg and adds a note in the code, I had it working before, no idea why this doesn't work anymore (gstreamer 1.x?) and not going to care.
If all the other encodings work, please close.
Hmmm... quick summation (I'll throw in some tracebacks and such below... good news?).
Windows 0.17.0 r11653 (so listed from xpra.org/beta .. but indicating itself to be r11649 (1 change) )
2016-01-12 18:29:23,867 sound output Warning: bencode import failed: No module named 'xpra.net.bencode' 2016-01-12 18:31:10,473 the speex sound sink has stopped
2016-01-12 17:15:34,515 sound output Warning: bencode import failed: No module named 'xpra.net.bencode' 2016-01-12 17:15:49,492 the speex+gdp sound sink has stopped
2016-01-12 17:11:27,784 sound output pipeline error: gst-stream-error-quark: Internal data flow error. (1) 2016-01-12 17:11:27,784 sound output gstbasesrc.c(2943) 2016-01-12 17:11:27,784 sound output gst_base_src_loop () 2016-01-12 17:11:27,784 sound output /GstPipeline:pipeline0/GstAppSrc:src: streaming task paused, reason not-negotiated (-4) 2016-01-12 17:11:27,784 sound output push-buffer error: <enum GST_FLOW_FLUSHING of type GstFlowReturn> 2016-01-12 17:11:27,784 sound output push-buffer error: <enum GST_FLOW_FLUSHING of type GstFlowReturn> 2016-01-12 17:11:27,783 stopping speaker because of error: push-buffer error: <enum GST_FLOW_FLUSHING of type GstFlowReturn> 2016-01-12 17:11:27,784 sound output push-buffer error: <enum GST_FLOW_FLUSHING of type GstFlowReturn> 2016-01-12 17:11:27,784 sound output stopping 2016-01-12 17:11:28,269 sound output using audio codec: flac+gdp
with the irregular bonus of:
2016-01-12 18:39:47,490 received console event CTRL_C 2016-01-12 18:39:47,490 error closing None Traceback (most recent call last): File "xpra\net\protocol.pyc", line 942, in close AttributeError: 'NoneType' object has no attribute 'input_bytecount'
OSX 0.17.0 r11653
2016-01-12 18:48:27,330 Warning: invalid value for speaker-codec: opus+gdp 2016-01-12 18:48:27,330 valid options: vorbis, vorbis+ogg, flac+gdp, mp3, wav, wavpack, speex+gdp, speex
(And the same message for opus and flac, with appropriate change to first line.)
I suppose I can try to test with gstreamer 0.10 as best I can... but it looks like vorbis+ogg isn't the only codec misbehaving.
I'll also see about taking a stab at that automated testing command you provided, see what I can get out of it...
Windows 0.17.0 r11653 (so listed from xpra.org/beta .. but indicating itself to be r11649 (1 change) )
Well spotted, I'll look into it.
Causing client system warnings as the sound crashes
Those are caused by the switch to gstreamer 1.x and will be shown (until I remove them) whether the sound works or not, I've mentioned them in ticket:1041#comment:3. Just ignore them for now.
Doesn't speex / speex+gdp work? Works fine here.
Throwing same error as vorbis+ogg — flac+gdp:
Failing "elegantly" — flac, opus, opus+gdp
They're not really "failing", just not available.
In the case of flac, that's expected (see just above), but opus should be available.
It is on my systems and we even want to make it the default codec. Problems related to opus belong in #1074.
I refused to give up and came up with a way to get flac to work with gstreamer 1.x, done in r11686. (an ugly fix..)
I also attempted (and failed) with those combinations: vorbis+ogg, flac (which is flac+ogg), opus+rtp, opus+webm..
For the record: some codec names are short (ie: "flac") because they were introduced before we had a choice of muxers to use... we probably should support both short and long names (ie: "flac" == "flac+ogg").
also:
-d sound
, we now have a -d gstreamer
for this
Raising for the imminent 0.17 release.
Well... not so sure this late is better than never... but here goes.
Re-tested with 0.17.0 r12465 osx client against 0.17.2 r12453 (unknown changes) fedora 23 server (I think I'll have to add something to those build scripts to be sure it deletes old revision numbers).
Checking the list of available codecs from /usr/lib64/python2.7/site-packages/xpra/sound/gstreamer_util.py
, it looks like all the listed codec are working rather nicely except VORBIS_MKA
, which I'm presuming is largely intended for the html5 client (?).
As I was going through the codecs though, I started noticing that, with a 1440p display and using the BBC HD sync test page from youtube, I was occasionally getting some significantly different av-sync results.
All work though. I suspect closing this and addressing av-sync fine tuning in another ticket is the best plan... but let me know if you'd feel better to have me testing with a variety of clients before closing.
all the listed codec are working rather nicely except VORBIS_MKA
This should be recorded in #1090
I was occasionally getting some significantly different av-sync results
This should go in #1164.
Closing, even though some codecs don't seem to work on osx: #1276.
gdp muxers were broken, see #1276.
It would have been good to catch this during the 0.17 release cycle :(
See ticket:1276#comment:13. gdp is no more.
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1075