xpra icon
Bug tracker and wiki

Opened 3 years ago

Closed 22 months ago

Last modified 3 weeks ago

#849 closed task (fixed)

sound improvements: refactorings, cleanups,

Reported by: Antoine Martin Owned by: alas
Priority: critical Milestone: 0.16
Component: sound Version: trunk
Keywords: Cc: onlyjob@…

Description (last modified by Antoine Martin)

Follow up from #400, #669 - related to #835.

  • would be nice to share more code between the client and server
  • handle the sound sequence mechanism for "microphone" just like we do for "speaker"
  • use a cython or ctypes wrapper for accessing the pulse properties rather execing a subprocess
  • avoid the osx pool autorelease warnings?
  • setup process tagging in subprocess only?
  • allow us to use python3 / gstreamer 1.0 in the subprocess from a python2 caller (requires execing a process to get the list of codecs supported..)

Attachments (103)

sound-autotune-queue.patch (7.7 KB) - added by Antoine Martin 2 years ago.
experimenting with auto-tuning the queue to minimize overruns and underruns whilst keeping the overall delay low
palib.patch (13.0 KB) - added by Antoine Martin 2 years ago.
support for using palib instead of execing pactl and parsing the output
palib.tar.xz (13.1 KB) - added by Antoine Martin 2 years ago.
fixed palib: add setup.py for installation, ensure we can call it multiple times without causing problems with signals
python-palib.spec (1.6 KB) - added by Antoine Martin 2 years ago.
specfile
palib-1.0.tar.xz (12.5 KB) - added by Antoine Martin 2 years ago.
updated source with py3k fixes
palib-1.0.tar.2.xz (13.0 KB) - added by Antoine Martin 2 years ago.
updated source with py3k fixes and server_info support
sound-autotune-queue-v2.patch (6.9 KB) - added by Antoine Martin 2 years ago.
updated patch
sound-show-queue-graph.patch (7.2 KB) - added by Antoine Martin 2 years ago.
shows the min/max/cur values of the sound buffer as a graph on session info
queue-levels-graph.png (9.0 KB) - added by Antoine Martin 2 years ago.
the new queue level graph shown in the session info window
sound-source-jitter.png (22.5 KB) - added by Antoine Martin 2 years ago.
example of sound source jitter causing overruns and underruns
100ms-precision.png (13.7 KB) - added by Antoine Martin 2 years ago.
switch to 100ms sampling so we can see more clearly the changes
sound-tune.patch (10.2 KB) - added by Antoine Martin 2 years ago.
try to tune the queue dynamically to minimize buffering
lower.png (18.4 KB) - added by Antoine Martin 2 years ago.
shows how we gradually lower the buffer level by lowering the max level
underruns.png (28.1 KB) - added by Antoine Martin 2 years ago.
harmless underruns
win7-lower-buffer.png (18.2 KB) - added by Antoine Martin 2 years ago.
a very clear example of what we're trying to do with the max level: lower the current level
sound-clean-split-gstreamer.patch (22.4 KB) - added by Antoine Martin 2 years ago.
work on completely removing any gstreamer imports from the client and server: everything goes through the wrapper, and we only call it once to probe
ticket849_osx_fedora21_0-16-0-r10380_defaults-Features.png (109.6 KB) - added by alas 2 years ago.
osx & fedora 21 0.16.0 r 10380 default ... encodings
ticket849_osx_fedora21_0-16-0-r10380_defaults-Graphs.png (125.5 KB) - added by alas 2 years ago.
osx & fedora 21 0.16.0 r 10380 default ... encodings graph
ticket849_osx_fedora21_0-16-0-r10380_both-vp8-info.png (95.0 KB) - added by alas 2 years ago.
osx & fedora 21 0.16.0 r 10380 server & client vp8 ... encodings info
ticket849_osx_fedora21_0-16-0-r10380_both-vp8-graph.png (124.1 KB) - added by alas 2 years ago.
osx & fedora 21 0.16.0 r 10380 server and client vp8 ... encodings graph
sound-clean-split-gstreamer-v3.patch (50.3 KB) - added by Antoine Martin 2 years ago.
allows us to completely remove all gstreamer imports from the client and server, we only deal with sound properties and let the sound subprocess do what it claims to be able to handle through these properties
fedora21_0.16.0_r10306,osx_0.16.0_r10380,default_speaker_codecs,spotify_music,focus_on_spotify_tab.png (99.3 KB) - added by pvenkateswaralu 2 years ago.
fedora21_0.16.0_r10306,osx_0.16.0_r10380,default_speaker_codecs,youtube_music,spotify_music,focus_on_spotify_tab.png (118.4 KB) - added by pvenkateswaralu 2 years ago.
fedora21_0.16.0_r10306,osx_0.16.0_r10380,default_speaker_codecs,youtube_music,spotify_music,focus_on_youtube_but_video_not_onscreen.png (100.8 KB) - added by pvenkateswaralu 2 years ago.
fedora21_0.16.0_r10306,osx_0.16.0_r10380,default_speaker_codecs,youtube_music,spotify_music,focus_on_youtube_video.png (107.2 KB) - added by pvenkateswaralu 2 years ago.
fedora21_0.16.0_r10306,osx_0.16.0_r10380,default_speaker_codecs,youtube_music,spotify_music,scrolling_youtube_video_tab.png (116.3 KB) - added by pvenkateswaralu 2 years ago.
fedora21_0.16.0_r10306,osx_0.16.0_r10380,default_speaker_codecs,youtube_music.png (116.0 KB) - added by pvenkateswaralu 2 years ago.
ticket849_0-16-0-r10442-win_r10306-fedora21_xterm.PNG (78.1 KB) - added by alas 2 years ago.
sound output with VORBIS and just xterm
ticket849_0-16-0-r10442-win_r10306-fedora21_youtube-music-while-not-scrolling.PNG (70.3 KB) - added by alas 2 years ago.
windows client with VORBIS and youtube video, not scrolling
ticket849_0-16-0-r10442-win_r10306-fedora21_youtube-music-while-scrolling.PNG (63.9 KB) - added by alas 2 years ago.
windows client with VORBIS and youtube video, while scrolling
ticket849_0-16-0-r10442-win_r10306-fedora21_youtube-music_video-fullscreen-4K.PNG (71.7 KB) - added by alas 2 years ago.
windows client with VORBIS and youtube video, fullscreen 4K
ticket849_0-16-0-r10442-win_r10306-fedora21_youtube-and-mp3player-music-while-scrolling_video-offscreen.PNG (79.2 KB) - added by alas 2 years ago.
windows client with VORBIS, youtube video and mp3 player in second tab, scrolling on youtube tab (video offscreen)
ticket849_0-16-0-r10442-win_r10306-fedora21_youtube-music-no scrolling-for-a-while_focus-tab-no-video.PNG (50.3 KB) - added by alas 2 years ago.
windows client with VORBIS, youtube and mp3 player, focus on (non-video) mp3 player tab
ticket849_0-16-0-r10442-win_r10306-fedora21_youtube-music-no scrolling_server--speaker-codec-mp3_client-default-vorbis.PNG (33.4 KB) - added by alas 2 years ago.
windows client, server with --speaker-codec=mp3, changes microphone codec, info
ticket849-windows-default-speaker-codec_info.txt (119.2 KB) - added by alas 2 years ago.
windows client, fedora 21 server, default codecs, xpra info
ticket849_0-16-0-r10380-osx_r10306-fedora21_xterm.png (85.7 KB) - added by pvenkateswaralu 2 years ago.
ticket849_0-16-0-r10380-osx_r10306-fedora21-0.16.0_youtube-music_video-fullscreen-4K.png (89.5 KB) - added by pvenkateswaralu 2 years ago.
ticket849_0-16-0-r10380-osx_r10306-fedora21-0.16.0_youtube-music-while-not-scrolling.png (93.0 KB) - added by pvenkateswaralu 2 years ago.
ticket849_0-16-0-r10380-osx_r10306-fedora21-0.16.0_youtube-music-while-scrolling.png (95.7 KB) - added by pvenkateswaralu 2 years ago.
ticket849_0.16.0-r10380-osx_fedora21-r10306-0.16.0_youtube-on-one-tab_spotify-on-another_focus-on-spotify-tab-no-video.png.png (90.8 KB) - added by pvenkateswaralu 2 years ago.
ticket849_0.16.0-r10380-osx_r10306-fedora21-0.16.0_youtube-in-one-tab_spotify-on-another_focus-on-youtube-with-video.png (107.7 KB) - added by pvenkateswaralu 2 years ago.
ticket849_0.16.0-r10380-osx_r10306-fedora21-0.16.0_youtube-on-one-tab_spotify-on-another_focus-on-youtube-tab-while-scrolling.png (97.9 KB) - added by pvenkateswaralu 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_Audio_Tab_Muted_Sound.png (71.5 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_AudioTabOpen_PlayingVideo&Audio_on_VideoTab.png (52.5 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_AudioTabOpen_PlayingVideoandAudioFromTwoDiffTabs.png (62.5 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_AudioTab_open_with_no_audio_no_video.png (61.6 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_AudioTab_Speaker_off.png (71.4 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_Only video(YouTube).png (72.0 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_Scrolling_VideoTab.png (225.3 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_Video_Scrolling.png (67.7 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_VideoTabOpen_PlayingAudioFromAudioTab(Spotify).png (47.8 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_VideoTabOpen_PlayingAudioSimultaneously_FromDiffTabs.png (81.2 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_Video_Tab_open_Switching_Speaker_on.png (64.3 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBoth(Flac_S:0.16.0Rev10306_C:0.16.0Rev10306_VideoTab_Speaker_Off.png (48.5 KB) - added by pharindranath 2 years ago.
Fedora21_Default both (Flac)Video& audio( YouTube)S:0.16.0Rev 10306C:0.16.0Rev10306.png (70.4 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBout(FLAC)_S:0.16.0Rev10306_AudioTabOpen_speaker_On.png (59.5 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBout(FLAC)_S:0.16.0Rev10306_C:0.16.0Rev10306_Comments.odt (15.7 KB) - added by pharindranath 2 years ago.
Fedora21_Sesion_Info audio only.png (58.4 KB) - added by pharindranath 2 years ago.
Fedora21_DefaultBout(FLAC)_S:0.16.0Rev10306_C:0.16.0Rev10306_xtern.png (6.4 KB) - added by pharindranath 2 years ago.
Fedora 21_AudioS:0.16.0Rev10306C:0.16.0Rev10306_Video_Scrolling__Graph.png (55.4 KB) - added by pharindranath 2 years ago.
sound-negotiate-muxers.patch (19.4 KB) - added by Antoine Martin 2 years ago.
allows more detailed selection of the codec used: codec + muxer
sound-negotiate-muxers-v2.patch (20.1 KB) - added by Antoine Martin 2 years ago.
newer work in progress patch
sound-gstreamer1-as-default.patch (2.0 KB) - added by Antoine Martin 2 years ago.
makes gstreamer1 the default on platforms where we can easily use it (not osx or win32), pending because it causes problems with exe and log files on win32…
849mediumhighlatency.png (51.8 KB) - added by J. Max Mena 2 years ago.
Connected to a home server from other side of city on wi-fi with mild latency.
xpra-ticket849_sound-buffer-with-just-gedit-and-looping-xpra-info-bug.PNG (57.9 KB) - added by alas 2 years ago.
sound buffer graphs with only an xterm and gedit, but with #985 looping xpra info outputting client side
ticket849_0-16-0-r10655_freeze-with-constant-stream-of-info_at-5.48.39.png (277.7 KB) - added by alas 2 years ago.
file-name says it all…
ticket849_0-16-0-r10655_freeze-with-constant-info_wireless_at-15.51.34.png (72.5 KB) - added by alas 2 years ago.
ticket849_constant-client-info-stream_once-it-stabilizes_at-15.46.50.png (285.5 KB) - added by alas 2 years ago.
ticket849_0-16-0-r10655_with-constant-info_once-it-stabilizes_wireless_at-15.53.43.png (78.8 KB) - added by alas 2 years ago.
ticket849_stable-wireless-near-router_at-15.57.42.png (305.5 KB) - added by alas 2 years ago.
ticket849_wirelesslap-begins_walking-away-from-router_at-15.58.08.png (274.3 KB) - added by alas 2 years ago.
ticket849_walked-to-next-room_wireless_at-15.58.37.png (291.4 KB) - added by alas 2 years ago.
walked to next room... brief spinners/jitter
ticket849_getting-spinners_walked-downstairs-and-out-door_wireless_at-15.59.29.png (294.1 KB) - added by alas 2 years ago.
ticket849_out-front-door_spinners-continue_at-16.00.22.png (244.6 KB) - added by alas 2 years ago.
ticket849_out-the-door-but-stationary_spinners-recover_at-16.00.35.png (295.5 KB) - added by alas 2 years ago.
ticket849_out-front-door-but-still-stationary_spinner-recovery-continues_at-16.00.48.png (298.9 KB) - added by alas 2 years ago.
ticket849_still-out-front-door-stationary_spinners-as-recovered-as-they-gonna-get_at-16.01.13.png (293.5 KB) - added by alas 2 years ago.
ticket849_out-front-door_begin-walking-garage_relatively-still_buffers-coping-well_at-16.01.35.png (287.5 KB) - added by alas 2 years ago.
ticket849_walking-through-ground-garage_more-spinners_at-16.01.49.png (283.8 KB) - added by alas 2 years ago.
ticket849_stopping-at-back-door_spinners-recover_at-16.02.09.png (314.8 KB) - added by alas 2 years ago.
ticket849_stationary-by-back-door-despite-heat_buffers-continue-to-stabilize_at-16.02.54.png (296.3 KB) - added by alas 2 years ago.
ticket849_stationary-at-back-door-in-heat-til-end-of-patience_buffers-continue-to-stabilize_at-16.03.22.png (285.9 KB) - added by alas 2 years ago.
ticket849_stationary-at-back-door-beyond-patience_where-are-my-cigarettes_buffers-remarkably-stable_at-16.04.11.png (285.9 KB) - added by alas 2 years ago.
ticket849_initial-fumbling-for-keys-for-door_spinners-as-corner-is-turned-by-security-cameras_at-16.04.22.png (289.3 KB) - added by alas 2 years ago.
ticket849_cigarettes-obviously-left-inside_leap-for-door-induces-more-spinners_at-16.04.39.png (274.0 KB) - added by alas 2 years ago.
ticket849_inside-again-downstairs_fumbling-with-door-produces-yet-more-spinners_at-16.05.01.png (280.3 KB) - added by alas 2 years ago.
ticket849_back-inside_patient-at-bottom-of-stairs_buffers-recover_at-16.05.27.png (274.1 KB) - added by alas 2 years ago.
ticket849_climbing-back-up-stairs-in-router-ward-direction_buffers-firing-for-effect_at-16.05.47.png (277.5 KB) - added by alas 2 years ago.
ticket849_up-the-stairs_walking-ever-more-router-ward_at-16.06.01.png (271.2 KB) - added by alas 2 years ago.
ticket849_walked-not-only-router-ward_but-past-router-a-meter-and-change_at-16.07.08.png (279.5 KB) - added by alas 2 years ago.
ticket849_wireless-lap-complete_back-at-rest-wireless_at-16.12.34.png (74.3 KB) - added by alas 2 years ago.
at-seat-at-rest_no-encryption_16.55.58.png (268.9 KB) - added by alas 2 years ago.
walk-and-return graph travelogue starting point - sitting down
walking_halfway-between-initial-and-secondary-router_no-encryption_16.56.24.png (275.1 KB) - added by alas 2 years ago.
travelogue point two, up and walking away from one router, toward another - buffer seems to over compensate
stopped-by-second-router_no-encryption_16.56.45.png (270.3 KB) - added by alas 2 years ago.
travelogue stop three, stopping near second router - buffer still hasn't re-adjusted
walk-around-corner-from-second-router_some-spinners_no-encryption_16.57.20.png (269.3 KB) - added by alas 2 years ago.
travelogue point 4, walking past second router and around a corner, signal passing through walls
walk-around-second-corner_more-spinners_no-encryption_16.57.43.png (267.1 KB) - added by alas 2 years ago.
travelogue point 5, turn and walk around a second corner and down half a flight of stairs, seeing some spinners but signal stabilizes quickly - buffer copes very well
return-from-around-second-corner_no-encryption_16.58.19.png (253.4 KB) - added by alas 2 years ago.
travelogue point 6, returning from around second corner and down some stairs; still around one corner - buffer adjusts very well, despite renewed spinners
ticket849_buffer-level-when-restarting-sound.png (144.7 KB) - added by alas 2 years ago.
buffer level jumps when re-starting sound with control channel
ongoing-wav-with-videos.png (216.0 KB) - added by alas 2 years ago.
wav, listening to videos on youtube, with latency (frame total) at +/-157
ongoing-wav-once-latency-drops.png (222.1 KB) - added by alas 2 years ago.
wav, listening to videos on youtube, after latency (frame total) drops back to +/- 105
wav-once-125ms-100ms-23percent-ended.png (96.9 KB) - added by alas 2 years ago.
wav, at time when 125ms 100ms 23percent tc delay ended
ongoing-wav-once-disabled-125ms-100ms-23percent.png (83.7 KB) - added by alas 2 years ago.
wav as latency drops after ending tc delay
wav-with-all-tc-disabled-but-rising-latency.png (91.2 KB) - added by alas 2 years ago.
wav, as latency rises without tc being used

Change History (163)

comment:1 Changed 3 years ago by Antoine Martin

Description: modified (diff)
Status: newassigned
Summary: sound improvements:sound improvements: refactorings, cleanups,

comment:2 Changed 3 years ago by Antoine Martin

Some fixes (backported already, see ticket:669#comment:43) and some changes worth recording (some of which we may want to backport): r9328.

comment:3 Changed 3 years ago by onlyjob

Cc: onlyjob@… added

comment:4 Changed 3 years ago by Antoine Martin

r9407 allows us to try to use gstreamer 1.x with python2 for probing and debugging (not really usable yet).

In order to completely move to a subprocess for all the gstreamer bits, we need to ensure that (almost) all calls to browser/xpra/trunk/src/xpra/sound/gstreamer_util.py go through the subprocess wrapper.
Either by adding new commands to the wrapper for probing (ie: getting the list of plugins) or by ensuring the data is exported back to the client / server via info packets from the subprocess.

Grepping the source:

  • bug report tool: can continue to use direct calls, should the info found in the client (if present and if we have any)
  • session info: only needs version information, we can include that in info (small cost)
  • ui client base: uses gstreamer util for populating caps and for validating command line options. Both need to use the wrapper, it will be a tad slower, but this is a one-off.
  • server base and server source: same as client, one-off.
  • the "main" script queries the plugins when the "help" option is used, no problem in using a subprocess to get that

Packaging updates:

  • switch to gstreamer 1.x for Fedora and Stretch (and maybe Jessie?)
  • win32: no idea about the availability of the gi bindings, will need to figure that out
  • osx: we already use a subprocess wrapper script, should be possible to tweak the environment to make it load gstreamer 1.x

comment:5 Changed 2 years ago by Antoine Martin

  • some fixes were needed to avoid some new warnings with gstreamer 1.x: r9501
  • r9519 tries to continue on overrun, we try to re-fill the pipeline to a usable level - will need thorough testing on OSX as the restart code was written for handling overruns on OSX..

If this works out, we can enhance the logic to take the average value and range of the queue values to try to minimize its size (helps with #835).

comment:6 Changed 2 years ago by Antoine Martin

  • cleanups in r9527
  • r9528 changes the default appsink settings (and maybe we should backport the drop=true property?)
  • r9529 makes it easier to tweak the appsink settings, ie (showing the defaults):
    XPRA_SOURCE_APPSINK="appsink name=sink emit-signals=true max-buffers=10 drop=true sync=true async=true qos=true" \
        xpra start :10
    

Changed 2 years ago by Antoine Martin

Attachment: sound-autotune-queue.patch added

experimenting with auto-tuning the queue to minimize overruns and underruns whilst keeping the overall delay low

Changed 2 years ago by Antoine Martin

Attachment: palib.patch added

support for using palib instead of execing pactl and parsing the output

Changed 2 years ago by Antoine Martin

Attachment: palib.tar.xz added

fixed palib: add setup.py for installation, ensure we can call it multiple times without causing problems with signals

comment:7 Changed 2 years ago by Antoine Martin

See the work in progress patches above based on the code found in pypacl: we want the palib parts which are nice, but not the pypactl bits which are not finished and not usable. This would require packaging it, which shouldn't be too hard (GPLv3+), a setup.py has already been added.

Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by Antoine Martin

Attachment: python-palib.spec added

specfile

Changed 2 years ago by Antoine Martin

Attachment: palib-1.0.tar.xz added

updated source with py3k fixes

Changed 2 years ago by Antoine Martin

Attachment: palib-1.0.tar.2.xz added

updated source with py3k fixes and server_info support

comment:8 Changed 2 years ago by Antoine Martin

As of r9537, the code will now try to load palib and print a warning before falling back to execing pactl as before.
I have posted beta RPM builds of python-palib.

Everything seems to work, and faster too. It is cleaner / safer than the previous code which relies on execing "pactl" and parsing strings (and hoping they won't change in the future!).
We also get more information out of the pulseaudio server:

$ ./xpra/sound/pulseaudio_util.py 
device.alsa_input.pci-0000_00_14.2.analog-stereo                 : Built-in Audio Analog Stereo
device.alsa_input.usb-046d_HD_Pro_Webcam_C920_15475ECF-02-C920.analog-stereo : HD Pro Webcam C920 Analog Stereo
device.alsa_output.pci-0000_00_14.2.analog-stereo                : Built-in Audio Analog Stereo
device.alsa_output.pci-0000_00_14.2.analog-stereo.monitor        : Monitor of Built-in Audio Analog Stereo
devices                                                          : 4
pulseaudio.cookie                                                : 394338217
pulseaudio.default_sink_name                                     : alsa_output.pci-0000_00_14.2.analog-stereo
pulseaudio.default_source_name                                   : alsa_input.pci-0000_00_14.2.analog-stereo
pulseaudio.host_name                                             : desktop
pulseaudio.id                                                    : 1000@0d37226ca0064aaab1db9e66016c45f5/2311
pulseaudio.server                                                : {0d37226ca0064aaab1db9e66016c45f5}unix:/run/user/1000/pulse/native
pulseaudio.server_name                                           : pulseaudio
pulseaudio.server_version                                        : 6.0
pulseaudio.user_name                                             : antoine
pulseaudio.wrapper                                               : palib

about python palib: this library is old (2010), the code is ugly, badly indented, etc..
But it works.

Ideally, we could re-write the bits we use from palib using libpulseaudio - which is also ugly, less friendly (the code is generated automatically from the library) - but since we use very little and don't actually use the mainloop, this would be cleaner.

comment:9 Changed 2 years ago by Antoine Martin

As of r9633 + r9636 (+some minor fixes in r9634 + r9635), we can run gstreamer 1.x from the GTK2 client or server (or even gstreamer 0.10 from the GTK3 client if one wanted to do so - apart from testing, I can't imagine why you would want to do that).

All calls to the sound subpackage now go through a subprocess, which can use a different gobject / glib library.

By default, we still run sound with the same python version as the client or server that controls it, but this can be overriden on posix with:

XPRA_SOUND_COMMAND="/usr/bin/python3 /usr/bin/xpra" /usr/bin/python2 xpra start ...

OSX and win32 would require a lot more work to support this as we would need to package two versions of the python interpreters (2.7.x and 3.x), each with all the dependencies.


Here are some new useful hidden subcommands, used by some xpra help command line options and for populating the list of codecs on startup:

$ python2 /usr/bin/xpra _sound_query sources
sources=pulsesrc,autoaudiosrc,alsasrc,osssrc,oss4src,audiotestsrc
$ python2 /usr/bin/xpra _sound_query encoders
encoders=mp3,flac,wavpack,wav
$ python2 /usr/bin/xpra _sound_query decoders
decoders=mp3,flac,wavpack,wav

Note how you get different results when running with python3 (where we now disable flac to avoid a bug):

$ python3 /usr/bin/xpra _sound_query encoders
encoders=mp3,wavpack,wav
Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by Antoine Martin

updated patch

comment:10 Changed 2 years ago by Antoine Martin

palib leaks file descriptors and is unmaintained, we need something better.
(see #912)

Changed 2 years ago by Antoine Martin

shows the min/max/cur values of the sound buffer as a graph on session info

Changed 2 years ago by Antoine Martin

Attachment: queue-levels-graph.png added

the new queue level graph shown in the session info window

comment:11 Changed 2 years ago by Antoine Martin

I've wasted time on pulseaudio for nothing, see ticket:912#comment:4 ...

But as of r10227, we now show the sound buffer level and min/max limits in use:
the new queue level graph shown in the session info window

I want to do a bit more testing on various platforms to see how this behaves, what range of values we get, etc..
@afarr: feel free to provide feedback on this.

Then it shouldn't be too hard to adjust the levels in a more gradual way.
We ought to be able to figure out how low we can get the level by keeping a history of the level range we observe.
Actually lowering the level is a bit more difficult:

  • we can drop packets - as we now do when we hit overruns, but how many milliseconds this drops will vary - and is not very predictable, and this can be more than we want...
  • playing with the queue limits as we do now, for both underruns and overruns, which drops packets after they've been parsed - which means that we are dealing with time limits at this point not just raw packets (this is more in line with what we're trying to achieve)
  • changing the playback speed - but I couldn't get that to work when I tried: looks like we're supposed to use seek events, maybe this conflicts with a live source

Problems:

  • wav is not playing nice (fills the buffer way too quickly) - not sure we care, could just drop this encoding
  • testing network conditions is hard...
  • bound to be platform quirks (osx...)
Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by Antoine Martin

Attachment: sound-source-jitter.png added

example of sound source jitter causing overruns and underruns

comment:12 Changed 2 years ago by Antoine Martin

As of r10231, it is a bit easier to stress test the sound level:

XPRA_SOUND_SOURCE_JITTER=700 xpra start ..

Will introduce random jitter in the delivery of sound buffers (given in milliseconds, the random value will be between 1 and the value specified). This will cause the client side levels to go up, and occasionally hit overruns:
example of sound source jitter causing overruns and underruns

Changed 2 years ago by Antoine Martin

Attachment: 100ms-precision.png added

switch to 100ms sampling so we can see more clearly the changes

comment:13 Changed 2 years ago by Antoine Martin

The data sampling is still asynchronous, but as of r10234 we run it every 100ms which makes the graph data more precise:
switch to 100ms sampling so we can see more clearly the changes

Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by Antoine Martin

Attachment: sound-tune.patch added

try to tune the queue dynamically to minimize buffering

Changed 2 years ago by Antoine Martin

Attachment: lower.png added

shows how we gradually lower the buffer level by lowering the max level

Changed 2 years ago by Antoine Martin

Attachment: underruns.png added

harmless underruns

Changed 2 years ago by Antoine Martin

Attachment: win7-lower-buffer.png added

a very clear example of what we're trying to do with the max level: lower the current level

comment:14 Changed 2 years ago by Antoine Martin

Recap: the point of these changes is to try to keep the buffer level low as this helps with #835.

Dealing with underruns is relatively easy: we just temporarily increase the min-level which causes data to accumulate in the buffer (visually, the green line goes up and back down again)
Going below "min" usually triggers an underrun, but that doesn't always causes the sound to stutter or stop.
Here's what it looks like:
harmless underruns



Dealing with overruns is much harder:

  • we want to be able to cause overruns to lower the buffer level to make the queue drop some buffers, it looks like this:

a very clear example of what we're trying to do with the max level: lower the current level

  • if we get too many overruns then we need to raise the max level, because overruns cause the sound to crackle

Those two behaviours conflict with each other..
The events are asynchronous and multi-threaded, which makes it very hard to tie things together. For example, the "Level" can go above the "max" threshold without triggering an overrun if the pipeline consumes the buffers in time...
Changing the "max" level also causes the sound pipeline to stutter, so we can't change it too often.

Here's what we try to do in r10250:

  • We keep track of how many overruns we hit and when. If the number increases (only keeping track of the last 60 seconds) then we raise the limit to try to prevent more overruns.
  • We lower the max level if the range of values of the current level is small enough that it should fit in a smaller space (the vertical range of the blue line) - which often causes an overrun, and:
  • When we hit an overrun, we temporarily lower the max level, which drains some of the buffer contents

(this lowering causes the range to go up... which can cause the max level to go back up, but by that point the work has been done already)

New tunables:

  • XPRA_SOUND_GRACE_PERIOD=2000 number of milliseconds to wait for before we do anything with underruns and overruns, allows the pipeline time to settle down a bit - probably find as it is. (could eventually be replaced by an event, like the one we get when the decoder is processing the codec and gives us the codec name - meh)
  • XPRA_SOUND_MARGIN=50 (from 0 to 200), tunes how aggressively we try to lower the max level (from 0=no margin=aggressive to 200 which lets the buffer level go wild) - this one may well need tuning. Too low and we get too many overruns, too high and the buffer level is too high...
  • XPRA_SOUND_FAKE_OVERRUN no longer exists..

Testing notes / some ideas of things to try to see what effect they have:

  • to see what the sound subprocess does, run the client with XPRA_SOUND_SUBPROCESS_DEBUG=1 xpra attach ...
  • try different apps that generate sound (ie: flash, html5, music player)
  • start and stop the sound (ie: music player) will often trigger underruns
  • start and stop from the tray
  • change the volume up and down
  • useful to try: various codecs, various OS, various python versions (ie: to test gstreamer 1.x with python3, see comment:9 : XPRA_SOUND_COMMAND=python3 /usr/bin/xpra - we want to switch to python3 / gstreamer 1.x sooner rather than later, see #903)
  • various network conditions (see also comment:12)
  • various application workloads (heavy video will use more bandwidth, add latency and tax the cpu)
  • various encodings (again: changes the latency/throughput of the network connection)
  • there's also the new fault injection code to test: XPRA_WRAPPER_FAULT_INJECTION_RATE=200 xpra attach ...
  • I have not tested OSX - I hope it doesn't cause too many problems, there were deadlocks before when we hit too many overruns... (could be related to the leak warning we see on osx - ticket:533#comment:69)
  • it would be worth testing just before and after r10250 to get a better baseline to compare against

During my testing I found (mostly with mp3, and also with vorbis):

  • Gb lan can keep the buffer below ~120ms
  • wifi bumps it up to ~250ms
  • win32 is a tad higher than Linux

PS:

  • r10249 + r10252 + r10255: minor tweaks, the current sound caps can be seen with xpra info | grep caps
  • gstreamer 1.x compat fixes: r10251 + r10253
  • r10275 re-enables opus (only with gstreamer 1.x) and speex: a good time to take a look at those again and see if they are better than mp3 for our use case (and also re-test wavpack / flac - though I am seeing overruns with those..)
  • r10276 fixes vorbis - beware though: we can't mix python2/gstreamer-0.10 servers with python3/gstreamer-1.x clients without getting a stream of warnings: GStreamer-CRITICAL **: gst_segment_to_stream_time: assertion 'segment->format == format' failed (not much we can do about this I am afraid... I'll have to write some code to skip this combination since the warning is deep in gstreamer code and we can't really avoid it since gstreamer is the one that is generating the packet metadata that causes the problems) And here's the proof:
    • this works (0.10 to 0.10, 1.0 to 0.10, 1.0 to 1.0):
      gst-launch-0.10 -q audiotestsrc ! vorbisenc ! gdppay ! filesink location=/dev/stdout | gst-launch-0.10 filesrc location=/dev/stdin ! gdpdepay ! vorbisdec ! autoaudiosink
      gst-launch-1.0 -q audiotestsrc ! vorbisenc ! gdppay ! filesink location=/dev/stdout | gst-launch-0.10 filesrc location=/dev/stdin ! gdpdepay ! vorbisdec ! autoaudiosink
      gst-launch-1.0 -q audiotestsrc ! vorbisenc ! gdppay ! filesink location=/dev/stdout | gst-launch-1.0 filesrc location=/dev/stdin ! gdpdepay ! vorbisdec ! autoaudiosink
      
    • this doesn't (0.10 to 1.0), not sure why it works when we do this same pipeline from our code using python gstreamer...:
      gst-launch-0.10 -q audiotestsrc ! vorbisenc ! gdppay ! filesink location=/dev/stdout | gst-launch-1.0 filesrc location=/dev/stdin ! gdpdepay ! vorbisdec ! autoaudiosink
      
  • the graphs only show the client side buffer levels, which is only half of the problem - to evaluate the server side latency you will still need to rely on visual inspection using one of the sync videos (see ticket:835#comment:1)
Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:15 Changed 2 years ago by Antoine Martin

Owner: changed from Antoine Martin to alas
Status: assignednew

@afarr: ready for some testing / feedback.

comment:16 Changed 2 years ago by Antoine Martin

Priority: majorcritical

Many more improvements have been made, in particular r10305 which makes vorbis, speex and opus so good that they should now displace mp3 as the new default codec in 0.16 (and it is such a noticeable improvement that this should probably be backported too).
We can keep mp3 around for the html5 (and android?) clients, opus is not available with python2 (does not work which is a shame) but we can use it server side by using python3 for the sound wrappers.
So the default order should probably now be: opus, vorbis, mp3, flac, speex, wav, ..
But obviously, this will require a lot more testing. mp3 has served us well until now.

flac is still disabled on win32 (see #749 for details, maybe we should also re-open #300? see also #299)
Though we can now force enable any codec using:

XPRA_SOUND_CODEC_ENABLE_XXXXXXX=1

(replace XXXXX with the codec name in uppercase, ie: FLAC)

The latency is so much better that we get sound sync (#835) almost for free as long as the network allows us to keep the client-side buffer low (see comments above for details).

I would also like to add a new capability to clients so we can use different container formats with new clients: it looks like gdppay + gdpdepay is more lightweight than using oggmux + oggdemux.
PS: the latency improvement may cost a little bit more CPU usage.
PS2: we should probably think about switching to GStreamer 1.x by default server side in 0.16, or at least have an easier way of switching (config file instead of env var)
PS3: minor updates in r10345 (commit message is wrong).

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:17 Changed 2 years ago by J. Max Mena

Done some thorough testing on the following client OSs against a trunk Fedora 21 Server:

  • Win8.1
  • Fedora 20
  • Fedora 21
  • OSX 10.10 and 10.6.8

I still have a few more OSs I can try, but I have a solid 'feel' for how the sound sync is working in each OS with their default codecs (my next step is to try with the different codecs in different client OSs). As far as I can tell it's pretty close. Sometimes sound can actually get slightly ahead of video - I've seen it do this in Windows and OSX. Fedora works well - although video got choppy on my low end machine. (Smo thinks it's using too much CPU, though)

The only cases where I've had a problem was after I was using a session for a couple hours (doing boring QA things...Trac among others), and the sound queue had steadily climbed to 150-200ms. Other than that, it's surprisingly good coming from 14.XX and 15.XX!


I'm not entirely sure what info you'd like, so if you want anything more concrete than first impressions(maybe figure out a screen record with sound?), let me know and I'll try and collect it.

comment:18 Changed 2 years ago by alas

Testing a bit with 0.16.0 win32 r10306 client against a fedora 21 0.16.0 r10306 server...

  • I tried with XPRA_SOUND_SOURCE_JITTER=500, on a random mp3 playing website with chrome (http://www.mp3juices.cc/ no video playing), and it looks like the max level adjusts pretty well to the "topography" of the sound buffer levels (in the neighborhood of 500 ms). the min level doesn't seem to be adjusting at all in this case though... for the most part rolling at a flat 0 ms.
  • Adding a couple of tabs of videos with sound, the sound levels begin to occasionally overrun the max level (in this case running at a steady 565 ish ms). It doesn't look like th emax level is adjusting any higher... maybe the overruns aren't sufficient? (I'm certainly not hearing any static and I don't see any sign of speaker restarts or anything outputting client side with no flags.)
  • Running the same with XPRA_SOUND_SUBPROCESS_DEBUG=1 client side, before re-starting the mp3 player, the client-side output is showing overrun counts = 0 consistently, but the sound levels seem to run around 400 ms (with a lot more dips to 0 ms) - but the max level, rather than the 565 ish ms, seems to have settled at 625 ms (resulting in a lot more whitespace between the sound level and the max level in the case of videos playing dialog, several tabs of youtube, but without the mp3 player.
  • Starting the mp3 player in addition, the sound levels seem to climb back up to 500-550 range, but the max level remains steady at 625 (far fewer drops to zero of the sound level, and the min level seems to actually rise as high as 40 or 50 more often... still doesn't seem to be adjusting upward though, in case that's significant.

With XPRA_SOUND_SOURCE_JITTER=0

  • Before starting any sound ... the min sound level seems to have jumped to 25 ms, the max is at 100 ms, but the levels themselves seem to be below the min buffer, mostly at 0 with spikes up to the 12 - 25 ms range.
  • Starting videos with sound, the max level remains steady at 100 ms, the levels themselves seem to sharply spike up and down between 0 and about 103 ms, but the minimums seem to be ignoring the actual levels and periodically "wedging" up to 25 ms, then sloping back down to 0.
  • Starting the mp3 playing, the max level immediately jumps to 150 ms, but the levels themselves are (initially at least) only spiking into the 105 ms range... though as time passes they do seem to more regularly hit the 130-140 ms range. The minimum levels seem to be doing a much better job of predicting and "rising to" the actual levels though.

A quick couple of milliseconds of the client-side debug output from the above (I assume it will look about like you expect):

2015-08-19 18:55:15,756 sound-sink export(info, ...)
2015-08-19 18:55:15,756 sound-sink send: adding 'info' message (0 items already in queue)
2015-08-19 18:55:15,756 sound-sink add_packet_to_queue(info ...)
2015-08-19 18:55:15,756 sound-sink processing packet add_data
2015-08-19 18:55:15,756 sound-sink add_data(522 bytes, {'duration': 26122449, 'sequence': 0, 'time': 1440035724113L}) queue_state=pushing
2015-08-19 18:55:15,756 sound-sink processing packet add_data
2015-08-19 18:55:15,770 sound-sink pushed   522 bytes, new buffer level:  78ms, queue state=pushing
2015-08-19 18:55:15,770 sound-sink set_min_level pct= 0, cmtt=  0, mtt=  0
2015-08-19 18:55:15,770 sound-sink set_max_level lrange=130, last_max_update=237s
2015-08-19 18:55:15,770 sound-sink set_max_level overrun count=1 , margin= 50, pct= 0, cmst=152, mst=186
2015-08-19 18:55:15,770 sound-sink add_data(417 bytes, {'duration': 26122449, 'sequence': 0, 'time': 1440035724113L}) queue_state=pushing
2015-08-19 18:55:15,770 sound-sink pushed   417 bytes, new buffer level: 104ms, queue state=pushing
2015-08-19 18:55:15,770 sound-sink set_min_level pct= 0, cmtt=  0, mtt=  0
2015-08-19 18:55:15,770 sound-sink set_max_level lrange=130, last_max_update=237s
2015-08-19 18:55:15,770 sound-sink set_max_level overrun count=1 , margin= 50, pct= 0, cmst=152, mst=186
2015-08-19 18:55:15,786 sound-sink processing packet add_data
2015-08-19 18:55:15,786 sound-sink add_data(522 bytes, {'duration': 26122449, 'sequence': 0, 'time': 1440035724142L}) queue_state=pushing
2015-08-19 18:55:15,786 sound-sink pushed   522 bytes, new buffer level: 130ms, queue state=pushing
2015-08-19 18:55:15,786 sound-sink set_min_level pct= 0, cmtt=  0, mtt=  0
2015-08-19 18:55:15,786 sound-sink set_max_level lrange=130, last_max_update=237s
2015-08-19 18:55:15,786 sound-sink set_max_level overrun count=1 , margin= 50, pct= 0, cmst=152, mst=186

I'll see about finding some local sound & video to play (rather than websites) to see if it behaves roughly similarly. I'll also try some different jitter settings, and maybe some encodings, as soon as I get the chance.

Am I right in assuming that, currently, video encodings won't have any effect on the sound channel levels? (I can't imagine trying to encode video with rgb will be kind to processors, but is it worth trying crazy combinations like that to see what effect it has on the sound... or rather, if it has any effect?)

comment:19 Changed 2 years ago by Antoine Martin

..but I have a solid 'feel' for how the sound sync is working..


This ticket is not sound sync.
This ticket is about more general improvements, which will help with sound sync. See comment:14 : Recap: the point of these changes is to try to keep the buffer level low as this helps with #835.
We need data on how well this is working. The latency graph was custom made for easily visualizing this data.


the sound queue had steadily climbed to 150-200ms


Did it ever correct itself back down? What does it look like? Where are the numbers?

Also, please try vorbis and opus: we want to change the default codec order, at least in trunk, see comment:16.


still doesn't seem to be adjusting upward though, in case that's significant.


No, we don't want the min level to go up permanently, it is only used briefly to force the buffers to fill up when we get underruns, to try to prevent further overruns.
That said, if in the future we find that the sound is ahead of the video then we can use it to delay the sound and bring it in sync. I don't have any numbers to judge.


Am I right in assuming that, currently, video encodings won't have any effect on the sound channel levels?


Less efficient encodings (especially at high res), like rgb without any compression, will use up more bandwidth and each picture packet will be much bigger and so they will cause more jitter in the sound buffer levels.
The jitter hack is good for seeing how well the code deals with it, real jitter is better because it tell us how jitter happens in the field.


Since I don't see other codecs being tested and it works very very well for me, r10372 changes the preferred order to be: VORBIS, OPUS, FLAC, MP3, WAV, WAVPACK, SPEEX.
So vorbis should get used and tested now by default.

Ideally, we want to know for each encoding and on a wide variety of platforms (maybe make a table):

  • bandwidth used: min, max, avg - under different conditions (ie: no sound vs music player)
  • buffer levels: min, max...
  • issues encountered
  • overruns, sound quality (crackling?) etc
Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:20 Changed 2 years ago by Antoine Martin

Important note: r10391 switches from Bytes/s to bits/s as the unit used for sound. (more standard for sound)
Vorbis seems quite happy at 96Kbit/s.

The latency fix has been applied to v0.14.x (r10430) and v0.15.x (r10429)

Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by Antoine Martin

work on completely removing any gstreamer imports from the client and server: everything goes through the wrapper, and we only call it once to probe

comment:21 Changed 2 years ago by alas

We were able to do a good bit of testing with osx 0.16.0 r10380 against fedora 21 0.16.0 r10380 with a variety of codecs. I'll see about attaching in an orderly fashion for your perusal (not sure if inline will be more clutter or more useful, so I'll leave that for later).

We'll try to run more of the same with more platforms as we can.

Changed 2 years ago by alas

osx & fedora 21 0.16.0 r 10380 default ... encodings

Changed 2 years ago by alas

osx & fedora 21 0.16.0 r 10380 default ... encodings graph

Changed 2 years ago by alas

osx & fedora 21 0.16.0 r 10380 server & client vp8 ... encodings info

Changed 2 years ago by alas

osx & fedora 21 0.16.0 r 10380 server and client vp8 ... encodings graph

comment:22 Changed 2 years ago by alas

Err... looks like we have been so focused on video, we tested with a variety of encodings, rather than codecs.

Will re-run with osx and other platforms focusing on different codecs, graphs... and details of issues. Soon. (Let me know if you'd like more graphs of further encodings permutations though...)

comment:23 Changed 2 years ago by Antoine Martin

The OSX client build you used only supports mp3 and wav, that is going to seriously limit your testing. When I test my r10380 beta OSX build, I do get vorbis. Was this one of your own builds?
There is a new beta (r10446) which has all the codecs available (except for opus which requires python3).
Though by the time you read this, there may be new changes worth testing.

The server build has a few more (wavpack and flac) but is missing the new default (vorbis) and speex - how did you install the server package?? (the required gstreamer packages are already listed as dependencies and should have been installed automatically).
(You will need the python3 xpra build to be able to test opus - documented above).

Also, those graphs don't show anything really interesting. I guess that everything was running smoothly? (it doesn't state what the client window was doing or what sort of sound was playing)

The vp8 info shows that you used encodings=vp8 rather than encoding=vp8 (the 's' makes all the difference). Was this intentional?

Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by Antoine Martin

allows us to completely remove all gstreamer imports from the client and server, we only deal with sound properties and let the sound subprocess do what it claims to be able to handle through these properties

comment:24 Changed 2 years ago by Antoine Martin

(not merging the patch above just yet because it could break things and I haven't tested it enough yet - but it is definitely the right way of doing things, we no longer exec the subprocess multiple times for probing, sound starts up quicker, we save memory, should make things more reliable when mixing python2 and python3 subprocess, etc..)

comment:25 Changed 2 years ago by pvenkateswaralu

I ran xpra with default speaker codecs for both client and server. There was no weird behavior. The sound was great and continuous. Attached screenshots.

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:26 Changed 2 years ago by alas

I suspect that osx client might've been running on a 10.6.8 old osx machine... maybe that was why the VORBIS (and others) weren't available?

Anyway, pvekateswaralu will post some new graphs from a newer osx.

Meanwhile I did some more testing with windows client 0.16.0 r10442 against a fedora 21 0.16.0 r10306.

Started with default codecs, which ran VORBIS on both sides.

Have a screenshot for graphs with:

  • just an xterm
  • youtube music video
  • youtube music video while scrolling
  • youtube with music playing, tab fullscreen 4K
  • youtube in one tab, mp3 player in another tab, focus on youtube tab with video
  • youtube in one tab, mp3 player in another tab, focus on youtube tab with video while scrolling
  • youtube in one tab, mp3 player in another tab, focus on the mp3 player tab (no video)

After starting the initial youtube video, I noticed a little stutter in the sound, and saw the following when I checked the server output (not certain it corresponded, but this was the only time I saw this output:

2015-08-26 17:02:03,870 sound-source pipeline warning: Can't record audio fast enough
2015-08-26 17:02:03,870 sound-source pipeline warning: ["gstbaseaudiosrc.c(840): gst_base_audio_src_create (): /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0:\nDropped 1080640 samples. This is most likely because downstream can't keep up and is consuming samples too slowly."]

When I started the mp3 player in addition to the video, I heard another short stutter, but didn't see any sign of output either client or server side.


I then tried to start the server with --speaker-codec=mp3, but checking in the session info I saw that it was listing the microphone codec as being mp3, but the speaker codecs as unaffected.

A quick check for info output this:

[tlaloc@jimador ~]$ xpra info :13 | grep codec
client.encoding.avcodec2.version=(56, 41, 100)
client.speaker.codec=vorbis
client.speaker.codec_description=Vorbis
server.argv=('/usr/bin/xpra', '--bind-tcp=0.0.0.0:1201', '--mdns=no', '--start-c            hild=xterm', '--start-child=xterm', 'start', ':13', '--speaker-codec=mp3', '--daemon=no')
window[3].encoding.pipeline_option[0].csc=codec_spec(swscale)
window[3].encoding.pipeline_option[0].encoder=codec_spec(x264)
window[3].encoding.pipeline_option[1].encoder=codec_spec(x264)
window[3].encoding.pipeline_option[2].csc=codec_spec(swscale)
window[3].encoding.pipeline_option[2].encoder=codec_spec(x264)
window[3].encoding.pipeline_option[3].csc=codec_spec(swscale)
window[3].encoding.pipeline_option[3].encoder=codec_spec(x264)
window[3].encoding.pipeline_option[4].csc=codec_spec(swscale)
window[3].encoding.pipeline_option[4].encoder=codec_spec(x264)
window[3].encoding.pipeline_option[5].csc=codec_spec(cython)
window[3].encoding.pipeline_option[5].encoder=codec_spec(x264)

Also attaching an xpra info so you can check the client and server builds real quick if anything looks odd (should both be your builds, but I've burned myself with cross-wired repos that I'm still mediocre at installing).

Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by alas

sound output with VORBIS and just xterm

Changed 2 years ago by alas

windows client with VORBIS and youtube video, not scrolling

Changed 2 years ago by alas

windows client with VORBIS and youtube video, while scrolling

Changed 2 years ago by alas

windows client with VORBIS and youtube video, fullscreen 4K

Changed 2 years ago by alas

windows client with VORBIS, youtube video and mp3 player in second tab, scrolling on youtube tab (video offscreen)

Changed 2 years ago by alas

windows client with VORBIS, youtube and mp3 player, focus on (non-video) mp3 player tab

Changed 2 years ago by alas

windows client, server with --speaker-codec=mp3, changes microphone codec, info

Changed 2 years ago by alas

windows client, fedora 21 server, default codecs, xpra info

comment:27 Changed 2 years ago by Antoine Martin

@pvenkateswaralu: I don't know what the "default codec" is, as this will vary depending on what codecs you have installed and what the server supports.


sound-source pipeline warning: Can't record audio fast enough
sound-source pipeline warning: ["gstbaseaudiosrc.c(840): gst_base_audio_src_create (): \
    /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0:\nDropped 1080640 samples. \
    This is most likely because downstream can't keep up and is consuming samples too slowly."]


That's interesting. If the buffers accumulate too quickly, we do drop them, but we do this in the gstreamer capture element at the end of the compression pipeline (aka "appsink") not in the soundcard capture element at the beginning of the pipeline (aka "pulsesrc" on Linux).
The pipeline goes something like this: pulsesrc -> audioconvert (changes sample rate, etc) -> volume control -> encoder -> container -> capture (appsink).

So my guess is that the compression is taking too long.. And I don't want to add a queue element in this pipeline, as this adds latency and is difficult enough to deal with client side (that's the point of this ticket).
We may have an issue if the CPU is not capable of compressing in realtime. Most modern CPUs are more than fast enough to compress sound in realtime no matter what encoding is used. So I am tempted to just make a note of this and hope for the best.


When I started the mp3 player in addition to the video, I heard another short stutter, but didn't see any sign of output either client or server side.


Does this show up on the latency graph as a spike?
If you can reproduce, you may want to run the client and/or server with -d sound and capture the few KB of log output around the time of the event, then look for underruns or overruns in the output. Those events are no longer logged at "info" level, only at "debug" level because the new sound code triggers them far too often.


I then tried to start the server with --speaker-codec=mp3, but checking in the session info I saw that it was listing the microphone codec as being mp3, but the speaker codecs as unaffected.


Thanks, that's fixed in r10454.

This graph shows a spike in latency (probably as you started scrolling):
windows client with VORBIS and youtube video, while scrolling

  • did you hear any crackling?
  • did the level come back down again later?

This graph shows the latency going through the roof:
windows client with VORBIS and youtube video, fullscreen 4K
So I don't think we should worry too much about the sound doing badly in this case, we should deal with the underlying problem instead which is that the encode+send takes far too long (over 2 seconds!). Can you please file a separate ticket for this?

All the other graphs look fairly normal to me: the latency is in the 50 to 150ms range which is fine.

The xpra info looks about right. FYI: the important bits are found using:

$ xpra info | grep client.speaker
client.speaker.buffers=4829
client.speaker.bytes=971134
client.speaker.caps={'application/x-gdp': {}}
client.speaker.codec=vorbis
client.speaker.codec_description=Vorbis
client.speaker.pid=11353
client.speaker.pipeline=pulsesrc ! volume name=volume volume=1.0 ! vorbisenc ! gdppay crc-payload=0 crc-header=0 ! appsink name=sink emit-signals=true max-buffers=10 drop=true sync=false async=false qos=false
client.speaker.state=active
client.speaker.time=1440641552
client.speaker.volume=100

As per comment:14 :

  • do you see the level going back down again after a spike
  • testing with wifi / slower / bad network conditions
  • testing the other codecs
  • compatibility with older clients and servers
  • etc..

PS: r10451 is the final code cleanup which completely separates the gstreamer bits from the main process, it even prevents the gstreamer bindings from ever being imported into the main process by accident.
This should have no user visible changes, but will reduce the memory footprint of the client and server quite a bit. (though the same libraries will still be imported by the sound helper subprocess when used)

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:28 in reply to:  27 Changed 2 years ago by pvenkateswaralu

Replying to antoine:

@pvenkateswaralu: I don't know what the "default codec" is, as this will vary depending on what codecs you have installed and what the server supports.

@antoine: The default speaker codec is the "Vorbis".

Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by pvenkateswaralu

comment:29 Changed 2 years ago by pvenkateswaralu

Owner: changed from alas to Antoine Martin

I did some testing with OSX client 0.16.0 r10380 against a fedora21 0.16.0 r10306.


Started with the default codecs, which ran VORBIS on both sides.

Have a screenshot for graphs with:

  • just an xterm
  • youtube music video
  • youtube music video while scrolling
  • youtube with music playing, tab fullscreen 4K
  • youtube in one tab, mp3 player in another tab, focus on youtube tab with video
  • youtube in one tab, mp3 player in another tab, focus on youtube tab with video while scrolling
  • youtube in one tab, mp3 player in another tab, focus on the mp3 player tab (no video)


After starting youtube video on one tab and spotify mp3 on another tab, I noticed a little slutter in the beginning, but didn't see any output on either client side or the server side.

Here's a quick check for the info output.

[maint@Fedora21-Server-289 ~]$ xpra info :13 | grep codec
 client.encoding.avcodec2.version=(56, 41, 100)
 client.speaker.codec=vorbis
 client.speaker.codec_description=Vorbis
 window[2].encoding.pipeline_option[0].csc=codec_spec(swscale)
 window[2].encoding.pipeline_option[0].encoder=codec_spec(x264)
 window[2].encoding.pipeline_option[1].encoder=codec_spec(x264)
 window[2].encoding.pipeline_option[2].csc=codec_spec(swscale)
 window[2].encoding.pipeline_option[2].encoder=codec_spec(x264)
 window[2].encoding.pipeline_option[3].csc=codec_spec(swscale)
 window[2].encoding.pipeline_option[3].encoder=codec_spec(x264)
 window[2].encoding.pipeline_option[4].csc=codec_spec(swscale)
 window[2].encoding.pipeline_option[4].encoder=codec_spec(x264)
 window[2].encoding.pipeline_option[5].csc=codec_spec(cython)
 window[2].encoding.pipeline_option[5].encoder=codec_spec(x264)
Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:30 Changed 2 years ago by pharindranath

I did some testing with Fedora 21 client 0.16.0 r10306

against a fedora 21 0.16.0 r10306.

Started with default codecs, which ran FLAC on both sides.

Have a screenshot for graphs with:

  • just an xterm
  • youtube music video
  • youtube music video while scrolling
  • youtube with music playing, tab fullscreen 4K
  • youtube in one tab, Spotify player in another tab, focus on youtube tab with video
  • youtube in one tab, Spotify player in another tab, focus on youtube tab with video while scrolling
  • youtube in one tab, Spotify player in another tab, focus on the Spotify player tab (no video)
  • Playing youtube with Tab open but speaker Off.
  • Playing youtube with Tab open but speaker On.
  • Playing Spotify with Tab open but speaker Off.
  • Playing Spotify with Tab open but speaker On.
  • Playing youtube with Tab open but muting audio.
  • Playing Spotify with Tab open but muting audio.

Sound Quality found good for all the different possibilities tested.

When playing the both the video-audio from youtube and audio from Spotify in two different Tabs simultaneously, the Audio from Tab closed played in the background but the sound quality remained unchanged.

Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by pharindranath

Changed 2 years ago by Antoine Martin

allows more detailed selection of the codec used: codec + muxer

Changed 2 years ago by Antoine Martin

newer work in progress patch

Changed 2 years ago by Antoine Martin

makes gstreamer1 the default on platforms where we can easily use it (not osx or win32), pending because it causes problems with exe and log files on win32...

comment:31 Changed 2 years ago by J. Max Mena

NOTE: I meant to post this comment last week. Apparently the computer I was using decided it would be better to hibernate until now...

Running a Windows 8.1 trunk r10380 client against a trunk 10399 Fedora 21 server:

Default codec was Vorbis (noticeably better than MP3)

  • Watching video files and Netflix through Google-Chrome has the sound queue relatively low (usually under 100ms).
  • Streaming music through Pandora(Chrome / Firefox - flash) also keeps the buffer relatively low
  • Playing local MP3 files also keeps the buffer low (under 100s)

Throughout testing I did adjust the volume periodically - no noticeable effect.

Screenshots of the Graphs page can be provided upon request; in addition to media files used.


As for now, I'll start switching speaker codecs and comparing them to each other. I have a nice consistent test I can use thanks to the video and audio files. Now that I'm using my home machines I'll have access to a much wider array of media files.


This weekend I'll get around to shrinking the Windows partition on my old laptop and see about triple booting (if the bootloader will let me - I have had issues in the past) Win7/Fedora 21/Linux Mint(or something else; maybe Fedora 22) and using that as a client, as it has an AMD GPU. Of course, that's after I fix my Linux server - I broke a fan the other day and need to completely take it apart...ugh.

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:32 Changed 2 years ago by Antoine Martin

Screenshots of the Graphs page can be provided upon request..


Those can be useful if they show unusual or unexpected behaviour. The dozens attached to this ticket so far, not so much.

Reminder: this ticket is about managing the client-side sound queue and keeping it as low as we can. So we want to stress the system (client / server / connection / sound server / etc..) to trigger spikes in latency.
Hopefully the heuristics can force overruns and push the queue levels back down. (see the second graph in comment:14 for an example of how it is supposed to work)

I am most interested in cases where this does not work as it should:

  • the queue level remains high when it could be lowered more
  • the sound quality suffers (crackling or other)
  • any other bad behaviour

Since changing the defaults seems to be the most effective way to get some testing done on a particular option, I have now changed the default gstreamer version we run on Linux (OSX and win32 would take a lot more work to package it, but it would be worth doing - just so we stop running woefully out of date code, with libraries we are unable to build from source on win32..), this is split into multiple changesets because they all touch different areas (and we may want to undo / change them individually): r10466, r10467, r10468, r10469, r10470, r10471 + r10472.

There are beta builds available win32, OSX. (looks like Fedora rpmbuild is still choking on the full vp9 selftests.. will fix)
PS: I have just spotted a last minute problem: the sound subprocess seems to go zombie when we exit the client on win32 (and maybe osx too?)... exiting reliably vs killing all subprocesses, pick one...

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:33 Changed 2 years ago by Antoine Martin

Owner: changed from Antoine Martin to alas

(should not be assigned to me unless there are issues to address, which is what we are trying to find - it is testing time)

comment:34 Changed 2 years ago by Antoine Martin

Lots more improvements and fixes: r10490, r10491, r10492, r10493, r10494, r10495, r10498, r10500. Also tested on Ubuntu and Debian with the gstreamer 1.x bindings.

comment:35 Changed 2 years ago by J. Max Mena

Running a Fedora 20 r10475 trunk server and connected a Win8.1 r10504 client:

EDIT:

  • Using vorbis for sound codec

end edit

  • Connected on SJSU(my university..)'s wifi - latency isn't too much bigger
  • Idling with just an Xterm and Firefox, the graphs show I'm using 2500kbps sound (nothing playing...) and an average sound queue between 100-300ms.

Will attach screenshot.

Last edited 2 years ago by J. Max Mena (previous) (diff)

Changed 2 years ago by J. Max Mena

Attachment: 849mediumhighlatency.png added

Connected to a home server from other side of city on wi-fi with mild latency.

comment:36 Changed 2 years ago by Antoine Martin

@maxmylyn: thanks, this one is interesting.
Did it ever come back down again? The buffer levels range from about 125ms to ~425ms, it should eventually (within a minute or so) try to lower the level down to the 0-300ms range (assuming that there aren't more spikes increasing the range during that time).

This sort of latency will probably require #835 to get the sound more or less in sync. The difficulty here is that the video will also suffer from this 300ms jitter! (was it far off as it is?)

The 2Mbps is a bit high! I suspect the accounting is off by a factor of 8. (256Kbps)


@pharindranath: please do not use word processors for recording comments. Having to click the link, click download, click open-with-word-processor, just to read 5 lines of plain text is cumbersome.

Here's the text found in the document I have now deleted:

Sound Quality found good for all the different possibilities tested.
First Tab had youtube and second Tab had Spotify.
When playing the both the video-audio from youtube and audio from Spotify in two different Tabs simultaneously, the Audio from Tab closed played in the background but the sound quality remained unchanged.

Client and server: Fedora 21, Xpra 0.16.0 r10306

What are the xterm, session info versions and "Scrolling_VideoTab.png" for? Can I delete them?

Having dozens of screenshots with very similar contents makes it very difficult to analyse the results.
Is there anything noteworthy in them? All I can see is that the buffer levels stay below ~125ms, which is great but does not tell us how well the code manages the queue level when under stress.

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:37 Changed 2 years ago by J. Max Mena

Did it ever come back down again?


Nope. Stayed that way. For what it's worth, interacting with anything had some mild latency. When I attempted earlier while sitting on my laptop outside with a weaker Wi-Fi signal, Firefox was entirely unusable until I disabled the speaker. I suspect the 256kbps was too much bandwidth on the bare edge of the router's range. The nice thing about Xpra is that when nothing is updating, no bandwidth is used; however the speaker continues to send data even when no sound is playing - I'm not sure if this is a bug or not.


Was the video as far off?


Yes, actually. I would say most video/image updates had a similar (250-300ms) range of latency.


2Mbps is a bit high!


My bad...I apparently cannot read the "b/s", so it's actually 256kbps


I'll have an hour or two tomorrow to test this more thoroughly from the spotty Wi-Fi at my university.


EDIT:

Came back to a week old Trunk r10380 Server on my Fedora 21 VM. After reconnecting the sound queue was a bit high and laggy, but after a minute or so of usage the levels went back down, and now the session is good as new. I'm going to stop it and update the server to latest latest to include the improvements, and continue testing with different codecs.

Last edited 2 years ago by J. Max Mena (previous) (diff)

comment:38 Changed 2 years ago by Antoine Martin

Owner: changed from alas to J. Max Mena

Nope. Stayed that way.


Please provide a graph and a sample -d sound output, the heuristics should try to lower the queue levels back down.


however the speaker continues to send data even when no sound is playing


From xpra, we don't know if sound is playing or not, so we just capture and forward all the time. The actual bandwidth used by "no sound" will vary depending on which codec is used, it will be higher now with the lower latency codec tweaks (many more small chunks).

comment:39 Changed 2 years ago by J. Max Mena

Re-attempted a connection(from the same client against the same server) and now the connection is failing from the Win8.1 client with the following messages:

2015-09-02 13:14:32,069 unknown string message: 0xc0f4 / 198 / 0
2015-09-02 13:14:37,088 unknown string message: 0xc0f4 / 198 / 0
2015-09-02 13:14:50,142 unknown string message: 0xc0f4 / 198 / 0
2015-09-02 13:14:51,144 unknown string message: 0xc0f4 / 198 / 0
2015-09-02 13:14:52,312 failed to receive anything, not an xpra server?
2015-09-02 13:14:52,312   could also be the wrong username, password or port
2015-09-02 13:14:52,312   or maybe this server does not support 'unknown' compre
ssion or 'bencode' packet encoding?
2015-09-02 13:14:52,326 Connection lost

EDIT:

Not entirely sure why it's not working, I'll have to re-attempt it in a bit; after my class.

Last edited 2 years ago by J. Max Mena (previous) (diff)

comment:40 Changed 2 years ago by J. Max Mena

Owner: changed from J. Max Mena to alas

Okay I've re-attempted a connection (Mind you this is the exact same client, server, and network location as Monday) several times and now the network no longer allows me to connect from here using SSH. (which is what I was using before..even with encryption) And, I can't test using TCP because I don't have the networking setup properly. I'll attempt again from home on the bare edge of our Wi-Fi. Until then, I think it's best to disregard what I encountered until I can recreate it somewhere else.

comment:41 Changed 2 years ago by onlyjob

Is there a time frame for switching to GStreamer 1.x?
Older GStreamer is going to be removed from Debian and Xpra is affected...
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785859

comment:42 Changed 2 years ago by Antoine Martin

Is there a time frame for switching to GStreamer 1.x?


GStreamer 1.x is now the default on most platforms.

comment:43 Changed 2 years ago by onlyjob

Trunk or 0.15.x or both?

comment:44 Changed 2 years ago by Antoine Martin

Trunk, 0.15.x is in maintenance mode.

Changed 2 years ago by alas

sound buffer graphs with only an xterm and gedit, but with #985 looping xpra info outputting client side

comment:45 Changed 2 years ago by alas

Well, I found at least one more interesting case: I connected with just an xterm and a gedit launched from a second xterm... but when I opened the session info to the graphs page, I ran into the infinite loop xpra info bug of #985... which impacted the sound buffers pretty dramatically (looks like the max buffer levels adjusted pretty well to that madness):

sound buffer graphs with only an xterm and gedit, but with #985 looping xpra info outputting client side

comment:46 Changed 2 years ago by Antoine Martin

@afarr: the interesting part would have come after you stopped getting those messages (closing the session info window), did the sound buffer level ever go back down again?

The bug in #985 is now fixed.
Interestingly, since 0.15.x one cannot simulate those long latency spikes with this switch:

XPRA_FAKE_UI_LOCKUPS=500 xpra attach ...

because the sound is now processed without any UI thread intervention - which is great for overall latency.

You should be able to reproduce those big spikes with the jitter env var from comment:12 though. Or you can try again with the "buggy", pre-r10657 versions. If this is better at generating latency spikes, I'll have to try to emulate that behaviour.

Changed 2 years ago by alas

file-name says it all...

Changed 2 years ago by alas

walked to next room... brief spinners/jitter

comment:47 Changed 2 years ago by alas

@afarr: the interesting part would have come after you stopped getting those messages (closing the session info window), did the sound buffer level ever go back down again?

Uhh... once I close the session info window, I kind of lose the graph...


Anyway... I've here attached a graphical travelogue of a lap around the building with 0.16.0 r10624 osx client (failed to get hold of a post #985 fix client, so I had to go with a pre-bug version) - and left the timestamps on the screenshots, which will hopefully be at least interesting, if not useful, information.

I may be wrong, but the interesting stuff looks like what happens when I get spinners, when the session recovers initially... and how it stabilizes in the minute or so afterward... so I'll post three or four graphs inline... and leave it to you to look through the rest (and thereby not clutter the ticket too crazily).

You might, though, also find the difference between the walking away from the router graph (15.58.37) vs. walking back toward the router (16.06.01) interesting to compare.

In any case, if there are any cases that you'd like me to double-check, let me know (I'm particularly intrigued by what was happening at 16.01.35 ... looks like the graphs when I was inside and approaching the wi-fi routers... perhaps a portion of our floor is particularly porous?).

In any case...

Walking downstairs and out the front door... I started getting some serious spinners:



Then the spinners continued for a while:



Then the session spinners recovered:



... and finally the session and buffers seemed to stabilize:


comment:48 Changed 2 years ago by alas

I ran some more tests - walking away from routers around one corner, then a second, then turning around and coming back.

OSX client 0.16.0 r10624 (no session info bug) v. fedora 21 0.16.0 r10655.

I ran with --opengl=on for one pass (osx latop with Intel Iris graphics card), then without for the second - only to then notice that the 0.16 client doesn't have the OpenGL blacklisted. I did get slightly different results though (was testing just by running cnn videos perhaps different content produced different results?), so I'll attach both sets.

I also tried running the same test with a password enabled and with encryption on, I'll attach those also.

comment:49 Changed 2 years ago by Antoine Martin

only to then notice that the 0.16 client doesn't have the OpenGL blacklisted


Correct, see r9291.
This is due to the number of stability improvements in the OpenGL backend and the lack of maintenance and bugs found in the pixmap one.

Changed 2 years ago by alas

walk-and-return graph travelogue starting point - sitting down

Changed 2 years ago by alas

travelogue point two, up and walking away from one router, toward another - buffer seems to over compensate

Changed 2 years ago by alas

travelogue stop three, stopping near second router - buffer still hasn't re-adjusted

Changed 2 years ago by alas

travelogue point 4, walking past second router and around a corner, signal passing through walls

Changed 2 years ago by alas

travelogue point 5, turn and walk around a second corner and down half a flight of stairs, seeing some spinners but signal stabilizes quickly - buffer copes very well

Changed 2 years ago by alas

travelogue point 6, returning from around second corner and down some stairs; still around one corner - buffer adjusts very well, despite renewed spinners

comment:50 Changed 2 years ago by alas

Looking over those several piles of screenshots, it actually looks like the entire walk across the building (past a secondary router along the way) around a couple of corners and halfway down a flight of stairs has very little impact with one of the tests, the Level and the Max running as close together as any number of the other uninteresting shots previously attached.

Using encryption didn't seem to make the graphs any more interesting.

One of the tests did seem to show the Max bar diverging from the Level bar while walking away from the routers, but recovering as I started walking back toward the routers (Why it was different than the other pass when the encryption was also off I don't know, vicissitudes of the router? My hand was covering the wireless antenna?, or not covering it?)

In any case, here's a quick posting of that somewhat interesting case.

Started with a baseline at rest.

walk-and-return graph travelogue starting point - sitting down


Then started walking, captured this while about halfway between initial and second router.

travelogue point two, up and walking away from one router, toward another - buffer seems to over compensate


As I reached that second router, the Level continued at a relatively stable range, but the Max didn't adjust back down to match.

travelogue stop three, stopping near second router - buffer still hasn't re-adjusted


As I then walked around a corner, there were some latency "hills", at which point the Levels seemed to climb... which the Max seemed to have anticipated well - despite the Level plummeting a few times as the latency triggered some spinners.

travelogue point 4, walking past second router and around a corner, signal passing through walls


As I then walked around another corner, and started walking down some stairs, the Max seemed to very well match the Level, mostly.

travelogue point 5, turn and walk around a second corner and down half a flight of stairs, seeing some spinners but signal stabilizes quickly - buffer copes very well


Walking back up the stairs and to only around one corner, the Max seemed to adjust much better to the Level changes, aside from the occasional spinner.

travelogue point 6, returning from around second corner and down some stairs; still around one corner - buffer adjusts very well, despite renewed spinners


As I then approached the routers the Max seemed to match the Level even better, so I won't bother you with any more graphs to examine.

Changed 2 years ago by alas

buffer level jumps when re-starting sound with control channel

comment:51 Changed 2 years ago by alas

Apologies in advance... but I found another interesting case. Whenever I stop/re-start the sound via control channel, no matter which codec (unsurprisingly), I see a jump in the initial sound buffer levels. Presumably this isn't something that should be expected regularly, but I thought I'd mention it.

buffer level jumps when re-starting sound with control channel

comment:52 Changed 2 years ago by Antoine Martin

Whenever I stop/re-start the sound via control channel, no matter which codec (unsurprisingly), I see a jump in the initial sound buffer levels


When the sound (re)starts, we set the initial level at ~450ms. You would see the same curve if you pop up session info as soon as you start the client.

The downward curve shown above is exactly the type of curve that we want to see when we try to push the sound buffer levels back down. (only difference is that the blue curve might be higher at that point, and we try to push it down)

comment:53 Changed 2 years ago by Antoine Martin

For testing and simulating network conditions, see ticket:999#comment:1

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:54 Changed 2 years ago by ashtonbarr8800

Owner: changed from alas to ashtonbarr8800

I tried with XPRA_SOUND_SOURCE_JITTER=500 on a random mp3 playing website with chrome (​http://www.lesmp3.com/ no video playing), and it looks like the max level adjusts pretty well to the "topography" of the sound buffer levels (in the neighborhood of 500 ms). the min level doesn't seem to be adjusting at all in this case though... for the most part rolling at a flat 0 ms.

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:55 Changed 2 years ago by Antoine Martin

Owner: changed from ashtonbarr8800 to alas

Note: the min level are only adjusted temporarily to prevent underruns, as we can to keep the queue levels low.
@afarr: still waiting for reproducible test cases, or evidence that we should close or even do more on this ticket. Re-assigning to you since @ashtonbarr8800 is not updating this either.

Changed 2 years ago by alas

Attachment: ongoing-wav-with-videos.png added

wav, listening to videos on youtube, with latency (frame total) at +/-157

Changed 2 years ago by alas

wav, listening to videos on youtube, after latency (frame total) drops back to +/- 105

Changed 2 years ago by alas

wav, at time when 125ms 100ms 23percent tc delay ended

Changed 2 years ago by alas

wav as latency drops after ending tc delay

Changed 2 years ago by alas

wav, as latency rises without tc being used

comment:56 Changed 2 years ago by alas

As long as I have been having issues with osx clients with no mp3 (#970 - comment 19) and fedora 21 servers with no vorbis (#1052) - decided to test the wav heuristics a bit.

Looks like there are issues when using wav codec with latency (total frames) of 150 or more.

Testing using a tc setting of tc qdisc add dev eth0 root netem delay 125ms 100ms 23%, I ran into the following overrun and less overrun graphs with wav:

With latency higher-


wav, listening to videos on youtube, with latency (frame total) at +/-157


As latency drops-


wav, listening to videos on youtube, after latency (frame total) drops back to +/- 105


Then, once the tc delays are stopped-

wav, at time when 125ms 100ms 23percent tc delay ended


And, as the latency settles back toward usual-

wav as latency drops after ending tc delay


And then, interestingly, as the latency rises again without turning to tc:

wav, as latency rises without tc being used


Not sure how often wav should be expected to be used in a network setting with latencies like this, but it definitely looked like an area where the heuristics seem to be missing a beat.

Checking a number of different tc settings with a connection using mp3, however, showed little of any (new) interest. I'll go through more thoroughly and then also try with a windows client a bit (with mp3 and wav and vorbis, if I can set up a server that will run it) and see if I find anything interesting.

For the most part though, it looks like the overrun buffer heuristics are doing well with un-crazy scenarios.

comment:57 Changed 23 months ago by Antoine Martin

Follow ups: #970, #1074. #1075

comment:58 Changed 22 months ago by alas

One last couple of notes (Some last couple of...) — I won't bother to clutter with more screenshots though.

Testing some more with 0.17.0 clients (win32 r11649 (1 change) = 11653; osx r11687) against a 0.17.0 server (fedora 23 r11692); using tc to induce delay and drop.


mp3

  • mp3 w/delay: Buffers generally look good... no real sign of stutters/overruns until about 40 ms delay ... and even with 50 ms 50 ms 25% the buffers are mostly good and only occasional overruns/stutters (often accompanied by spinners... oddly better sync'd than with 3% drop). win32/osx client about the same. Hard to regularly induce 'hiccups' even at 80ms 50ms 35% delay.
  • mp3 w/drop/loss: Win32/osx similar... Buffers look good up to 3%, where only very occasional 'hiccups' begin to be seen. At 5% they are still infrequent, but worse when they hit... at 7% not common, but often very bad when they hit, & often accompanied by spinners... At 7% sometimes plays well for a significant while, but when it hits some bad drop it becomes a storm.

vorbis

  • vorbis w/delay: Win32/osx clients handle about the same - delay seems to be handled by max-buffer up to 40ms, at 50ms some occasional overruns/hiccups are noticeable if one listens long enough, though the Session Info graph doesn't seem to indicate any overruns... and even at 50ms 25ms 25% (or 50ms 50ms 25%) overruns/hiccups are rare to occasional (except where accompanied by spinners). Sound is still rather stable, with only relatively infrequent 'hiccups' with delay set as high as 80ms 50ms 35%.
  • vorbis w/drop/loss: Drops of up to 3% are well handled by win32/osx clients, with the graphs indicating max buffer is adjusting pretty well. At 3% drop either client will occasionally have sound 'hiccups', if sound is running long enough (maybe 5 long minutes before any 'hiccups' are heard). At 4% drop the osx client seems to have a more noticeable uptick in 'hiccups' than the win32 client, with the max buffer looking like it's adjusting pretty closely (sometimes a 'hiccup is heard when the "level" and the "max" intersect, without the "level" seeming to cross the "max"), but by 5% drop they both begin to have some noticeable 'hiccups', though sound can often run for several minutes between with no problems. At 7-8% loss the buffer graphs still look pretty good, but sound will cut out with regular 'hiccups' (not to mention spinners).

opus (only packaged with the win32 client I was testing with)

  • opus w/delay: Up to 40 ms, delay doesn't produce any sign of drops; at 50ms delay, the Session Info seems to be indicating overruns, but the sound still seems perfect to my not-very-sensitive ear. With 50ms 25ms 25% I start getting hiccups/stutters... mostly in conjunction with spinners, but even up to 80ms 50ms 35% the hiccups/stutters aren't very apparent ... until the tc hits a hot streak and the "level" graph starts dropping to 0 periodically (sometimes inducing spinners, sometimes just a sound stutter).
  • opus w/loss/drop: With loss 3%, some very occasional 'hiccups' are heard, butffer still looks good though. At 4% the 'hiccups' come often enough to be noticeable. Buffers continue to look good and 'hiccups' don't seem to come much more often, or be much more severe, through 7% loss. At 8% spinners are common, and sound drops with them (but otherwise generally pretty good, and graphs likewise...)

comment:59 Changed 22 months ago by alas

Resolution: fixed
Status: newclosed

Ok, I don't think there's anything left to deal with in this ticket... any new issues will want a new ticket at this point.

Closing.

comment:60 Changed 3 weeks ago by Antoine Martin

See also: #1617.

Note: See TracTickets for help on using tickets.