xpra icon
Bug tracker and wiki

Opened 2 years ago

Last modified 5 weeks ago

#845 new enhancement

html5 client sound support

Reported by: Josh Owned by: alas
Priority: critical Milestone: 1.0
Component: html5 Version: trunk
Keywords: Cc: aradtech, rektide@…

Description

We should be able to process sound-data packets to allow sound forwarding to the HTML5 client.

Web Audio API's decodeAudioData doesn't seem to decode an Xpra mp3 frame, at least in Chrome 42.

Right now I am experimenting with aurora.js which is able to decode wav, flac, mp3 among others in javascript.

However, the packets have to be processed in the main page, essentially in the same thread as drawing. I expect this will have a significant impact on performance...

Change History (27)

comment:1 Changed 23 months ago by Josh

Milestone: future

r9229 adds some more implementation details.

We pass the packet data received from the sound-data packets to aurora.js.

The mp3 decoder must be forced to enagage https://github.com/audiocogs/aurora.js/issues/48

There is no sound output - Firefox doesn't show any errors but Chrome shows buffer underflow.

It looks like when we ask for mp3, Xpra is sending an mpeg frame per sound-data packet.

I'm not sure how to proceed.


On a side note - the Web Audio API is completely useless.

comment:2 Changed 23 months ago by Josh

Use the /index.html?sound=true flag to turn on the sound "support"

comment:3 Changed 23 months ago by Josh

If you enable sound with r9230 you'll notice that we log the aurora Player.format property.

After the second sound-data payload, enough data is available to correctly identify the stream format:

{ formatID: "mp3", sampleRate: 44100, channelsPerFrame: 2, bitrate: 32000, floatingPoint: true }

comment:4 Changed 23 months ago by Josh

Okay, I think that it makes sense to get sound support working with the simpler wav format first.

r9232 + r9233 switches the supported sound decoder to wav.

The current implementation works with wav for a window that has already opened before we connect - but when opening a new window it seems to stop processing the stream.

This suggests that whatever we get in the mp3 sound-data packets is not going to decode without some processing on the client side - are we guaranteed a complete mp3 frame from an Xpra sound-data packet?

comment:5 Changed 23 months ago by Antoine Martin

@joshiggins: no guarantees given or implied... that said, yes, you should be getting one mp3 frame (or more).

I don't understand why opening a new window can affect the sound.

Last edited 23 months ago by Antoine Martin (previous) (diff)

comment:6 Changed 23 months ago by Josh

@antoine: it's most definitely a problem with the html5 client. Initialising the broadway.js decoder for a window after aurora.js has started causes it to stop. But if the window is initialised before aurora - they play nicely. Will have to think carefully about a fix for this...

comment:7 Changed 17 months ago by Antoine Martin

Milestone: future0.16

Is there anything worthy of mention in the 0.16 release?
If not, can we schedule this for 0.17?

comment:8 Changed 12 months ago by Antoine Martin

@maxmylyn: what's the status of this one?

comment:9 Changed 12 months ago by Antoine Martin

Milestone: 0.160.17
Owner: changed from Josh to maxmylyn

comment:10 Changed 12 months ago by maxmylyn

Owner: changed from maxmylyn to Antoine Martin

As far as I can tell, it's not enabled as of r12379.

And, the menu option for the buggy sound feature has been removed as well.

EDIT: As far as I can remember, we(the Xpra project) never quite got it working.

Last edited 12 months ago by maxmylyn (previous) (diff)

comment:11 Changed 11 months ago by Antoine Martin

Milestone: 0.170.18
Priority: majorcritical
Status: newassigned

Re-scheduling.

comment:12 Changed 9 months ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

comment:13 Changed 4 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas
Status: assignednew

A lot of work on this has been recorded in #1341 and #1194, see ticket:1341#comment:7 onwards.
More in r14443.

We now have a range of codecs, both native ones (via mediasource API - native deoding) and legacy ones (via aurora - javascript decoding).
We (pre-)select the "best" one: native first, high quality first, then the rest.


About aurora

There seems to be something wrong with "legacy: mp3" and "legacy: flac" (aurora plugins) so I've commented those out for now.
We should be able to get those to work wherever "legacy: wav" already works - which is everywhere?. Looks like this is the issue referred to in comment:1: aurora.js issue 146: Continuous decoding which points to this fork: Websocket Fork of Aurora.js, see WS_NOTES. I've tried switching to that version, but I must be missing something as it didn't help. It also has the websocket in a worker.

This thread: aurora.js issue 133: How to play the live stream delay less talks about buffering and reducing latency.

aurora issue 170: Modernizing Aurora also discusses streams and buffering.

Maybe we should switch to SoundManager2. (but not for this release..)


Now for the fun part: the mediasoure API is supposed to tell us what is supported, and we already have a blacklist on top of that... but it looks like we may need to blacklist some more based on what platform we're running on? (sigh)
Or maybe in the case of the mpeg4 container, we should tune it better to make it work? (as it is meant to be decoded by the OS libraries)

PlatformCodecs WorkingCodecs Broken?
Win7 + Chromemp3, webm: opus, vorbis, legacy: wav
Win7 + IE 11none! legacy: all
Win7 + Firefox 50legacy: wav, mpeg4: aac, mp3
XP + Firefox 50legacy: wav
Fedora 25 + Chrome 54mp3, web: opus, vorbis - legacy: wav
Fedora 25 + Firefox 49mpeg4: aac, mp3 (not reliably?) - legacy: wav
OSX 10.9.x + Safari 9.1.3not sure - none?

Win 8.1: dunno, VM is having sound problems. Win 10 also.

@afarr: can you get any sound to work with IE? (any other combinations you can test would be a bonus)

Last edited 4 months ago by Antoine Martin (previous) (diff)

comment:14 Changed 4 months ago by maxmylyn

Owner: changed from alas to Antoine Martin

can you get any sound to work with IE?


No :(


Checked:

  • Win7
  • Win81
  • Win10

All available codecs, none are working. Interestingly, the LEGACY option prints like it's working fine, but I get no sound. As for the mpeg4 option - I get an error complaining about an unknown MIME type - MEDIA12899: AUDIO/VIDEO: Unknown MIME type.. And, of course, no sound.

comment:15 Changed 4 months ago by Antoine Martin

Just a thought, maybe some browsers need us to either turn off XPRA_SOUND_BUNDLE_METADATA or chunk the packets so that the metadata headers travel in their own chunk.

comment:16 Changed 4 months ago by Antoine Martin

Owner: changed from Antoine Martin to maxmylyn

r14488 adds the broken combinations to our growing blacklist.
(which means no sound with Microsoft browsers for now - I don't have time for this crap)

All the remaining combinations should work, if not please shout!

comment:17 Changed 4 months ago by maxmylyn

Owner: changed from maxmylyn to Antoine Martin

Upped server to r14493 and walked through the available codecs on my available browsers

  • Opened up Firefox and set a song to loop and connected and walked through the available codecs and here are my results:
OS+Browser Option Result?
Fedora 24 Firefox mpeg4: aac No sound whatsoever - although I heard a beep when I backspaced on an xterm when I shouldn't have
Fedora 24 Firefox mpeg4: mp3 I hear one frame's worth of sound, but that's it - Firefox indicates sound is playing but I hear nothing
Fedora 24 Firefox legacy: wav works
Fedora 24 Chrome mp3 works
Fedora 24 Chrome webm: opus works
Fedora 24 Chrome webm: vorbis works
Fedora 24 Chrome legacy: wav works
Windows 8 Firefox legacy: wav works

At this point I had to reboot my laptop as Chrome was broken because it doesn't like new monitors, so I took the opportunity to close Firefox on the remote machine and opened up Chrome to listen to Pandora..a quick reboot fixed local Chrome..moving along..

Windows 8 Chrome mp3 works
Windows 8 Chrome webm: opus works
Windows 8 Chrome webm: vorbis works
Windows 8 Chrome legacy: wav works

Looks like Firefox in Fedora is problematic - I'll continue on with OSX in a minute due to my poor choices. (should've daemonized the server..whatever)

comment:18 Changed 4 months ago by maxmylyn

And, moving on to OSX (OSX 10.11.5 Late 2013 Mac Pro aka Trashcan):

OSX Firefox mpeg4: aac Firefox indicates sound is being played, but I hear nothing
OSX Firefox mpeg4: mp3 Same as aac
OSX Firefox legacy: wav works
OSX Chrome mp3 works
OSX Chrome webm: opus works
OSX Chrome webm: vorbis works
OSX Chrome legacy: wav works
OSX Safari mpeg4: aac doesn't work, and doesn't indicate if sound is being played
OSX Safari mp3 I get one frame of sound, then silence
OSX Safari wav causes the page to continuously refresh until Safari crashes - definitely blacklist this one
OSX Safari legacy: wav works

I'm seeing similar behavior in Safari and Firefox as in Linux - most of the options don't work, but the legacy wav option works fine.

Last edited 4 months ago by maxmylyn (previous) (diff)

comment:19 Changed 4 months ago by Antoine Martin

Owner: changed from Antoine Martin to maxmylyn

Updated in r14494:

  • Firefox had worked for me when I tried it with the mpeg4 codecs (which is why only Firefox on win32 was blacklisted until now), but I don't have time to figure out which combination might work - all disabled now
  • the Safari blacklist is now the same as Firefox
  • + "wav" blacklisted for Safari
  • r14495 removes "mp3" for OSX Safari (I assume this means that "mpeg4: mp3" was not an option at all with Safari?)

comment:20 Changed 4 months ago by maxmylyn

Owner: changed from maxmylyn to Antoine Martin

All the non-working codecs have now been blacklisted - all that remain, while limited, work fine.

Passing back to you for either more work or for closure.

comment:21 Changed 3 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas

Sounds like we have some problems: ticket:1341#comment:15
Please record accurately:

  • which browsers / OS are affected
  • the exact codec (ie: mp3 is supported by native and legacy code)
  • for opus, install the gstreamer required plugins: see #1074 which seems to imply that you need gst-plugins-bad-1.0

And also, is the default sound plugin we choose the best possible one? (for each browser / OS)

comment:22 Changed 3 months ago by rektide

Cc: rektide@… added

comment:23 Changed 2 months ago by alas

The sound issues cited in #1341 are no longer repro'able (I'll put some more details into #1341, but refrain from cluttering this with repetitious details).

As for the question of the "best possible" sound plugin... I'm not sure my testing gives me very reliable results (18-23 ms average ping time, over low bandwidth network wifi connection, through vpn, back to my usual network... measured after most all other users have stepped away from their keyboards in favor of a foozball {sp?} table). That said, under these conditions the mp3 sound is as stable as the connection itself, while the vorbis and opus seem to contribute to a quicker client ping timeout result... while wav cuts out in a matter of seconds and often triggers a ping timeout inside of a minute (with opus/vorbis it is often several minutes before a client ping timeout... and running mp3 sometimes remains stable for 10s of minutes).

Assuming it's feasible, I'd probably output warnings with wav if the pings are anything less that really really ideal, and give warnings with vorbis and/or opus if the pings are rather bad. (Exactly what those ranges are is hard to say when I have no clean network to work back from, but if there's a particular debug flag, maybe stats?, that you want me to test with to give you numbers where even opus/vorbis are liable to become problematic, I can do that.)

Mp3 seems like it should be the default for anything less than ideal.

I'll try to scare up some time to test firefox, and maybe even safari... just to quiet the OCD in the back of my head... but I suspect firefox will be similar (safari is anyone's guess).

comment:24 Changed 2 months ago by Antoine Martin

@afarr: always specify what browser and OS combination was used for testing.
We may want to better define the "target network requirements". I think it should be something like 10Mbps with <50ms latency, low packet loss (no wifi).
Supporting more extreme network constraints can be done, but it is likely to require some work to support properly. (ie: UDP #639)

vorbis and opus compress better than mp3, so I don't see how this could contribute to bandwidth or congestion issues. Unless I can reproduce it here with a "tc" setup (ie: ticket:999#comment:6), I am tempted to leave the default as it is for now.

Wav is uncompressed... so it will use too much bandwidth in most cases. We support it because we can, and because some users may prefer a lossless codec. Talking about which, it looks like we may have to re-test with native flac too:

comment:25 Changed 7 weeks ago by alas

Sorry, all the versions client OS and browser and server were also posted in #1341, I guess I thought writing the versions down there would magically duplicate the information here? (Ok, I actually just remembered poorly.)

2.0 r14909 fedora 25 server. OSX 10.12.1 client. Tested with Chrome 56 and Opera 42. Found no issues aside from my own network problems (and with a slightly improved network, I'm not seeing issues with opus or vorbis anymore, so just improving bandwidth, despite all the wifi & vpn & internet & so on issues persisting, is enough to make keeping them as defaults seem the best idea... as a rule of thumb, if you have enough bandwidth for video, you should be able to use vorbis or opus just fine).

In the name of curiosity/OCD, I also gave firefox a try with OSX 10.12.1 ... started with 44.0 (found an old dmg to install), then updated to 51.0.1.

Running against a 2.0 r14995 64-bit fedora 25 server.

In both cases, the only sound option presented was legacy wav.

With the old 44.0 firefox the wav just failed, without any sign of errors — saw this in console logs.

got hello: server version 2.0 accepted our connection Client.js:123:2
audio codecs supported by the server: Array [ "opus", "vorbis+mka", "flac", "mp3", "wav+lz4", "wav", "wavpack", "speex", "flac", "opus+mka", 2 more… ] Client.js:123:2
audio: requesting wav stream from the server Client.js:123:2
startup complete Client.js:123:2
server connection is OK Client.js:588:4
audio start of aurora wav stream Client.js:123:2

Updating to 51.0.1 firefox... I was at least told that there were errors.

Console logs:

server connection is OK  Client.js:588:4
audio start of aurora wav stream  Client.js:123:2
error processing audio data: TypeError: invalid arguments
Stack trace:
XpraSource.prototype._on_data@http://10.0.32.138:14500/js/lib/aurora/aurora-xpra.js:28:31
XpraClient.prototype._process_sound_data_aurora@http://10.0.32.138:14500/js/Client.js:1571:2
XpraClient.prototype._process_sound_data@http://10.0.32.138:14500/js/Client.js:1548:4
XpraClient.prototype._route_packet@http://10.0.32.138:14500/js/Client.js:381:3
XpraProtocolWorkerHost.prototype.open/<@http://10.0.32.138:14500/js/Protocol.js:40:6
  Client.js:113:2
audio aurora context is closed already, dropping sound buffer

Server logs:

2017-02-06 17:01:38,007 client 2: audio start of aurora wav stream
2017-02-06 17:01:38,135 sound source using audio codec wav
2017-02-06 17:01:41,353 client 2: error processing audio data: TypeError: invalid arguments
2017-02-06 17:01:41,354 client 2: audio aurora context is closed already, dropping sound buffer
2017-02-06 17:01:41,355 client 2: audio aurora context is closed already, dropping sound buffer
2017-02-06 17:01:41,356 client 2: audio aurora context is closed already, dropping sound buffer
2017-02-06 17:01:41,359 sound source stopping
2017-02-06 17:01:41,364 client 2: audio aurora context is closed already, dropping sound buffer

I'll pass this one back to you, in case you want to blacklist the wav on firefox... and in case you have a trick to allow me to try some other codecs with firefox.

comment:26 Changed 6 weeks ago by Antoine Martin

I'll pass this one back to you, in case you want to blacklist the wav on firefox... and in case you have a trick to allow me to try some other codecs with firefox


I've just tried it with Firefox 51 in an osx 10.10.x VM and it worked OK by the looks of things (it processes the sound packages without errors, I cannot check the sound output because this VM doesn't have any sound drivers..)

  • r15003 makes it easier to test blacklisted codecs: add "ignore_audio_blacklist=true" to the connect page and the normally blacklisted options will be available
  • r15004 re-enables mp3 via aurora, because I can't remember why it was disabled... and it's better than wav. (and maybe this should go in the 1.x branch)

comment:27 Changed 5 weeks ago by Antoine Martin

r15004 re-enables "legacy mp3" (mp3 decoding via aurora) - in r15081 for the 1.0 branch.

Note: See TracTickets for help on using tickets.