We should be able to process sound-data packets to allow sound forwarding to the HTML5 client.
Web Audio API's decodeAudioData doesn't seem to decode an Xpra mp3 frame, at least in Chrome 42.
Right now I am experimenting with aurora.js which is able to decode wav, flac, mp3 among others in javascript.
However, the packets have to be processed in the main page, essentially in the same thread as drawing. I expect this will have a significant impact on performance...
r9229 adds some more implementation details.
We pass the packet data received from the sound-data packets to aurora.js.
The mp3 decoder must be forced to enagage https://github.com/audiocogs/aurora.js/issues/48
There is no sound output - Firefox doesn't show any errors but Chrome shows buffer underflow.
It looks like when we ask for mp3, Xpra is sending an mpeg frame per sound-data packet.
I'm not sure how to proceed.
On a side note - the Web Audio API is completely useless.
Use the /index.html?sound=true
flag to turn on the sound "support"
If you enable sound with r9230 you'll notice that we log the aurora Player.format property.
After the second sound-data payload, enough data is available to correctly identify the stream format:
{ formatID: "mp3", sampleRate: 44100, channelsPerFrame: 2, bitrate: 32000, floatingPoint: true }
Okay, I think that it makes sense to get sound support working with the simpler wav format first.
r9232 + r9233 switches the supported sound decoder to wav.
The current implementation works with wav for a window that has already opened before we connect - but when opening a new window it seems to stop processing the stream.
This suggests that whatever we get in the mp3 sound-data packets is not going to decode without some processing on the client side - are we guaranteed a complete mp3 frame from an Xpra sound-data packet?
@joshiggins: no guarantees given or implied... that said, yes, you should be getting one mp3 frame (or more).
I don't understand why opening a new window can affect the sound.
@antoine: it's most definitely a problem with the html5 client. Initialising the broadway.js decoder for a window after aurora.js has started causes it to stop. But if the window is initialised before aurora - they play nicely. Will have to think carefully about a fix for this...
Is there anything worthy of mention in the 0.16 release? If not, can we schedule this for 0.17?
@maxmylyn: what's the status of this one?
As far as I can tell, it's not enabled as of r12379.
And, the menu option for the buggy sound feature has been removed as well.
EDIT: As far as I can remember, we(the Xpra project) never quite got it working.
Re-scheduling.
Milestone renamed
A lot of work on this has been recorded in #1341 and #1194, see ticket:1341#comment:7 onwards. More in r14443.
We now have a range of codecs, both native ones (via mediasource API - native decoding) and legacy ones (via aurora - javascript decoding). We (pre-)select the "best" one: native first, high quality first, then the rest.
There seems to be something wrong with "legacy: mp3" and "legacy: flac" (aurora plugins) so I've commented those out for now. We should be able to get those to work wherever "legacy: wav" already works - which is everywhere?. Looks like this is the issue referred to in comment:1: WS_NOTES. I've tried switching to that version, but I must be missing something as it didn't help. It also has the websocket in a worker.
This thread: aurora.js issue 133: How to play the live stream delay less talks about buffering and reducing latency.
aurora issue 170: Modernizing Aurora also discusses streams and buffering.
Maybe we should switch to SoundManager2. (but not for this release..)
Now for the fun part: the mediasoure API is supposed to tell us what is supported, and we already have a blacklist on top of that... but it looks like we may need to blacklist some more based on what platform we're running on? (sigh) Or maybe in the case of the mpeg4 container, we should tune it better to make it work? (as it is meant to be decoded by the OS libraries)
Platform | Codecs Working | Codecs Broken? |
---|---|---|
Win7 + Chrome | mp3, webm: opus, vorbis, legacy: wav | |
Win7 + IE 11 | none! | legacy: all |
Win7 + Firefox 50 | legacy: wav, | mpeg4: aac, mp3 |
XP + Firefox 50 | legacy: wav | |
Fedora 25 + Chrome 54 | mp3, web: opus, vorbis - legacy: wav | |
Fedora 25 + Firefox 49 | mpeg4: aac, mp3 (not reliably?) - legacy: wav | |
OSX 10.9.x + Safari 9.1.3 | not sure - none? |
Win 8.1: dunno, VM is having sound problems. Win 10 also.
@afarr: can you get any sound to work with IE? (any other combinations you can test would be a bonus)
can you get any sound to work with IE?
No :(
Checked:
All available codecs, none are working. Interestingly, the LEGACY option prints like it's working fine, but I get no sound. As for the mpeg4 option - I get an error complaining about an unknown MIME type - MEDIA12899: AUDIO/VIDEO: Unknown MIME type.
. And, of course, no sound.
Just a thought, maybe some browsers need us to either turn off XPRA_SOUND_BUNDLE_METADATA
or chunk the packets so that the metadata headers travel in their own chunk.
r14488 adds the broken combinations to our growing blacklist. (which means no sound with Microsoft browsers for now - I don't have time for this crap)
All the remaining combinations should work, if not please shout!
Upped server to r14493 and walked through the available codecs on my available browsers
OS+Browser | Option | Result? |
Fedora 24 Firefox | mpeg4: aac | No sound whatsoever - although I heard a beep when I backspaced on an xterm when I shouldn't have |
Fedora 24 Firefox | mpeg4: mp3 | I hear one frame's worth of sound, but that's it - Firefox indicates sound is playing but I hear nothing |
Fedora 24 Firefox | legacy: wav | works |
Fedora 24 Chrome | mp3 | works |
Fedora 24 Chrome | webm: opus | works |
Fedora 24 Chrome | webm: vorbis | works |
Fedora 24 Chrome | legacy: wav | works |
Windows 8 Firefox | legacy: wav | works |
At this point I had to reboot my laptop as Chrome was broken because it doesn't like new monitors, so I took the opportunity to close Firefox on the remote machine and opened up Chrome to listen to Pandora..a quick reboot fixed local Chrome..moving along..
Windows 8 Chrome | mp3 | works |
Windows 8 Chrome | webm: opus | works |
Windows 8 Chrome | webm: vorbis | works |
Windows 8 Chrome | legacy: wav | works |
Looks like Firefox in Fedora is problematic - I'll continue on with OSX in a minute due to my poor choices. (should've daemonized the server..whatever)
And, moving on to OSX (OSX 10.11.5 Late 2013 Mac Pro aka Trashcan):
OSX Firefox | mpeg4: aac | Firefox indicates sound is being played, but I hear nothing |
OSX Firefox | mpeg4: mp3 | Same as aac |
OSX Firefox | legacy: wav | works |
OSX Chrome | mp3 | works |
OSX Chrome | webm: opus | works |
OSX Chrome | webm: vorbis | works |
OSX Chrome | legacy: wav | works |
OSX Safari | mpeg4: aac | doesn't work, and doesn't indicate if sound is being played |
OSX Safari | mp3 | I get one frame of sound, then silence |
OSX Safari | wav | causes the page to continuously refresh until Safari crashes - definitely blacklist this one |
OSX Safari | legacy: wav | works |
I'm seeing similar behavior in Safari and Firefox as in Linux - most of the options don't work, but the legacy wav option works fine.
Updated in r14494:
All the non-working codecs have now been blacklisted - all that remain, while limited, work fine.
Passing back to you for either more work or for closure.
Sounds like we have some problems: ticket:1341#comment:15 Please record accurately:
gst-plugins-bad-1.0
And also, is the default sound plugin we choose the best possible one? (for each browser / OS)
The sound issues cited in #1341 are no longer repro'able (I'll put some more details into #1341, but refrain from cluttering this with repetitious details).
As for the question of the "best possible" sound plugin... I'm not sure my testing gives me very reliable results (18-23 ms average ping time, over low bandwidth network wifi connection, through vpn, back to my usual network... measured after most all other users have stepped away from their keyboards in favor of a foozball {sp?} table). That said, under these conditions the mp3 sound is as stable as the connection itself, while the vorbis and opus seem to contribute to a quicker client ping timeout result... while wav cuts out in a matter of seconds and often triggers a ping timeout inside of a minute (with opus/vorbis it is often several minutes before a client ping timeout... and running mp3 sometimes remains stable for 10s of minutes).
Assuming it's feasible, I'd probably output warnings with wav if the pings are anything less that really really ideal, and give warnings with vorbis and/or opus if the pings are rather bad. (Exactly what those ranges are is hard to say when I have no clean network to work back from, but if there's a particular debug flag, maybe stats?, that you want me to test with to give you numbers where even opus/vorbis are liable to become problematic, I can do that.)
Mp3 seems like it should be the default for anything less than ideal.
I'll try to scare up some time to test firefox, and maybe even safari... just to quiet the OCD in the back of my head... but I suspect firefox will be similar (safari is anyone's guess).
@afarr: always specify what browser and OS combination was used for testing. We may want to better define the "target network requirements". I think it should be something like 10Mbps with <50ms latency, low packet loss (no wifi). Supporting more extreme network constraints can be done, but it is likely to require some work to support properly. (ie: UDP #639)
vorbis and opus compress better than mp3, so I don't see how this could contribute to bandwidth or congestion issues. Unless I can reproduce it here with a "tc" setup (ie: ticket:999#comment:6), I am tempted to leave the default as it is for now.
Wav is uncompressed... so it will use too much bandwidth in most cases. We support it because we can, and because some users may prefer a lossless codec. Talking about which, it looks like we may have to re-test with native flac too:
Sorry, all the versions client OS and browser and server were also posted in #1341, I guess I thought writing the versions down there would magically duplicate the information here? (Ok, I actually just remembered poorly.)
2.0 r14909 fedora 25 server. OSX 10.12.1 client. Tested with Chrome 56 and Opera 42. Found no issues aside from my own network problems (and with a slightly improved network, I'm not seeing issues with opus or vorbis anymore, so just improving bandwidth, despite all the wifi & vpn & internet & so on issues persisting, is enough to make keeping them as defaults seem the best idea... as a rule of thumb, if you have enough bandwidth for video, you should be able to use vorbis or opus just fine).
In the name of curiosity/OCD, I also gave firefox a try with OSX 10.12.1 ... started with 44.0 (found an old dmg to install), then updated to 51.0.1.
Running against a 2.0 r14995 64-bit fedora 25 server.
In both cases, the only sound option presented was legacy wav.
With the old 44.0 firefox the wav just failed, without any sign of errors — saw this in console logs.
got hello: server version 2.0 accepted our connection Client.js:123:2 audio codecs supported by the server: Array [ "opus", "vorbis+mka", "flac", "mp3", "wav+lz4", "wav", "wavpack", "speex", "flac", "opus+mka", 2 more… ] Client.js:123:2 audio: requesting wav stream from the server Client.js:123:2 startup complete Client.js:123:2 server connection is OK Client.js:588:4 audio start of aurora wav stream Client.js:123:2
Updating to 51.0.1 firefox... I was at least told that there were errors.
Console logs:
server connection is OK Client.js:588:4 audio start of aurora wav stream Client.js:123:2 error processing audio data: TypeError: invalid arguments Stack trace: XpraSource.prototype._on_data@http://10.0.32.138:14500/js/lib/aurora/aurora-xpra.js:28:31 XpraClient.prototype._process_sound_data_aurora@http://10.0.32.138:14500/js/Client.js:1571:2 XpraClient.prototype._process_sound_data@http://10.0.32.138:14500/js/Client.js:1548:4 XpraClient.prototype._route_packet@http://10.0.32.138:14500/js/Client.js:381:3 XpraProtocolWorkerHost.prototype.open/<@http://10.0.32.138:14500/js/Protocol.js:40:6 Client.js:113:2 audio aurora context is closed already, dropping sound buffer
Server logs:
2017-02-06 17:01:38,007 client 2: audio start of aurora wav stream 2017-02-06 17:01:38,135 sound source using audio codec wav 2017-02-06 17:01:41,353 client 2: error processing audio data: TypeError: invalid arguments 2017-02-06 17:01:41,354 client 2: audio aurora context is closed already, dropping sound buffer 2017-02-06 17:01:41,355 client 2: audio aurora context is closed already, dropping sound buffer 2017-02-06 17:01:41,356 client 2: audio aurora context is closed already, dropping sound buffer 2017-02-06 17:01:41,359 sound source stopping 2017-02-06 17:01:41,364 client 2: audio aurora context is closed already, dropping sound buffer
I'll pass this one back to you, in case you want to blacklist the wav on firefox... and in case you have a trick to allow me to try some other codecs with firefox.
I'll pass this one back to you, in case you want to blacklist the wav on firefox... and in case you have a trick to allow me to try some other codecs with firefox
I've just tried it with Firefox 51 in an osx 10.10.x VM and it worked OK by the looks of things (it processes the sound packages without errors, I cannot check the sound output because this VM doesn't have any sound drivers..)
r15004 re-enables "legacy mp3" (mp3 decoding via aurora) - in r15081 for the 1.0 branch.
Hmmm... thought I would be just checking on the legacy:mp3 sound... but found that I wasn't able to get any sound at all, on windows 10, with either firefox 52 or chrome 57, against a 2.1 r15664 fedora 25 server.
It was so weird I double checked with a python client (2.1 r15608 windows from your beta dist), and found sound did work there - so it wasn't just a matter of me breaking my server somehow (it seemed the most likely answer at first).
That said, I'll give you some logs, on the off chance something will be painfully obvious and will fix everything.
With firefox 52 on windows 10, legacy:mp3.
Client console logs:
Using //@ to indicate sourceMappingURL pragmas is deprecated. Use //# instead[Learn More] zlib.js:3740:15 Successfully compiled asm.js code (total compilation time 185ms; unable to cache asm.js in synchronous scripts; try loading asm.js via <script async> or createElement('script')) Decoder.js Successfully compiled asm.js code (total compilation time 3ms; not stored in cache (too small to benefit)) Decoder.js sound enabled, audio codec string: aurora:mp3 index.html:233:9 connection_progress( Initializing , , 20 ) index.html:155:6 audio error probing codec 'audio/mp4; codecs="mp4a.40.2"' / 'audio/mp4; codecs="mp4a.40.2"': ReferenceError: is_safari is not defined Utilities.js:16:3 Utilities.error http://10.0.32.138:14500/js/Utilities.js:16:3 MediaSourceUtil.getMediaSourceAudioCodecs http://10.0.32.138:14500/js/MediaSourceUtil.js:180:6 XpraClient.prototype.init_audio http://10.0.32.138:14500/js/Client.js:184:24 XpraClient.prototype.init http://10.0.32.138:14500/js/Client.js:170:2 <anonymous> http://10.0.32.138:14500/index.html:322:8 resolve/</mightThrow http://10.0.32.138:14500/js/lib/jquery.js:3570:21 resolve/</process< http://10.0.32.138:14500/js/lib/jquery.js:3638:12 audio codec MediaSource NOK: 'mp3' / 'audio/mpeg' Utilities.js:22:3 audio error probing codec 'audio/mp4; codecs="mp3"' / 'audio/mp4; codecs="mp3"': ReferenceError: is_safari is not defined Utilities.js:16:3 Utilities.error http://10.0.32.138:14500/js/Utilities.js:16:3 MediaSourceUtil.getMediaSourceAudioCodecs http://10.0.32.138:14500/js/MediaSourceUtil.js:180:6 XpraClient.prototype.init_audio http://10.0.32.138:14500/js/Client.js:184:24 XpraClient.prototype.init http://10.0.32.138:14500/js/Client.js:170:2 <anonymous> http://10.0.32.138:14500/index.html:322:8 resolve/</mightThrow http://10.0.32.138:14500/js/lib/jquery.js:3570:21 resolve/</process< http://10.0.32.138:14500/js/lib/jquery.js:3638:12 audio codec MediaSource NOK: 'ogg' / 'audio/ogg' Utilities.js:22:3 audio codec MediaSource NOK: 'wav' / 'audio/wav' Utilities.js:22:3 audio codec MediaSource NOK: 'flac' / 'audio/flac' Utilities.js:22:3 audio error probing codec 'audio/webm; codecs="opus"' / 'audio/webm; codecs="opus"': ReferenceError: is_safari is not defined Utilities.js:16:3 audio error probing codec 'audio/webm; codecs="vorbis"' / 'audio/webm; codecs="vorbis"': ReferenceError: is_safari is not defined Utilities.js:16:3 audio codec MediaSource NOK: 'vorbis+ogg' / 'audio/ogg; codecs="vorbis"' Utilities.js:22:3 audio codec MediaSource NOK: 'speex+ogg' / 'audio/ogg; codecs="speex"' Utilities.js:22:3 audio codec MediaSource NOK: 'flac+ogg' / 'audio/ogg; codecs="flac"' Utilities.js:22:3 audio codec MediaSource NOK: 'opus+ogg' / 'audio/ogg; codecs="opus"' Utilities.js:22:3 getMediaSourceAudioCodecs( false )= Array [ ] Utilities.js:22:3 audio codec aurora OK 'wav' / 'lpcm' Utilities.js:22:3 audio codec aurora OK 'mp3' / 'mp3' Utilities.js:22:3 audio codec aurora NOK 'flac' / 'flac' Utilities.js:22:3 using aurora audio codec: mp3 Client.js:164:2 audio codecs: Array [ "wav", "mp3" ] Client.js:164:2 connection_progress( Connecting to server , 10.0.32.138:14500 , 40 ) index.html:155:6 we have webworker support Client.js:330:3 we can use websocket in webworker Client.js:339:5 connection_progress( Opening WebSocket connection , ws://10.0.32.138:14500 , 60 ) index.html:155:6 Using //@ to indicate sourceMappingURL pragmas is deprecated. Use //# instead zlib.js:3740:15 connection_progress( WebSocket connection established , , 80 ) index.html:155:6 forge.md.algorithms= Object { md5: Object, sha1: Object, sha256: Object } Client.js:836:3 digests: Array [ "hmac", "hmac+md5", "xor", "hmac+md5", "hmac+sha1", "hmac+sha256" ] Client.js:840:3 sending hello Client.js:803:3 return all encodings: Array [ "jpeg", "png", "rgb", "rgb32" ] Client.js:735:3 return all encodings: Array [ "jpeg", "png", "rgb", "rgb32" ] Client.js:735:3 _get_keyboard_layout() keyboard_layout= us Client.js:680:2 hello capabilities: [object Object] Client.js:817:2 got hello: server version 2.1 accepted our connection Client.js:164:2 audio codecs supported by the server: Array [ "opus", "vorbis+mka", "flac", "mp3", "wav+lz4", "wav+lzo", "wav", "wavpack", "speex", "flac", 3 more… ] Client.js:164:2 audio: requesting mp3 stream from the server Client.js:164:2 connection_progress( Session started , , 100 ) index.html:155:6 startup complete Client.js:164:2 server connection is OK Client.js:757:4 audio start of aurora mp3 stream Client.js:164:2 lost window 28 , remaining: Array [ "1", "2" ] Client.js:1625:2
Server, however, shows nothing amiss.
2017-04-18 16:13:59,043 xpra client 20 disconnected. 2017-04-18 16:13:59,052 HTML5 Microsoft Windows client version 2.1 2017-04-18 16:13:59,052 automatic picture encoding enabled 2017-04-18 16:13:59,053 also available: 2017-04-18 16:13:59,053 jpeg, png, rgb32 2017-04-18 16:13:59,058 client root window size is 1640x735 with 1 display: 2017-04-18 16:13:59,059 HTML (434x194 mm - DPI: 95x96) 2017-04-18 16:13:59,059 Canvas 2017-04-18 16:13:59,073 server virtual display now set to 1680x1050 (best match for 1640x735) 2017-04-18 16:13:59,086 keyboard mapping already configured (skipped) 2017-04-18 16:13:59,115 DPI set to 96 x 96 2017-04-18 16:13:59,155 client 21: got hello: server version 2.1 accepted our connection 2017-04-18 16:13:59,157 client 21: audio codecs supported by the server: opus,vorbis+mka,flac,mp3,wav+lz4,wav+lzo,wav,wavpack,speex,flac,opus+mka,flac,mp3+mpeg4 2017-04-18 16:13:59,157 client 21: audio: requesting mp3 stream from the server 2017-04-18 16:13:59,254 client 21: startup complete 2017-04-18 16:13:59,276 using pulseaudio device: 2017-04-18 16:13:59,278 'Monitor of Xpra Speaker' 2017-04-18 16:13:59,322 the remote printer 'HTML5 client' has been configured 2017-04-18 16:14:00,117 sound source using audio codec mp3 2017-04-18 16:14:00,120 client 21: audio start of aurora mp3 stream
The firefox 52 windows 10 legacy:wav is very pretty with the yellow and red log output... I'll just add a snippet, since I assume you know it will fail.
Client console:
got hello: server version 2.1 accepted our connection Client.js:164:2 audio codecs supported by the server: Array [ "opus", "vorbis+mka", "flac", "mp3", "wav+lz4", "wav+lzo", "wav", "wavpack", "speex", "flac", 3 more… ] Client.js:164:2 audio: requesting wav stream from the server Client.js:164:2 connection_progress( Session started , , 100 ) index.html:155:6 startup complete Client.js:164:2 server connection is OK Client.js:757:4 lost window 30 , remaining: Array [ "1", "2" ] Client.js:1625:2 audio start of aurora wav stream Client.js:164:2 error processing audio data: TypeError: invalid arguments Stack trace: XpraSource.prototype._on_data@http://10.0.32.138:14500/js/lib/aurora/aurora-xpra.js:28:31 XpraClient.prototype._process_sound_data_aurora@http://10.0.32.138:14500/js/Client.js:1893:4 XpraClient.prototype._process_sound_data@http://10.0.32.138:14500/js/Client.js:1867:4 XpraClient.prototype._route_packet@http://10.0.32.138:14500/js/Client.js:453:3 XpraProtocolWorkerHost.prototype.open/<@http://10.0.32.138:14500/js/Protocol.js:46:6 Client.js:154:2 XpraClient.prototype.error http://10.0.32.138:14500/js/Client.js:154:2 XpraClient.prototype._process_sound_data http://10.0.32.138:14500/js/Client.js:1871:3 XpraClient.prototype._route_packet http://10.0.32.138:14500/js/Client.js:453:3 XpraProtocolWorkerHost.prototype.open/< http://10.0.32.138:14500/js/Protocol.js:46:6 audio aurora context is closed already, dropping sound buffer Client.js:159:2 server connection is not responding, drawing spinners... Client.js:755:4 server connection is OK ...
Server.
windows 10 firefox 52 legacy wav fedora 25 server 2.1 r15664 2017-04-18 15:11:34,759 client 8: got hello: server version 2.1 accepted our connection 2017-04-18 15:11:34,759 client 8: audio codecs supported by the server: opus,vorbis+mka,flac,mp3,wav+lz4,wav+lzo,wav,wavpack,speex,flac,opus+mka,flac,mp3+mpeg4 2017-04-18 15:11:34,764 client 8: audio: requesting wav stream from the server 2017-04-18 15:11:34,831 the remote printer 'HTML5 client' has been configured 2017-04-18 15:11:34,832 using pulseaudio device: 2017-04-18 15:11:34,832 'Monitor of Xpra Speaker' 2017-04-18 15:11:34,901 client 8: startup complete 2017-04-18 15:11:35,184 client 8: audio start of aurora wav stream 2017-04-18 15:11:35,595 sound source using audio codec wav 2017-04-18 15:11:36,931 client 8: error processing audio data: TypeError: invalid arguments 2017-04-18 15:11:36,936 client 8: audio aurora context is closed already, dropping sound buffer 2017-04-18 15:11:36,942 sound source stopping 2017-04-18 15:11:36,955 client 8: audio aurora context is closed already, dropping sound buffer 2017-04-18 15:11:36,955 client 8: audio aurora context is closed already, dropping sound buffer
Testing the same with the same fedora 25 2.1 r15664 server and the same windows 10 OS, but with chrome 57, I'm getting nearly the same results.
With opus. Client console logs.
got hello: server version 2.1 accepted our connection Client.js:164 audio codecs supported by the server: ["opus", "vorbis+mka", "flac", "mp3", "wav+lz4", "wav+lzo", "wav", "wavpack", "speex", "flac", "opus+mka", "flac", "mp3+mpeg4"] Client.js:1207 audio waiting for source open event on [object MediaSource] index.html:155 connection_progress( Session started , , 100 ) Client.js:164 startup complete Client.js:164 audio media source open Client.js:164 using audio codec string for opus+mka: audio/webm; codecs="opus" Client.js:164 audio: requesting opus+mka stream from the server Client.js:757 server connection is OK Client.js:164 audio start of mediasource opus+mka stream Client.js:1198 audio play! Client.js:154 audio buffer error: null XpraClient.error @ Client.js:154 audio_error @ Client.js:1174 (anonymous) @ Client.js:1237 Client.js:164 close_audio_mediasource: audio_source_buffer=[object SourceBuffer], media_source=[object MediaSource], video=[object HTMLAudioElement] Client.js:1281 Uncaught (in promise) DOMException: The play() request was interrupted by a new load request. XpraClient._close_audio_mediasource @ Client.js:1281 XpraClient.close_audio @ Client.js:1248 audio_error @ Client.js:1182 (anonymous) @ Client.js:1237 Client.js:154 audio error XpraClient.error @ Client.js:154 audio_error @ Client.js:1180 (anonymous) @ Client.js:1199 Client.js:164 close_audio_mediasource: audio_source_buffer=null, media_source=null, video=null Client.js:1625 lost window 32 , remaining: ["1", "2"]
Server.
2017-04-18 14:58:34,179 client 2: audio start of mediasource opus+mka stream 2017-04-18 14:58:35,060 sound source using audio codec opus 2017-04-18 14:58:35,060 sound source using container format matroska 2017-04-18 14:58:36,000 client 2: audio buffer error: null 2017-04-18 14:58:36,006 client 2: close_audio_mediasource: audio_source_buffer=[object SourceBuffer], media_source=[object MediaSource], video=[object HTMLAudioElement] 2017-04-18 14:58:36,006 client 2: audio error 2017-04-18 14:58:36,006 client 2: close_audio_mediasource: audio_source_buffer=null, media_source=null, video=null 2017-04-18 14:58:36,012 sound source stopping
Chrome 57 with vorbis. Client console log.
got hello: server version 2.1 accepted our connection Client.js:164 audio codecs supported by the server: ["opus", "vorbis+mka", "flac", "mp3", "wav+lz4", "wav+lzo", "wav", "wavpack", "speex", "flac", "opus+mka", "flac", "mp3+mpeg4"] Client.js:1207 audio waiting for source open event on [object MediaSource] index.html:155 connection_progress( Session started , , 100 ) Client.js:164 startup complete Client.js:1625 lost window 33 , remaining: ["1", "2"] Client.js:164 audio media source open Client.js:164 using audio codec string for vorbis+mka: audio/webm; codecs="vorbis" Client.js:164 audio: requesting vorbis+mka stream from the server Client.js:757 server connection is OK Client.js:164 audio start of mediasource vorbis+mka stream Client.js:1198 audio play! Client.js:154 audio buffer error: [object MediaError] XpraClient.error @ Client.js:154 audio_error @ Client.js:1174 (anonymous) @ Client.js:1237 Client.js:154 SRC_NOT_SUPPORTED XpraClient.error @ Client.js:154 audio_error @ Client.js:1176 (anonymous) @ Client.js:1237 Client.js:164 close_audio_mediasource: audio_source_buffer=[object SourceBuffer], media_source=[object MediaSource], video=[object HTMLAudioElement] Client.js:1277 Uncaught ReferenceError: me is not defined at XpraClient._close_audio_mediasource (Client.js:1277) at XpraClient.close_audio (Client.js:1248) at audio_error (Client.js:1182) at SourceBuffer.<anonymous> (Client.js:1237) XpraClient._close_audio_mediasource @ Client.js:1277 XpraClient.close_audio @ Client.js:1248 audio_error @ Client.js:1182 (anonymous) @ Client.js:1237 Client.js:164 audio media source open Client.js:164 using audio codec string for vorbis+mka: audio/webm; codecs="vorbis" Client.js:154 audio setup error for 'audio/webm; codecs="vorbis"': DOMException: Failed to execute 'addSourceBuffer' on 'MediaSource': The MediaSource's readyState is not 'open'. at MediaSource.<anonymous> (http://10.0.32.138:14500/js/Client.js:1226:26) XpraClient.error @ Client.js:154 (anonymous) @ Client.js:1228 Client.js:164 close_audio_mediasource: audio_source_buffer=[object SourceBuffer], media_source=[object MediaSource], video=[object HTMLAudioElement]
Server, doesn't seem to notice any problems.
2017-04-18 16:45:44,915 new client (this session does not allow sharing) 2017-04-18 16:45:44,962 xpra client 23 disconnected. 2017-04-18 16:45:44,968 HTML5 Microsoft Windows client version 2.1 2017-04-18 16:45:44,971 automatic picture encoding enabled 2017-04-18 16:45:44,971 also available: 2017-04-18 16:45:44,971 jpeg, png, rgb32 2017-04-18 16:45:44,975 client root window size is 1270x586 with 1 display: 2017-04-18 16:45:44,976 HTML (336x155 mm - DPI: 96x96) 2017-04-18 16:45:44,976 Canvas 2017-04-18 16:45:44,989 server virtual display now set to 1280x682 (best match for 1270x586) 2017-04-18 16:45:44,994 keyboard mapping already configured (skipped) 2017-04-18 16:45:45,024 DPI set to 96 x 96 2017-04-18 16:45:45,229 client 24: got hello: server version 2.1 accepted our connection 2017-04-18 16:45:45,237 client 24: audio codecs supported by the server: opus,vorbis+mka,flac,mp3,wav+lz4,wav+lzo,wav,wavpack,speex,flac,opus+mka,flac,mp3+mpeg4 2017-04-18 16:45:45,260 client 24: startup complete 2017-04-18 16:45:45,262 client 24: audio media source open 2017-04-18 16:45:45,262 client 24: using audio codec string for vorbis+mka: audio/webm; codecs="vorbis" 2017-04-18 16:45:45,263 client 24: audio: requesting vorbis+mka stream from the server 2017-04-18 16:45:45,347 the remote printer 'HTML5 client' has been configured 2017-04-18 16:45:45,401 using pulseaudio device: 2017-04-18 16:45:45,401 'Monitor of Xpra Speaker' 2017-04-18 16:45:46,272 sound source using audio codec vorbis 2017-04-18 16:45:46,637 client 24: audio start of mediasource vorbis+mka stream 2017-04-18 16:45:46,640 Handshake complete; enabling connection 2017-04-18 16:45:46,641 Disconnecting client 10.0.4.61:53464: 2017-04-18 16:45:46,642 new client (this session does not allow sharing) 2017-04-18 16:45:46,661 sound source stopping 2017-04-18 16:45:46,676 xpra client 24 disconnected. 2017-04-18 16:45:46,681 HTML5 Microsoft Windows client version 2.1 2017-04-18 16:45:46,682 automatic picture encoding enabled 2017-04-18 16:45:46,682 also available: 2017-04-18 16:45:46,682 jpeg, png, rgb32 2017-04-18 16:45:46,688 client root window size is 1270x740 with 1 display: 2017-04-18 16:45:46,688 HTML (336x196 mm - DPI: 96x95) 2017-04-18 16:45:46,689 Canvas 2017-04-18 16:45:46,696 server virtual display now set to 1280x762 (best match for 1270x740) 2017-04-18 16:45:46,712 keyboard mapping already configured (skipped) 2017-04-18 16:45:46,722 sound source using container format matroska 2017-04-18 16:45:46,734 DPI set to 96 x 96 2017-04-18 16:45:46,809 client 25: got hello: server version 2.1 accepted our connection 2017-04-18 16:45:46,837 client 25: startup complete 2017-04-18 16:45:46,897 the remote printer 'HTML5 client' has been configured
With Chrome 57, legacy mp3, don't see much in the logs, but not getting sound.
Client console.
sound enabled, audio codec string: aurora:mp3 index.html:155 connection_progress( Initializing , , 20 ) Utilities.js:22 audio codec MediaSource OK 'aac+mpeg4' / 'audio/mp4; codecs="mp4a.40.2"' Utilities.js:22 audio codec MediaSource OK 'mp3' / 'audio/mpeg' Utilities.js:22 audio codec MediaSource NOK: 'mp3+mpeg4' / 'audio/mp4; codecs="mp3"' Utilities.js:22 audio codec MediaSource NOK: 'ogg' / 'audio/ogg' Utilities.js:22 audio codec MediaSource NOK: 'wav' / 'audio/wav' Utilities.js:22 audio codec MediaSource NOK: 'flac' / 'audio/flac' Utilities.js:22 audio codec MediaSource OK 'opus+mka' / 'audio/webm; codecs="opus"' Utilities.js:22 audio codec MediaSource OK 'vorbis+mka' / 'audio/webm; codecs="vorbis"' Utilities.js:22 audio codec MediaSource NOK: 'vorbis+ogg' / 'audio/ogg; codecs="vorbis"' Utilities.js:22 audio codec MediaSource NOK: 'speex+ogg' / 'audio/ogg; codecs="speex"' Utilities.js:22 audio codec MediaSource NOK: 'flac+ogg' / 'audio/ogg; codecs="flac"' Utilities.js:22 audio codec MediaSource NOK: 'opus+ogg' / 'audio/ogg; codecs="opus"' Utilities.js:22 getMediaSourceAudioCodecs( false )= [aac+mpeg4: "audio/mp4; codecs="mp4a.40.2"", mp3: "audio/mpeg", opus+mka: "audio/webm; codecs="opus"", vorbis+mka: "audio/webm; codecs="vorbis""] Utilities.js:22 audio codec aurora OK 'wav' / 'lpcm' Utilities.js:22 audio codec aurora OK 'mp3' / 'mp3' Utilities.js:22 audio codec aurora NOK 'flac' / 'flac' Client.js:164 using aurora audio codec: mp3 Client.js:164 audio codecs: ["aac+mpeg4", "mp3", "opus+mka", "vorbis+mka", "wav"] index.html:155 connection_progress( Connecting to server , 10.0.32.138:14500 , 40 ) Client.js:330 we have webworker support Client.js:339 we can use websocket in webworker index.html:155 connection_progress( Opening WebSocket connection , ws://10.0.32.138:14500 , 60 ) index.html:155 connection_progress( WebSocket connection established , , 80 ) Client.js:836 forge.md.algorithms= Object {md5: Object, sha1: Object, sha256: Object} Client.js:840 digests: ["hmac", "hmac+md5", "xor", "hmac+md5", "hmac+sha1", "hmac+sha256"] Client.js:803 sending hello Client.js:735 return all encodings: ["jpeg", "png", "rgb", "rgb32"] Client.js:735 return all encodings: ["jpeg", "png", "rgb", "rgb32"] Client.js:680 _get_keyboard_layout() keyboard_layout= us Client.js:817 hello capabilities: [object Object] Client.js:164 got hello: server version 2.1 accepted our connection Client.js:164 audio codecs supported by the server: ["opus", "vorbis+mka", "flac", "mp3", "wav+lz4", "wav+lzo", "wav", "wavpack", "speex", "flac", "opus+mka", "flac", "mp3+mpeg4"] Client.js:164 audio: requesting mp3 stream from the server index.html:155 connection_progress( Session started , , 100 ) Client.js:164 startup complete Client.js:757 server connection is OK Client.js:164 audio start of aurora mp3 stream
And, again, no sign of anything amiss on the server side.
2017-04-18 16:53:09,586 Handshake complete; enabling connection 2017-04-18 16:53:09,592 HTML5 Microsoft Windows client version 2.1 2017-04-18 16:53:09,592 automatic picture encoding enabled 2017-04-18 16:53:09,592 also available: 2017-04-18 16:53:09,592 jpeg, png, rgb32 2017-04-18 16:53:09,597 client root window size is 1270x740 with 1 display: 2017-04-18 16:53:09,601 HTML (336x196 mm - DPI: 96x95) 2017-04-18 16:53:09,601 Canvas 2017-04-18 16:53:09,602 best resolution matching 1270x740 is unchanged: 1280x762 2017-04-18 16:53:09,605 keyboard mapping already configured (skipped) 2017-04-18 16:53:09,725 client 26: got hello: server version 2.1 accepted our connection 2017-04-18 16:53:09,725 client 26: audio codecs supported by the server: opus,vorbis+mka,flac,mp3,wav+lz4,wav+lzo,wav,wavpack,speex,flac,opus+mka,flac,mp3+mpeg4 2017-04-18 16:53:09,731 client 26: audio: requesting mp3 stream from the server 2017-04-18 16:53:09,769 client 26: startup complete 2017-04-18 16:53:09,813 using pulseaudio device: 2017-04-18 16:53:09,813 'Monitor of Xpra Speaker' 2017-04-18 16:53:09,840 the remote printer 'HTML5 client' has been configured 2017-04-18 16:53:10,285 client 26: audio start of aurora mp3 stream 2017-04-18 16:53:10,703 sound source using audio codec mp3 2017-04-18 16:53:27,783 New tcp connection received from 10.0.4.61:54186 2017-04-18 16:53:36,300 server virtual display now set to 1280x682 (best match for 1270x287) 2017-04-18 16:53:36,300 received updated display dimensions 2017-04-18 16:53:36,301 client display size is 1270x287 with 1 screen: 2017-04-18 16:53:36,301 HTML (336x76 mm - DPI: 96x95) 2017-04-18 16:53:36,302 Canvas 2017-04-18 16:53:36,310 DPI set to 96 x 96
Interestingly, I'm getting some server side errors with aac+mpeg4 (which I haven't enabled on the server) similar to the ones I was seeing with opus.
2017-04-18 15:14:07,122 client 9: audio codec aac+mpeg4 is not supported by the server 2017-04-18 15:14:07,133 client 9: startup complete 2017-04-18 15:14:07,133 client 9: audio media source open 2017-04-18 15:14:07,135 client 9: using audio codec string for opus+mka: audio/webm; codecs="opus" 2017-04-18 15:14:07,143 client 9: audio: requesting opus+mka stream from the server 2017-04-18 15:14:07,193 the remote printer 'HTML5 client' has been configured 2017-04-18 15:14:07,194 using pulseaudio device: 2017-04-18 15:14:07,194 'Monitor of Xpra Speaker' 2017-04-18 15:14:07,789 client 9: audio start of mediasource opus+mka stream 2017-04-18 15:14:07,855 client 9: audio buffer error: [object MediaError] 2017-04-18 15:14:07,855 client 9: SRC_NOT_SUPPORTED 2017-04-18 15:14:07,855 client 9: close_audio_mediasource: audio_source_buffer=[object SourceBuffer], media_source=[object MediaSource], video=[object HTMLAudioElement] 2017-04-18 15:14:07,855 client 9: audio media source open 2017-04-18 15:14:07,856 client 9: using audio codec string for opus+mka: audio/webm; codecs="opus" 2017-04-18 15:14:07,863 sound source stopping 2017-04-18 15:14:07,889 client 9: audio setup error for 'audio/webm; codecs="opus"': InvalidStateError: Failed to execute 'addSourceBuffer' on 'MediaSource': The MediaSource's readyState is not 'open'. 2017-04-18 15:14:07,895 client 9: close_audio_mediasource: audio_source_buffer=[object SourceBuffer], media_source=[object MediaSource], video=[object HTMLAudioElement] 2017-04-18 15:14:07,915 client 9: audio error: [object MediaError]
Had maxmylyn test with his Fedora against Fedora set up and he confirmed no sound with any html5 clients as well.
Passing this back to you.
Can you test with 2.0? This ticket was created for 1.0, but some of the later changes are in 2.0 only. It is possible that some of the changes from 2.1 caused regressions, in particular #1424.
With firefox 52 on windows 10, legacy:mp3. (..)
audio error probing codec .. ReferenceError: is_safari is not defined
Is fixed in r15665. Not sure if this will fix the audio - just this particular error. This would have affected both Safari and Firefox and cause all native codecs to fail detection.
The firefox 52 windows 10 legacy:wav is very pretty with the yellow and red log output... I'll just add a snippet, since I assume you know it will fail.
Why would I? "legacy:wav" is currently not blacklisted. As I said in comment:26, I'm not getting any errors.
We should have a blacklist for all the codecs that don't work. The remaining ones, the ones shown on the connect dialog page should all work, unless there are bandwidth limitation issues (which may be the case with wav).
With opus. (..)
Client.js:154 audio buffer error: null
(..)Client.js:1281 Uncaught (in promise) DOMException: The play() request was interrupted by a new load request.
Did you navigate away at that point?
Chrome 57 with vorbis. (..)
Client.js:154 SRC_NOT_SUPPORTED
(..)
Looks like chrome doesn't actually support vorbis despite claiming it does? Is this still on Windows 10? Does it behave the same on other platforms?
(..)
Client.js:1277 Uncaught ReferenceError: me is not defined
(..)
This is fixed in r15668, will now show us the real exception message in the output.
Interestingly, I'm getting some server side errors with aac+mpeg4 (which I haven't enabled on the server) similar to the ones I was seeing with opus.
client 9: audio codec aac+mpeg4 is not supported by the server
(..)client 9: audio: requesting opus+mka stream from the server
(..)client 9: SRC_NOT_SUPPORTED
(..)
Those aren't server side errors, they're client-side errors logged on the server via the remote-logging feature. (all prefixed with "client X:") The client is doing the right thing and selecting a different codec "opus+mka" since "aac+mpeg4" is not available on the server.
Seeing that sound used to work, and not just for me - see comment:17 and comment:18, I went back to 1.0 and tested against a Fedora 25 1.0.6-r15694 server:
OS | Browser | Codecs Available | Notes |
---|---|---|---|
Fedora 26 | Chrome 59 | mp3, webm:opus, webm:vorbis, legacy:wav, legacy:mp3 | |
Fedora 26 | Firefox 53 | legacy:mp3, legacy:wav | |
Windows 7 | Chrome 58 | mp3, webm:opus, webm:vorbis, legacy:wav, legacy:mp3 | |
Windows 7 | Firefox 53 | legacy:mp3, legacy:wav | |
Mac OS 10.10.x | Safari 10.1 | legacy:mp3, legacy:wav | wav is completely out of sync |
Mac OS 10.10.x | Chrome 58 | mp3, webm:opus, webm:vorbis, legacy:wav, legacy:mp3 | window paint / transparency issues? |
Mac OS 10.10.x | Firefox 53 | legacy:mp3, legacy:wav |
All the codecs listed as available worked OK, except for "legacy:mp3" which doesn't seem to be working anywhere (no console messages either), so it looks like this one needs to be removed (maybe revert r15081). It did work before, but maybe only in trunk.
There does seem to be a problem with trunk and all the codecs, I'm moving this regression to a separate ticket: #1507.
@afarr: apart from the "legacy" codecs issue, 1.0 should work OK.
Ok, I concur with your OSX and Windows results, testing with OSX 10.12 and Windows 10... as long as your 'OK' includes the caveat that the legacy wav sound tends to drop within 1-15 seconds.
Unfortunately, I seem to be having catastrophic errors with python clients with opengl enabled (including with a non-Intel driver)... both OSX and Windows. I suspect that will call for another ticket though.
Let me just throw together a little chart with my similar results.
All run against a 1.0.6 server ... listing as r14823 (unknown changes), but probably closer to r15649 or so... (I'll look into clearing those files again soon).
OS | Browser | Codecs available | Notes |
Windows 10 | Firefox 52.02 | legacy:mp3, legacy:wav | legacy:wav got "error processing audio data:" within 15 seconds |
Windows 10 | Chrome 57 | mp3, webm:opus, webm:vorbis, legacy:wav, legacy:mp3 | |
OSX 10.12 | Safari 10.0 | legacy:mp3, legacy:wav | |
OSX 10.12 | Chrome 55 | mp3, webm:opus, webm:vorbis, legacy:wav, legacy:mp3 | |
OSX 10.12 | Firefox 50.0.1 | legacy:mp3, legacy:wav | legacy:wav seems more stable than with windows 10 |
I would add that, the legacy:wav on the windows 10 firefox 52 had these server-side logs.
2017-04-24 17:36:14,860 sound source using audio codec wav 2017-04-24 17:36:28,222 client 25: error processing audio data: 2017-04-24 17:36:28,226 client 25: TypeError: invalid arguments 2017-04-24 17:36:28,226 client 25: audio aurora context is closed already, dropping sound buffer 2017-04-24 17:36:28,226 client 25: audio aurora context is closed already, dropping sound buffer 2017-04-24 17:36:28,227 client 25: audio aurora context is closed already, dropping sound buffer
... just in case that's of even the slightest use.
With the legacy:wav, same OS and browser, I also saw an additional error log before the sound broke the same way as with the wav.
2017-04-24 17:38:07,300 client 26: cannot paint, window not found: 2017-04-24 17:38:07,300 client 26: 26
Since the legacy:mp3 is well enough known, and you've made another ticket for the regression... well, I'll pass this back - since I'm not sure you want to keep it open just over the legacy:mp3 or move that to another ticket just for neatness/clarity.
I seem to be having catastrophic errors with python clients with opengl enabled
for reference, the opengl ticket is here: #1508
legacy:wav got "error processing audio data:" within 15 seconds
wav is always going to use up tons of bandwidth, which can starve the client sound output and make it unreliable - so let's not worry about this one too much.
I would add that, the legacy:wav on the windows 10 firefox 52 had these server-side logs.
client 25: TypeError: invalid arguments
Can you capture the client side javascript console debug output? (it should include line numbers and tell us the exact cause of this error)
client 26: cannot paint, window not found:
This error message was invalid, fixed in r15715.
Since the legacy:mp3 is well enough known, and you've made another ticket for the regression... well, I'll pass this back - since I'm not sure you want to keep it open just over the legacy:mp3 or move that to another ticket just for neatness/clarity.
I'm moving "legacy" (aurora software decoding) to a separate ticket: #1509, so we can close this one. (just the TypeError
info needed - this may help with aurora)
Can you capture the client side javascript console debug output? (it should include line numbers and tell us the exact cause of this error)
With a trunk r15716 Server (Fedora 26) connecting from Firefox with legacy:wav
selected, I got the same error as alas:
TypeError: invalid arguments Stack trace: XpraClient.prototype._process_sound_data_aurora@http://192.168.1.138:14500/js/Client.js:1929:5 XpraClient.prototype._process_sound_data@http://192.168.1.138:14500/js/Client.js:1877:4 XpraClient.prototype._route_packet@http://192.168.1.138:14500/js/Client.js:453:3 XpraProtocolWorkerHost.prototype.open/<@http://192.168.1.138:14500/js/Protocol.js:46:6 Client.js:156:2 XpraClient.prototype.error http://192.168.1.138:14500/js/Client.js:156:2 XpraClient.prototype._process_sound_data http://192.168.1.138:14500/js/Client.js:1881:3 XpraClient.prototype._route_packet http://192.168.1.138:14500/js/Client.js:453:3 XpraProtocolWorkerHost.prototype.open/< http://192.168.1.138:14500/js/Protocol.js:46:6
With a trunk r15716 Server (Fedora 26) connecting from Firefox with legacy:wav selected, I got the same error as alas:
As per comment:29, this ticket was meant for 1.0. The javascript trace you just posted lands in code that was added just yesterday, so it's unlikely to lead us to the same bug. I tried to reproduce it, saw it once and never again. Only from a Windows 7 client with Firefox, a local Firefox on Fedora didn't have any problems. Probably bandwidth constraints related.
I've improved the code in trunk some more (see ticket:1509#comment:1), and found that we didn't handle "end-of-stream" (fixed in r15718 - will backport in a different form). This may explain some errors which would happen when we stop the audio stream.
I am closing this ticket because 1.0 is old and it is what it is. If we find a fix for the aurora codecs (#1509), I'll backport it.
same issue When I Use Mp3 Juice. Some Mp3 Song Good Some Half Convert I Think It Is Server Issue.
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/845