html5 client

[Antoine Martin]

Similar to ​noVNC, and probably re-using some of its code (websockets, etc)

Made much easier by #474

I can start a websockets proxy (on port 8080) that points back to xpra (on port 10000):

./websockify.py --web ./Xpra/trunk/html5/ 8080 localhost:10000

And if I use the new tcp-proxy code:

xpra start :10 --bind-tcp=0.0.0.0:10000 --tcp-proxy=127.0.0.1:8080

When we make websocket request to the xpra port (10000), the http traffic goes through xpra to the proxy then back to xpra as tcp!
Eventually, the websockets proxy code should be moved into xpra, but this is good enough for experimenting and testing.

[Antoine Martin]

experimental html5 client is now capable of showing an xterm window!

[Antoine Martin]

modified bencoder that can talk to an xpra server and handle byte buffers efficiently (original code is MIT licensed)

[Antoine Martin]

What I've used:

• ​websockify (many licenses for all the different parts: LGPL version 3, MPL 2, BSD... see: ​LICENSE.txt)
• bencode.js (modified version - MIT licensed)
• The canvas code is based on ​shapesdemo, itself based on these tutorials: ​A Gentle Introduction to Making HTML5 Canvas Interactive, ​Selectable Shapes Part 2: Resizable, Movable Shapes on HTML5 Canvas.

The license reads: Free to use and distribute at will, so long as you are nice to people, etc. May need clarification... as free licenses are unrestricted, and asking for people to "be nice" may be incompatible with other free licenses!

• For zlib, I've used ​imaya's zlib.js (​english link), which is MIT licensed.

I will attach some code once it is cleaned up.

What still needs to be done:

• move the network code to a ​worker (see: ​Using webworkers)
• use ​FileReaderSync so packets get delivered in the same order as they are received!
• use ​Transferable objects from the worker thread to the ui thread
• instead of invalidating the whole canvas, repaint just the area we have updated using ​CanvasRenderingContext2D.drawImage or ​CanvasRenderingContext2D.putImageData
• handle png, jpeg and webp picture formats (ie: ​receiving image through websocket and ​Display image from blob using javascript and websockets)
• fix compatibility with other browsers (only works on Firefox for now!)
• handle browser window resizing (fit to page)
• handle window decorations using stylesheets
• tell server about the size of our decorations so it can start sending pixel data immediately without having to wait for the client map-event packet which is unnecessary using HTML5 (we can place things precisely wherever we like)
• better UI with connection dialog, options, etc (and add no decorations mode option for full-window content)
• handle focus, clicks and keyboard

And maybe further down the line:

• handle notifications (floating divs?), ​custom cursors, bell events (blink? bell icon?), ​clipboard, etc..
• use ​MediaSource API to pass video data to a video element to support h264/vp8/etc
• use webgl for rendering and csc
• use int protocol aliases to make packets shorter
• move websockify calls inside the xpra process to avoid all the back and forth, extra sockets, etc
• ​lz4 support (after removing the need for node.js from the module)
• implement ​rencode in js

[Antoine Martin]

Initial code merged in r5028: sort of works in chrome and firefox (partial window updates don't work, partial keyboard support, no mouse support, ..)

[Antoine Martin]

Many fixes (see changesets) in: r5029 + r5030 + r5031 + r5032 + r5033

Actually makes this almost usable!

[Antoine Martin]

Many more changes in r5034 to r5066, it looks a lot better:

• we capture all key and mouse events
• keyboard works well enough (including BackSpace)
• added server version check
• focus sort of works - most of the time...
• we can disconnect and restore the page to its original state
• some style effort to make it look less barren
• canvas now resizes to fit browser window size, and we tell the server
• popup mode (launch in window with minimal decorations)
• better debugging (only via javascript console now)
• handle window raise
• allow all params to be set via query string, ie: http://localhost:8080/index.html?host=localhost&port=8080&connect=true (and validate all values to prevent XSS)
• auto-connect option (see example URL above)
• use javascript strict mode
• option to show dialog or not (canvas then uses up all the window space)

[Antoine Martin]

updated screenshot of html5 client running in full-window mode

[Antoine Martin]

r5076 + r5077 + r5078 + r5079 + r5080 add a cython version of the bencoder, which makes it substantially faster (important since the html5 version uses the bencoder rather than rencode)

Running the updated bencode test shows the difference in performance:

test_compare_cython()
results: {'python': 1903, 'cython': 1255} (in milliseconds)

[Antoine Martin]

Many more improvements in r5081 to r5089, including support for fullscreen and maximized windows, window top bar icons, etc..

Sadly, I've come to realize that the way I have used the canvas is all wrong: each window should be its own DIV instead, and we can manage:

• decorations using regular CSS
• focus using CSS z-index
• window contents using a canvas
• maximizing/fullscreen using CSS
• buttons and events using regular javascript events instead of re-wiring clicks and events ourselves (then we can more easily have hover, etc)
• redraws are then managed by the browser (much faster)

etc

But I have to leave this for now... it is what it is.

[Antoine Martin]

Needs more time than I can afford to spend on it. Re-scheduling.

Using a CSS decorations on a canvas-per-window is a must though.

[Antoine Martin]

work by Joshua Higgins to make this use a canvas within a div for each window (much much better!)

[Josh]

superseding my previous patch, cleaned up, many improvements

[Josh]

Attached updated patch

• better CSS handling with styles for window types
• basic z-index stacking
• edge resizing
• shape.js is no longer required

[Antoine Martin]

slightly modified patch

[Antoine Martin]

The patch above makes the following changes on your patch:

• ignored changes in util.js and websock.js
• deleted some old commented out code
• removed empty function pass() (not referenced anywhere?)
• re-indented form section in index.html since it is mostly unchanged and removed extra closing div
• I've kept the early exit in do_send_new_screen_size (I seem to remember it avoided errors when closing the connection?)

I did this as I was reviewing the code.
I then tried to run it but it must have got stuck... will try again later.

[Antoine Martin]

Note: as part of #614, r6934 adds support for YAML as packet encoding.

@joshiggins: Any HTML5 updates?

[Josh]

Replying to totaam:

@joshiggins: Any HTML5 updates?

Adding missing window functions for icon, maximise and actually destroying windows when connection is closed (currently leaves the last drawn windows floating around). Started moving websockets code into a worker. Maybe it would be good to start SVN revisions to make it easier tracking bugs in these patches/?

There is a bug that affects even the unpatched html5 client which causes an invalid packet header error to be thrown with a large number of screen updates, which looks like the buffer in protocol.js to be at fault but nailing it has been proving difficult!

[Antoine Martin]

Adding missing...

Good stuff!

Maybe it would be good to start SVN revisions

svn repo info sent separately

which causes an invalid packet header error to be thrown with a large number of screen updates

Just a guess, but it might be that those are rgb updates compressed with zlib, try turning that off and see if it helps. (see #614)

[Josh]

r7227 merges latest html5-v2.patch from comment 14

[Josh]

Some updates in r7228 + r7230 + r7255 + r7271

• Draw window updates off screen and use requestAnimationFrame to redraw browser canvas
• Maximise and restore windows, destroy client windows on connection close
• Invalid packet header bug fixed by using websock.js more robust receive queue buffer
• r7271 moves communication and packet decoding code into a Webworker

However, using a Websocket within a Webworker is not supported in Firefox and it doesn't look like it's coming anytime soon (see ​https://bugzilla.mozilla.org/show_bug.cgi?id=504553) so the web worker method will only be used if WebKit? is detected.

It's usable at this stage with not so bad performance. In addition to some bits in comment 3 also needs:

• Separate client initialisation code and window styles from index.html
• Fullscreen / override windows need handling properly with CSS
• Send new screen size on browser resize
• Add lz4 support (​https://github.com/pierrec/node-lz4)
• Still needs to draw window icon

Seems to work well in Chrome, Safari and Firefox. Reports from Internet Explorer would be welcome.

[Sam F Beroz]

I'd very much like to try out the recent changes to the html5 client but I'm having an issue with the 0.14.x release (I was able to run the client with 0.13.x). The message I'm seeing in the log file is "no matching packet encoder found!". Looking at the code for 0.14.x I'm not sure the following logic is correct:

svn annotate ​http://xpra.org/svn/Xpra/tags/v0.14.x/src/xpra/net/protocol.py | less -N

    352   6964    antoine     def enable_encoder_from_caps(self, caps):
    353   6983    antoine         opts = packet_encoding.get_enabled_encoders(order=packet_encoding.PERFORMANCE_ORDER)
    354   6982    antoine         for e in opts:
    355   6982    antoine             if caps.boolget("rencode"):
    356   6982    antoine                 self.enable_encoder(e)
    357   6982    antoine                 return True
    358   6982    antoine         log.error("no matching packet encoder found!")
    359   6982    antoine         return False

Especially when comparing it to a prior version of the same method:
svn annotate -r 6981 ​http://xpra.org/svn/Xpra/trunk/src/xpra/net/protocol.py

  6964    antoine     def enable_encoder_from_caps(self, caps):
  6965    antoine         if packet_encoding.use_rencode and caps.boolget("rencode"):
  5076    antoine             self.enable_rencode()
  6965    antoine         elif packet_encoding.use_yaml and caps.boolget("yaml"):
  6964    antoine             self.enable_yaml()
  6965    antoine         elif packet_encoding.use_bencode and caps.boolget("bencode", True):
  6964    antoine             self.enable_bencode()
  6964    antoine         else:
  6969    antoine             log.error("no matching packet encoder found!")
  6969    antoine             return False
  6969    antoine         return True

However both methods are pretty different from how it was coded in 0.13.x. I'm not quite sure what to try next if this isn't the cause of my connection issue. Is the 0.14.x logic correct? Is there something else I should look at? Thanks - Sam

[Antoine Martin]

You're right, that looks completely wrong!
It must have got broke after the testing in #614. Does r7365 + r7374 fix things for you?

[Sam F Beroz]

That looks much better. I'll try it out later today. Thanks - Sam

[aradtech]

Tried it out and it works much better then before , great work! I was wondering if we could get a couple of switches to say make it so app starts full screen with no min/close window perhaps if we are only gonna use it for one app that will use the entire canvas space of the browser area.

[Antoine Martin]

Yeah, just tried it and it works great!
I've mirrored the html code here for easy access: ​http://xpra.org/html5/

I managed to hit this bug though:

ValueError: invalid literal for long() with base 10: '563.991455078125'
2014-09-17 10:20:33,210 failed to parse bencode packet: 6c31363a63...

Which decodes to:

$ python -c 'import binascii;print(binascii.unhexlify("6c31363a63..."))'
l16:configure-windowi1ei563.991455078125ei487.991455078125ei499ei316ed21:encodings.rgb_formatsl4:RGBX4:RGBAeee

So the configure window uses dimensions that are not whole numbers, and that crashes the server bdecoder because that's invalid.

---

Also saw these in the websockify server log (not sure if they are related):

code 400, message Bad request syntax ('\x88\x8f\\\xb9\xac\xc6_Q\xf8\xa7.\xde\xc9\xb2|\xda\xc0\xa9/\xdc\xc8')
code 400, message Bad HTTP/0.9 request type ('\x88\x8f\x0e\x06\xd6I')
code 400, message Bad HTTP/0.9 request type ('\x88\x8f#\xaf\x85\x15')

[Josh]

I think r7677 should fix sending window dimensions that are not whole numbers

[Josh]

@aradtech I like the idea of having a switch to full screen an app with no borders, should be trivial to implement but how would you handle for example, popup windows? Draw them with no chrome but in the position they requested?

[Antoine Martin]

I've made it easier to setup the html client in #689.

It would be nice if the host and port fields could be populated with the host and port from the http request.
Even better would be if it could auto-connect by default, and present a password box if the server requires authentication.

[Josh]

r7994 adds auto connect to host and port from http request, authentication coming soon.

[Antoine Martin]

I don't know how to help debug these issues: I am getting no windows about 30% of the time (just force reload the page), and sometimes I get a window but its contents do not get painted until I move it, the rest of the time it works as expected..

[aradtech]

joshiggins , could you please contact me via email at aradtech@…, we are interested in having you do some more work on the html5 client.

[Josh]

r8710 and r8712 makes the client side protocol handling more stable.

• packets were shifted from the buffer too early, when unlucky this caused random invalid packet errors
• we can now detect websocket in webworker feature support instead of picking up browser version cues
• this is required because Firefox 37+ will get support for this feature and assuming non-webkit browsers did not support it caused the functionality in Chrome to regress when they changed to Blink

It's more stable now. Next up, window drawing will be rationalised to iron out inconsistent behaviour (like sometimes not immediately painting contents), so that we have a solid foundation to start adding more encodings.

[extasic]

Replying to totaam:

Hi, I couldn't get the html5 client to work properly. Can you please check if I did something wrong, or if it might be an error?

In order to reproduce the behavior, I created a Dockerfile (​http://paste.ubuntu.com/10518212/). You can use Docker on Linux or Boot2Docker on Windows or OS X to create an Xpra container.

The image can be build with docker build -t xpratest . and a new container be started with docker run --rm -p 8001:8001 xpratest.

When I connect to the web, I would expect both, xeyes and xterm to be rendered properly. Instead, the pupils of xeyes are not moved but painted over, when a movement occurs (the mouse is moved), and the xterm window stays empty. The result looks like ​https://www.dropbox.com/s/7g8xojwce3z1x3t/Screenshot%202015-03-03%2019.38.29.png?dl=0.

I used the current SVN revision 8743 as well as 8696, which has once already worked before. If I am using an Ubuntu image instead of Debian Jessie, the result is the same.

Do you have any idea what the problem might be?

[Josh]

I couldn't build the supplied Dockerfile correctly. I had to put the packages on a single line and run the Xpra server as root, for some reason the created user wanted to use / as it's home directory.

However, you are missing the python-imaging package.

I use a container for development and I can share mine if you are interested.

[extasic]

Replying to joshiggins:

I couldn't build the supplied Dockerfile correctly. I had to put the packages on a single line and run the Xpra server as root, for some reason the created user wanted to use / as it's home directory.

Maybe you were using an older Docker version? I think I remember some changes on 1.5.

However, you are missing the python-imaging package.

Great, that solved my problem, thank you!

I use a container for development and I can share mine if you are interested.

That would be great, maybe I made some other mistakes.

[Antoine Martin]

However, you are missing the python-imaging package.

Great, that solved my problem, thank you!

In theory, the server should be able to work without python-imaging.. Not well, but it should be able to work using plain rgb encoding. (assuming that the html client handles that)

[Josh]

In theory, the server should be able to work without python-imaging.. Not well, but it should be able to work using plain rgb encoding. (assuming that the html client handles that)

Is there any difference in handling between rgb24 and rgb? the HTML5 client asks for rgb24...

[Josh]

r8746 + r8747 completes refactoring (and makes it the default) so that the client is more javascript-y. The implementation detail is largely unchanged but the following is notable

• r8719 separates client code from the HTML page
• r8745 is a slight improvement in XpraWindow.paint() so that we avoid painting twice
• The connection interface needs to be re-implemented based on callbacks from the new XpraClient object. Right now the behaviour is to connect automatically to the host and port from the HTTP request.

[Josh]

r8767 fixes a bug in the protocol handling that has evaded me for months, that usually manifests itself in windows not being drawn immediately.

[Josh]

r8781 adds jpeg and png support to the HTML5 client.

Right now we use the native Image() object which needs the data base64 encoded - seems like a faff but even so it turned out considerably faster than ​jpgjs or ​png.js.

[extasic]

First of all, thank you for your great job!

When using your latest changeset (r8781), there is a strange issue with some applications. If I run a QT application using wine, all menu elements (file menu, dropdowns, ...) open below the current window, not in front of it as expected. This only occurs when using the html5 client.

Do you have any idea what the problem might be? What information should I provide you in order to locate the issue?

Another issue is about the keyboard. When using a German keyboard layout, I can type all letters and numbers. There are some special characters like the exclamation mark I can type as well, but there is e.g. no key mapped to the period, so I'm unable to type it. Is it a known issue? Is there a way to configure the keyboard usage to include all German keys, maybe even including German "Umlauts"?

[Josh]

r9046 adds a style class for the window type MENU (we only had DROPDOWN_MENU and POPUP_MENU). Hopefully there shouldn't be any more menu elements appearing behind windows.

I've created a ticket for the keyboard problem #837.

[Josh]

H.264 decoding support is stabilising in r9045. It will start to be selected if mutually available since r9049.

Use the new connection interface introduced in r9050 to change the preferred encoding by navigating to ​http://server:port/connect.html

The default behaviour without specifying /connect.html is still to connect automatically. If the connection is not successful, it will redirect to the connection interface.

[extasic]

Replying to joshiggins:

r9046 adds a style class for the window type MENU (we only had DROPDOWN_MENU and POPUP_MENU). Hopefully there shouldn't be any more menu elements appearing behind windows.

Thank you for your update! Unfortunately the problem still persists for my useless. I'd provide a Dockerfile, but this didn't work the last time. All I did was install a current wine version and ran few Qt Applications. None of them (like VLC in the pictures above) worked properly: They displayed all menu items, dropdowns and tooltips behind the window. I had to move the window away in order to see them, and be able to select an entry.

Do you have any other idea what I might try, or what debug / log information I should provide?

I've created a ticket for the keyboard problem #837.

Great, thank you!

[Josh]

@extasic I can't seem to reproduce this with the latest revision.

The wine applications use the DIALOG type hint for popups but these are still correctly placed above the WINDOW type.....

If you can tell me which window class is applied to your popup windows it would be useful. This is easy in chrome just by right clicking and selecting inspect element. It will show the html markup which should specify a class="window-WINDOW_TYPE" for the popup window.

Failing that I can take another stab at the Dockerfile.

[Antoine Martin]

Not had time to look into it, but with the latest trunk I get:

disconnect: invalid packet format, not an xpra client?

It was fine just hours/days ago.

[extasic]

Replying to joshiggins:

@extasic I can't seem to reproduce this with the latest revision.

The wine applications use the DIALOG type hint for popups but these are still correctly placed above the WINDOW type.....

If you can tell me which window class is applied to your popup windows it would be useful. This is easy in chrome just by right clicking and selecting inspect element. It will show the html markup which should specify a class="window-WINDOW_TYPE" for the popup window.

This is how the resulting HTML looks in Chrome. The highlighted Canvas represents a file menu (just like in the VLC picture I posted above). The div above the dialog one represents the main window. So there might be an error in the window type, right? But shouldn't there be something like a z-index anyway? Any workaround for this?

Just to mention it again: This error occurs for me only when running any Qt application in wine. If I run a non-Qt application in wine, or a Qt application directly in Linux, I couldn't reproduce the error. If I use the native xpra client, the error doesn't occur, even when running a Qt application under wine.

[Antoine Martin]

My guess is that those applications use the wrong window type, it probably should not be NORMAL (sadly, lots of applications do that - ie: chrome does). We have some code in the xpra client to try to workaround such broken behaviour.
@joshiggins: maybe the window is "transient-for" or "override-redirect" and you can use that?

[Josh]

Added a workaround in r9138. I was able to reproduce with VLC under Wine, it creates NORMAL override-redirect windows for the menus.

[Antoine Martin]

You can ignore comment:45, I was hitting the wrong test server - I need sleep!

[Josh]

I've uploaded a short video clip demonstrating the current state of H.264 decoding as a result of changes referred to in comment:42

​https://www.youtube.com/watch?v=3yOoGWGyvgM

It's actually pretty good and remains usable.

[aradtech]

I can confirm Josh's changes work on F21.

[aradtech]

And would like to say thank you to Josh for your awesome work on the Xpra Html5 client and our requests.

[extasic]

Replying to joshiggins:

Added a workaround in r9138. I was able to reproduce with VLC under Wine, it creates NORMAL override-redirect windows for the menus.

Works for me too, thank you!

[Josh]

r9148 + r9149 adds support for LZ4 packet compressor

[Antoine Martin]

It's looking good for the release!

I have tagged v0.15.x, so let's move new work to a new ticket: #850

It will be easier to know what was delivered in this release and what remains to be done.

If you are hitting this ticket wanting to report a bug, please open a new ticket instead.

[Antoine Martin]

(update milestone)

[Antoine Martin]

There is now a wiki page for the HTML5 client: wiki/Clients/HTML5.

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/473