xpra icon
Bug tracker and wiki

Opened 4 years ago

Closed 3 years ago

Last modified 3 years ago

#473 closed task (fixed)

html5 client

Reported by: Antoine Martin Owned by: Josh
Priority: major Milestone: 0.15
Component: platforms Version:
Keywords: Cc: Josh

Description (last modified by Antoine Martin)

Similar to noVNC, and probably re-using some of its code (websockets, etc)

Made much easier by #474

I can start a websockets proxy (on port 8080) that points back to xpra (on port 10000):

./websockify.py --web ./Xpra/trunk/html5/ 8080 localhost:10000

And if I use the new tcp-proxy code:

xpra start :10 --bind-tcp=0.0.0.0:10000 --tcp-proxy=127.0.0.1:8080

When we make websocket request to the xpra port (10000), the http traffic goes through xpra to the proxy then back to xpra as tcp!
Eventually, the websockets proxy code should be moved into xpra, but this is good enough for experimenting and testing.

Attachments (6)

xpra-html5-PoC-client.png (164.0 KB) - added by Antoine Martin 4 years ago.
experimental html5 client is now capable of showing an xterm window!
bencode.js (5.4 KB) - added by Antoine Martin 4 years ago.
modified bencoder that can talk to an xpra server and handle byte buffers efficiently (original code is MIT licensed)
xpra-html5.png (441.3 KB) - added by Antoine Martin 4 years ago.
updated screenshot of html5 client running in full-window mode
xprahtml5.patch (26.7 KB) - added by Antoine Martin 4 years ago.
work by Joshua Higgins to make this use a canvas within a div for each window (much much better!)
3junexprahtml5.patch (57.8 KB) - added by Josh 4 years ago.
superseding my previous patch, cleaned up, many improvements
html5-v2.patch (55.2 KB) - added by Antoine Martin 4 years ago.
slightly modified patch

Download all attachments as: .zip

Change History (63)

comment:1 Changed 4 years ago by Antoine Martin

Description: modified (diff)
Owner: changed from Antoine Martin to Antoine Martin
Status: newassigned

comment:2 Changed 4 years ago by Antoine Martin

Description: modified (diff)

Changed 4 years ago by Antoine Martin

Attachment: xpra-html5-PoC-client.png added

experimental html5 client is now capable of showing an xterm window!

Changed 4 years ago by Antoine Martin

Attachment: bencode.js added

modified bencoder that can talk to an xpra server and handle byte buffers efficiently (original code is MIT licensed)

comment:3 Changed 4 years ago by Antoine Martin

What I've used:

The license reads: Free to use and distribute at will, so long as you are nice to people, etc. May need clarification... as free licenses are unrestricted, and asking for people to "be nice" may be incompatible with other free licenses!


I will attach some code once it is cleaned up.

What still needs to be done:

And maybe further down the line:

  • handle notifications (floating divs?), custom cursors, bell events (blink? bell icon?), clipboard, etc..
  • use MediaSource API to pass video data to a video element to support h264/vp8/etc
  • use webgl for rendering and csc
  • use int protocol aliases to make packets shorter
  • move websockify calls inside the xpra process to avoid all the back and forth, extra sockets, etc
  • lz4 support (after removing the need for node.js from the module)
  • implement rencode in js
Last edited 4 years ago by Antoine Martin (previous) (diff)

comment:4 Changed 4 years ago by Antoine Martin

Initial code merged in r5028: sort of works in chrome and firefox (partial window updates don't work, partial keyboard support, no mouse support, ..)

comment:5 Changed 4 years ago by Antoine Martin

Many fixes (see changesets) in: r5029 + r5030 + r5031 + r5032 + r5033

Actually makes this almost usable!

Last edited 4 years ago by Antoine Martin (previous) (diff)

comment:6 Changed 4 years ago by Antoine Martin

Description: modified (diff)

comment:7 Changed 4 years ago by Antoine Martin

Description: modified (diff)

comment:8 Changed 4 years ago by Antoine Martin

Description: modified (diff)

comment:9 Changed 4 years ago by Antoine Martin

Many more changes in r5034 to r5066, it looks a lot better:

  • we capture all key and mouse events
  • keyboard works well enough (including BackSpace)
  • added server version check
  • focus sort of works - most of the time...
  • we can disconnect and restore the page to its original state
  • some style effort to make it look less barren
  • canvas now resizes to fit browser window size, and we tell the server
  • popup mode (launch in window with minimal decorations)
  • better debugging (only via javascript console now)
  • handle window raise
  • allow all params to be set via query string, ie: http://localhost:8080/index.html?host=localhost&port=8080&connect=true (and validate all values to prevent XSS)
  • auto-connect option (see example URL above)
  • use javascript strict mode
  • option to show dialog or not (canvas then uses up all the window space)
Last edited 4 years ago by Antoine Martin (previous) (diff)

Changed 4 years ago by Antoine Martin

Attachment: xpra-html5.png added

updated screenshot of html5 client running in full-window mode

comment:10 Changed 4 years ago by Antoine Martin

r5076 + r5077 + r5078 + r5079 + r5080 add a cython version of the bencoder, which makes it substantially faster (important since the html5 version uses the bencoder rather than rencode)

Running the updated bencode test shows the difference in performance:

test_compare_cython()
results: {'python': 1903, 'cython': 1255} (in milliseconds)
Last edited 4 years ago by Antoine Martin (previous) (diff)

comment:11 Changed 4 years ago by Antoine Martin

Many more improvements in r5081 to r5089, including support for fullscreen and maximized windows, window top bar icons, etc..

Sadly, I've come to realize that the way I have used the canvas is all wrong: each window should be its own DIV instead, and we can manage:

  • decorations using regular CSS
  • focus using CSS z-index
  • window contents using a canvas
  • maximizing/fullscreen using CSS
  • buttons and events using regular javascript events instead of re-wiring clicks and events ourselves (then we can more easily have hover, etc)
  • redraws are then managed by the browser (much faster)

etc

But I have to leave this for now... it is what it is.

Last edited 4 years ago by Antoine Martin (previous) (diff)

comment:12 Changed 4 years ago by Antoine Martin

Milestone: 0.121.0

Needs more time than I can afford to spend on it. Re-scheduling.

Using a CSS decorations on a canvas-per-window is a must though.

Changed 4 years ago by Antoine Martin

Attachment: xprahtml5.patch added

work by Joshua Higgins to make this use a canvas within a div for each window (much much better!)

Changed 4 years ago by Josh

Attachment: 3junexprahtml5.patch added

superseding my previous patch, cleaned up, many improvements

comment:13 Changed 4 years ago by Josh

Attached updated patch

  • better CSS handling with styles for window types
  • basic z-index stacking
  • edge resizing
  • shape.js is no longer required

Changed 4 years ago by Antoine Martin

Attachment: html5-v2.patch added

slightly modified patch

comment:14 Changed 4 years ago by Antoine Martin

Cc: Josh added

The patch above makes the following changes on your patch:

  • ignored changes in util.js and websock.js
  • deleted some old commented out code
  • removed empty function pass() (not referenced anywhere?)
  • re-indented form section in index.html since it is mostly unchanged and removed extra closing div
  • I've kept the early exit in do_send_new_screen_size (I seem to remember it avoided errors when closing the connection?)

I did this as I was reviewing the code.
I then tried to run it but it must have got stuck... will try again later.

comment:15 Changed 3 years ago by Antoine Martin

Note: as part of #614, r6934 adds support for YAML as packet encoding.

@joshiggins: Any HTML5 updates?

comment:16 in reply to:  15 Changed 3 years ago by Josh

Replying to totaam:

@joshiggins: Any HTML5 updates?

Adding missing window functions for icon, maximise and actually destroying windows when connection is closed (currently leaves the last drawn windows floating around). Started moving websockets code into a worker. Maybe it would be good to start SVN revisions to make it easier tracking bugs in these patches/?

There is a bug that affects even the unpatched html5 client which causes an invalid packet header error to be thrown with a large number of screen updates, which looks like the buffer in protocol.js to be at fault but nailing it has been proving difficult!

comment:17 Changed 3 years ago by Antoine Martin

Adding missing...


Good stuff!

Maybe it would be good to start SVN revisions


svn repo info sent separately

which causes an invalid packet header error to be thrown with a large number of screen updates


Just a guess, but it might be that those are rgb updates compressed with zlib, try turning that off and see if it helps. (see #614)

comment:18 Changed 3 years ago by Josh

r7227 merges latest html5-v2.patch from comment 14

comment:19 Changed 3 years ago by Josh

Some updates in r7228 + r7230 + r7255 + r7271

  • Draw window updates off screen and use requestAnimationFrame to redraw browser canvas
  • Maximise and restore windows, destroy client windows on connection close
  • Invalid packet header bug fixed by using websock.js more robust receive queue buffer
  • r7271 moves communication and packet decoding code into a Webworker

However, using a Websocket within a Webworker is not supported in Firefox and it doesn't look like it's coming anytime soon (see https://bugzilla.mozilla.org/show_bug.cgi?id=504553) so the web worker method will only be used if WebKit? is detected.

It's usable at this stage with not so bad performance. In addition to some bits in comment 3 also needs:

  • Separate client initialisation code and window styles from index.html
  • Fullscreen / override windows need handling properly with CSS
  • Send new screen size on browser resize
  • Add lz4 support (https://github.com/pierrec/node-lz4)
  • Still needs to draw window icon


Seems to work well in Chrome, Safari and Firefox. Reports from Internet Explorer would be welcome.

comment:20 Changed 3 years ago by Sam F Beroz

I'd very much like to try out the recent changes to the html5 client but I'm having an issue with the 0.14.x release (I was able to run the client with 0.13.x). The message I'm seeing in the log file is "no matching packet encoder found!". Looking at the code for 0.14.x I'm not sure the following logic is correct:

svn annotate http://xpra.org/svn/Xpra/tags/v0.14.x/src/xpra/net/protocol.py | less -N

    352   6964    antoine     def enable_encoder_from_caps(self, caps):
    353   6983    antoine         opts = packet_encoding.get_enabled_encoders(order=packet_encoding.PERFORMANCE_ORDER)
    354   6982    antoine         for e in opts:
    355   6982    antoine             if caps.boolget("rencode"):
    356   6982    antoine                 self.enable_encoder(e)
    357   6982    antoine                 return True
    358   6982    antoine         log.error("no matching packet encoder found!")
    359   6982    antoine         return False

Especially when comparing it to a prior version of the same method:
svn annotate -r 6981 http://xpra.org/svn/Xpra/trunk/src/xpra/net/protocol.py

  6964    antoine     def enable_encoder_from_caps(self, caps):
  6965    antoine         if packet_encoding.use_rencode and caps.boolget("rencode"):
  5076    antoine             self.enable_rencode()
  6965    antoine         elif packet_encoding.use_yaml and caps.boolget("yaml"):
  6964    antoine             self.enable_yaml()
  6965    antoine         elif packet_encoding.use_bencode and caps.boolget("bencode", True):
  6964    antoine             self.enable_bencode()
  6964    antoine         else:
  6969    antoine             log.error("no matching packet encoder found!")
  6969    antoine             return False
  6969    antoine         return True

However both methods are pretty different from how it was coded in 0.13.x. I'm not quite sure what to try next if this isn't the cause of my connection issue. Is the 0.14.x logic correct? Is there something else I should look at? Thanks - Sam

comment:21 Changed 3 years ago by Antoine Martin

Owner: changed from Antoine Martin to Sam F Beroz
Status: assignednew

You're right, that looks completely wrong!
It must have got broke after the testing in #614. Does r7365 + r7374 fix things for you?

Last edited 3 years ago by Antoine Martin (previous) (diff)

comment:22 Changed 3 years ago by Sam F Beroz

That looks much better. I'll try it out later today. Thanks - Sam

comment:23 Changed 3 years ago by aradtech

Tried it out and it works much better then before , great work! I was wondering if we could get a couple of switches to say make it so app starts full screen with no min/close window perhaps if we are only gonna use it for one app that will use the entire canvas space of the browser area.

comment:24 Changed 3 years ago by Antoine Martin

Owner: changed from Sam F Beroz to Josh

Yeah, just tried it and it works great!
I've mirrored the html code here for easy access: http://xpra.org/html5/

I managed to hit this bug though:

ValueError: invalid literal for long() with base 10: '563.991455078125'
2014-09-17 10:20:33,210 failed to parse bencode packet: 6c31363a63...

Which decodes to:

$ python -c 'import binascii;print(binascii.unhexlify("6c31363a63..."))'
l16:configure-windowi1ei563.991455078125ei487.991455078125ei499ei316ed21:encodings.rgb_formatsl4:RGBX4:RGBAeee

So the configure window uses dimensions that are not whole numbers, and that crashes the server bdecoder because that's invalid.


Also saw these in the websockify server log (not sure if they are related):

code 400, message Bad request syntax ('\x88\x8f\\\xb9\xac\xc6_Q\xf8\xa7.\xde\xc9\xb2|\xda\xc0\xa9/\xdc\xc8')
code 400, message Bad HTTP/0.9 request type ('\x88\x8f\x0e\x06\xd6I')
code 400, message Bad HTTP/0.9 request type ('\x88\x8f#\xaf\x85\x15')
Last edited 3 years ago by Antoine Martin (previous) (diff)

comment:25 Changed 3 years ago by Josh

I think r7677 should fix sending window dimensions that are not whole numbers

comment:26 Changed 3 years ago by Josh

@aradtech I like the idea of having a switch to full screen an app with no borders, should be trivial to implement but how would you handle for example, popup windows? Draw them with no chrome but in the position they requested?

comment:27 Changed 3 years ago by Antoine Martin

I've made it easier to setup the html client in #689.

It would be nice if the host and port fields could be populated with the host and port from the http request.
Even better would be if it could auto-connect by default, and present a password box if the server requires authentication.

comment:28 Changed 3 years ago by Josh

r7994 adds auto connect to host and port from http request, authentication coming soon.

comment:29 Changed 3 years ago by Antoine Martin

I don't know how to help debug these issues: I am getting no windows about 30% of the time (just force reload the page), and sometimes I get a window but its contents do not get painted until I move it, the rest of the time it works as expected..

comment:30 Changed 3 years ago by aradtech

joshiggins , could you please contact me via email at aradtech@…, we are interested in having you do some more work on the html5 client.

comment:31 Changed 3 years ago by Josh

r8710 and r8712 makes the client side protocol handling more stable.

  • packets were shifted from the buffer too early, when unlucky this caused random invalid packet errors
  • we can now detect websocket in webworker feature support instead of picking up browser version cues
  • this is required because Firefox 37+ will get support for this feature and assuming non-webkit browsers did not support it caused the functionality in Chrome to regress when they changed to Blink

It's more stable now. Next up, window drawing will be rationalised to iron out inconsistent behaviour (like sometimes not immediately painting contents), so that we have a solid foundation to start adding more encodings.

comment:32 in reply to:  description Changed 3 years ago by extasic

Replying to totaam:

Hi, I couldn't get the html5 client to work properly. Can you please check if I did something wrong, or if it might be an error?

In order to reproduce the behavior, I created a Dockerfile (http://paste.ubuntu.com/10518212/). You can use Docker on Linux or Boot2Docker on Windows or OS X to create an Xpra container.

The image can be build with docker build -t xpratest . and a new container be started with docker run --rm -p 8001:8001 xpratest.

When I connect to the web, I would expect both, xeyes and xterm to be rendered properly. Instead, the pupils of xeyes are not moved but painted over, when a movement occurs (the mouse is moved), and the xterm window stays empty. The result looks like https://www.dropbox.com/s/7g8xojwce3z1x3t/Screenshot%202015-03-03%2019.38.29.png?dl=0.

I used the current SVN revision 8743 as well as 8696, which has once already worked before. If I am using an Ubuntu image instead of Debian Jessie, the result is the same.

Do you have any idea what the problem might be?

comment:33 Changed 3 years ago by Josh

I couldn't build the supplied Dockerfile correctly. I had to put the packages on a single line and run the Xpra server as root, for some reason the created user wanted to use / as it's home directory.

However, you are missing the python-imaging package.

I use a container for development and I can share mine if you are interested.

comment:34 in reply to:  33 Changed 3 years ago by extasic

Replying to joshiggins:

I couldn't build the supplied Dockerfile correctly. I had to put the packages on a single line and run the Xpra server as root, for some reason the created user wanted to use / as it's home directory.


Maybe you were using an older Docker version? I think I remember some changes on 1.5.

However, you are missing the python-imaging package.


Great, that solved my problem, thank you!

I use a container for development and I can share mine if you are interested.


That would be great, maybe I made some other mistakes.

comment:35 Changed 3 years ago by Antoine Martin

However, you are missing the python-imaging package.


Great, that solved my problem, thank you!


In theory, the server should be able to work without python-imaging.. Not well, but it should be able to work using plain rgb encoding. (assuming that the html client handles that)

comment:36 in reply to:  35 Changed 3 years ago by Josh

In theory, the server should be able to work without python-imaging.. Not well, but it should be able to work using plain rgb encoding. (assuming that the html client handles that)

Is there any difference in handling between rgb24 and rgb? the HTML5 client asks for rgb24...

comment:37 Changed 3 years ago by Josh

r8746 + r8747 completes refactoring (and makes it the default) so that the client is more javascript-y. The implementation detail is largely unchanged but the following is notable

  • r8719 separates client code from the HTML page
  • r8745 is a slight improvement in XpraWindow.paint() so that we avoid painting twice
  • The connection interface needs to be re-implemented based on callbacks from the new XpraClient object. Right now the behaviour is to connect automatically to the host and port from the HTTP request.

comment:38 Changed 3 years ago by Josh

r8767 fixes a bug in the protocol handling that has evaded me for months, that usually manifests itself in windows not being drawn immediately.

comment:39 Changed 3 years ago by Josh

r8781 adds jpeg and png support to the HTML5 client.

Right now we use the native Image() object which needs the data base64 encoded - seems like a faff but even so it turned out considerably faster than jpgjs or png.js.

comment:40 Changed 3 years ago by extasic

First of all, thank you for your great job!

When using your latest changeset (r8781), there is a strange issue with some applications. If I run a QT application using wine, all menu elements (file menu, dropdowns, ...) open below the current window, not in front of it as expected. This only occurs when using the html5 client.

http://i.imgur.com/q2m73Jw.png

Do you have any idea what the problem might be? What information should I provide you in order to locate the issue?

Another issue is about the keyboard. When using a German keyboard layout, I can type all letters and numbers. There are some special characters like the exclamation mark I can type as well, but there is e.g. no key mapped to the period, so I'm unable to type it. Is it a known issue? Is there a way to configure the keyboard usage to include all German keys, maybe even including German "Umlauts"?

Last edited 3 years ago by Antoine Martin (previous) (diff)

comment:41 Changed 3 years ago by Josh

r9046 adds a style class for the window type MENU (we only had DROPDOWN_MENU and POPUP_MENU). Hopefully there shouldn't be any more menu elements appearing behind windows.

I've created a ticket for the keyboard problem #837.

comment:42 Changed 3 years ago by Josh

H.264 decoding support is stabilising in r9045. It will start to be selected if mutually available since r9049.

Use the new connection interface introduced in r9050 to change the preferred encoding by navigating to http://server:port/connect.html

The default behaviour without specifying /connect.html is still to connect automatically. If the connection is not successful, it will redirect to the connection interface.

comment:43 in reply to:  41 Changed 3 years ago by extasic

Replying to joshiggins:

r9046 adds a style class for the window type MENU (we only had DROPDOWN_MENU and POPUP_MENU). Hopefully there shouldn't be any more menu elements appearing behind windows.

Thank you for your update! Unfortunately the problem still persists for my useless. I'd provide a Dockerfile, but this didn't work the last time. All I did was install a current wine version and ran few Qt Applications. None of them (like VLC in the pictures above) worked properly: They displayed all menu items, dropdowns and tooltips behind the window. I had to move the window away in order to see them, and be able to select an entry.

Do you have any other idea what I might try, or what debug / log information I should provide?

I've created a ticket for the keyboard problem #837.

Great, thank you!

comment:44 Changed 3 years ago by Josh

@extasic I can't seem to reproduce this with the latest revision.

The wine applications use the DIALOG type hint for popups but these are still correctly placed above the WINDOW type.....

If you can tell me which window class is applied to your popup windows it would be useful. This is easy in chrome just by right clicking and selecting inspect element. It will show the html markup which should specify a class="window-WINDOW_TYPE" for the popup window.

Failing that I can take another stab at the Dockerfile.

comment:45 Changed 3 years ago by Antoine Martin

Not had time to look into it, but with the latest trunk I get:

disconnect: invalid packet format, not an xpra client?

It was fine just hours/days ago.

comment:46 in reply to:  44 Changed 3 years ago by extasic

Replying to joshiggins:

@extasic I can't seem to reproduce this with the latest revision.

The wine applications use the DIALOG type hint for popups but these are still correctly placed above the WINDOW type.....

If you can tell me which window class is applied to your popup windows it would be useful. This is easy in chrome just by right clicking and selecting inspect element. It will show the html markup which should specify a class="window-WINDOW_TYPE" for the popup window.

This is how the resulting HTML looks in Chrome. The highlighted Canvas represents a file menu (just like in the VLC picture I posted above). The div above the dialog one represents the main window. So there might be an error in the window type, right? But shouldn't there be something like a z-index anyway? Any workaround for this?

http://i.imgur.com/QAkGYBN.png

Just to mention it again: This error occurs for me only when running any Qt application in wine. If I run a non-Qt application in wine, or a Qt application directly in Linux, I couldn't reproduce the error. If I use the native xpra client, the error doesn't occur, even when running a Qt application under wine.

comment:47 Changed 3 years ago by Antoine Martin

My guess is that those applications use the wrong window type, it probably should not be NORMAL (sadly, lots of applications do that - ie: chrome does). We have some code in the xpra client to try to workaround such broken behaviour.
@joshiggins: maybe the window is "transient-for" or "override-redirect" and you can use that?

comment:48 Changed 3 years ago by Josh

Added a workaround in r9138. I was able to reproduce with VLC under Wine, it creates NORMAL override-redirect windows for the menus.

comment:49 Changed 3 years ago by Antoine Martin

You can ignore comment:45, I was hitting the wrong test server - I need sleep!

comment:50 Changed 3 years ago by Josh

I've uploaded a short video clip demonstrating the current state of H.264 decoding as a result of changes referred to in comment:42

https://www.youtube.com/watch?v=3yOoGWGyvgM

It's actually pretty good and remains usable.

comment:51 Changed 3 years ago by aradtech

I can confirm Josh's changes work on F21.

comment:52 Changed 3 years ago by aradtech

And would like to say thank you to Josh for your awesome work on the Xpra Html5 client and our requests.

comment:53 in reply to:  48 Changed 3 years ago by extasic

Replying to joshiggins:

Added a workaround in r9138. I was able to reproduce with VLC under Wine, it creates NORMAL override-redirect windows for the menus.

Works for me too, thank you!

comment:54 Changed 3 years ago by Josh

r9148 + r9149 adds support for LZ4 packet compressor

comment:55 Changed 3 years ago by Antoine Martin

Resolution: fixed
Status: newclosed

It's looking good for the release!

I have tagged v0.15.x, so let's move new work to a new ticket: #850

It will be easier to know what was delivered in this release and what remains to be done.

If you are hitting this ticket wanting to report a bug, please open a new ticket instead.

comment:56 Changed 3 years ago by Antoine Martin

Milestone: 1.00.15

(update milestone)

comment:57 Changed 3 years ago by Antoine Martin

There is now a wiki page for the HTML5 client: wiki/Clients/HTML5.

Note: See TracTickets for help on using tickets.