Xpra trac archive

Xpra: Ticket #473: html5 client

Similar to noVNC, and probably re-using some of its code (websockets, etc)

Made much easier by #474

I can start a websockets proxy (on port 8080) that points back to xpra (on port 10000): 

./websockify.py --web ./Xpra/trunk/html5/ 8080 localhost:10000

And if I use the new tcp-proxy code: 

xpra start :10 --bind-tcp=0.0.0.0:10000 --tcp-proxy=127.0.0.1:8080

When we make websocket request to the xpra port (10000), the http traffic goes through xpra to the proxy then back to xpra as tcp! Eventually, the websockets proxy code should be moved into xpra, but this is good enough for experimenting and testing.


Sat, 14 Dec 2013 14:29:58 GMT - Antoine Martin: owner, status, description changed

Mon, 16 Dec 2013 17:26:24 GMT - Antoine Martin: description changed

Sun, 22 Dec 2013 06:56:47 GMT - Antoine Martin: attachment set

experimental html5 client is now capable of showing an xterm window!

Sun, 22 Dec 2013 07:00:09 GMT - Antoine Martin: attachment set

modified bencoder that can talk to an xpra server and handle byte buffers efficiently (original code is MIT licensed)

Sun, 22 Dec 2013 07:43:19 GMT - Antoine Martin:

What I've used:

The license reads: Free to use and distribute at will, so long as you are nice to people, etc. May need clarification... as free licenses are unrestricted, and asking for people to "be nice" may be incompatible with other free licenses!


I will attach some code once it is cleaned up.

What still needs to be done:

And maybe further down the line:

Wed, 25 Dec 2013 13:09:47 GMT - Antoine Martin:

Initial code merged in r5028: sort of works in chrome and firefox (partial window updates don't work, partial keyboard support, no mouse support, ..)

Thu, 26 Dec 2013 14:30:57 GMT - Antoine Martin:

Many fixes (see changesets) in: r5029 + r5030 + r5031 + r5032 + r5033

Actually makes this almost usable!

Sat, 28 Dec 2013 15:23:13 GMT - Antoine Martin: description changed

Sat, 28 Dec 2013 15:23:47 GMT - Antoine Martin: description changed

Mon, 30 Dec 2013 04:44:38 GMT - Antoine Martin: description changed

Mon, 30 Dec 2013 14:55:51 GMT - Antoine Martin:

Many more changes in r5034 to r5066, it looks a lot better:

Mon, 30 Dec 2013 15:02:44 GMT - Antoine Martin: attachment set

updated screenshot of html5 client running in full-window mode

Wed, 01 Jan 2014 12:45:02 GMT - Antoine Martin:

r5076 + r5077 + r5078 + r5079 + r5080 add a cython version of the bencoder, which makes it substantially faster (important since the html5 version uses the bencoder rather than rencode)

Running the updated bencode test shows the difference in performance: 

test_compare_cython()
results: {'python': 1903, 'cython': 1255} (in milliseconds)

Thu, 02 Jan 2014 13:21:07 GMT - Antoine Martin:

Many more improvements in r5081 to r5089, including support for fullscreen and maximized windows, window top bar icons, etc..

Sadly, I've come to realize that the way I have used the canvas is all wrong: each window should be its own DIV instead, and we can manage:

etc

But I have to leave this for now... it is what it is.

Mon, 03 Mar 2014 15:10:45 GMT - Antoine Martin: milestone changed

Needs more time than I can afford to spend on it. Re-scheduling.

Using a CSS decorations on a canvas-per-window is a must though.

Tue, 27 May 2014 02:30:51 GMT - Antoine Martin: attachment set

work by Joshua Higgins to make this use a canvas within a div for each window (much much better!)

Tue, 03 Jun 2014 14:58:07 GMT - Josh: attachment set

superseding my previous patch, cleaned up, many improvements

Tue, 03 Jun 2014 15:02:45 GMT - Josh:

Attached updated patch

Wed, 04 Jun 2014 14:39:38 GMT - Antoine Martin: attachment set

slightly modified patch

Wed, 04 Jun 2014 14:41:06 GMT - Antoine Martin: cc set

The patch above makes the following changes on your patch:

I did this as I was reviewing the code. I then tried to run it but it must have got stuck... will try again later.

Wed, 23 Jul 2014 13:04:20 GMT - Antoine Martin:

Note: as part of #614, r6934 adds support for YAML as packet encoding.

@joshiggins: Any HTML5 updates?

Sun, 03 Aug 2014 21:14:52 GMT - Josh:

Replying to totaam:

@joshiggins: Any HTML5 updates?

Adding missing window functions for icon, maximise and actually destroying windows when connection is closed (currently leaves the last drawn windows floating around). Started moving websockets code into a worker. Maybe it would be good to start SVN revisions to make it easier tracking bugs in these patches/?

There is a bug that affects even the unpatched html5 client which causes an invalid packet header error to be thrown with a large number of screen updates, which looks like the buffer in protocol.js to be at fault but nailing it has been proving difficult!

Mon, 04 Aug 2014 05:12:52 GMT - Antoine Martin:

Adding missing...


Good stuff!

Maybe it would be good to start SVN revisions


svn repo info sent separately

which causes an invalid packet header error to be thrown with a large number of screen updates


Just a guess, but it might be that those are rgb updates compressed with zlib, try turning that off and see if it helps. (see #614)

Sun, 10 Aug 2014 17:50:36 GMT - Josh:

r7227 merges latest html5-v2.patch from comment 14

Wed, 13 Aug 2014 01:14:49 GMT - Josh:

Some updates in r7228 + r7230 + r7255 + r7271

However, using a Websocket within a Webworker is not supported in Firefox and it doesn't look like it's coming anytime soon (see https://bugzilla.mozilla.org/show_bug.cgi?id=504553) so the web worker method will only be used if WebKit? is detected.
It's usable at this stage with not so bad performance. In addition to some bits in comment 3 also needs:


Seems to work well in Chrome, Safari and Firefox. Reports from Internet Explorer would be welcome.

Wed, 20 Aug 2014 12:55:36 GMT - Sam F Beroz:

I'd very much like to try out the recent changes to the html5 client but I'm having an issue with the 0.14.x release (I was able to run the client with 0.13.x). The message I'm seeing in the log file is "no matching packet encoder found!". Looking at the code for 0.14.x I'm not sure the following logic is correct:

svn annotate http://xpra.org/svn/Xpra/tags/v0.14.x/src/xpra/net/protocol.py | less -N 

    352   6964    antoine     def enable_encoder_from_caps(self, caps):
    353   6983    antoine         opts = packet_encoding.get_enabled_encoders(order=packet_encoding.PERFORMANCE_ORDER)
    354   6982    antoine         for e in opts:
    355   6982    antoine             if caps.boolget("rencode"):
    356   6982    antoine                 self.enable_encoder(e)
    357   6982    antoine                 return True
    358   6982    antoine         log.error("no matching packet encoder found!")
    359   6982    antoine         return False

Especially when comparing it to a prior version of the same method: svn annotate -r 6981 http://xpra.org/svn/Xpra/trunk/src/xpra/net/protocol.py 

  6964    antoine     def enable_encoder_from_caps(self, caps):
  6965    antoine         if packet_encoding.use_rencode and caps.boolget("rencode"):
  5076    antoine             self.enable_rencode()
  6965    antoine         elif packet_encoding.use_yaml and caps.boolget("yaml"):
  6964    antoine             self.enable_yaml()
  6965    antoine         elif packet_encoding.use_bencode and caps.boolget("bencode", True):
  6964    antoine             self.enable_bencode()
  6964    antoine         else:
  6969    antoine             log.error("no matching packet encoder found!")
  6969    antoine             return False
  6969    antoine         return True

However both methods are pretty different from how it was coded in 0.13.x. I'm not quite sure what to try next if this isn't the cause of my connection issue. Is the 0.14.x logic correct? Is there something else I should look at? Thanks - Sam

Wed, 20 Aug 2014 13:00:08 GMT - Antoine Martin: owner, status changed

You're right, that looks completely wrong! It must have got broke after the testing in #614. Does r7365 + r7374 fix things for you?

Wed, 20 Aug 2014 13:10:17 GMT - Sam F Beroz:

That looks much better. I'll try it out later today. Thanks - Sam

Tue, 16 Sep 2014 22:58:12 GMT - aradtech:

Tried it out and it works much better then before , great work! I was wondering if we could get a couple of switches to say make it so app starts full screen with no min/close window perhaps if we are only gonna use it for one app that will use the entire canvas space of the browser area.

Wed, 17 Sep 2014 03:24:44 GMT - Antoine Martin: owner changed

Yeah, just tried it and it works great! I've mirrored the html code here for easy access: http://xpra.org/html5/

I managed to hit this bug though: 

ValueError: invalid literal for long() with base 10: '563.991455078125'
2014-09-17 10:20:33,210 failed to parse bencode packet: 6c31363a63...

Which decodes to: 

$ python -c 'import binascii;print(binascii.unhexlify("6c31363a63..."))'
l16:configure-windowi1ei563.991455078125ei487.991455078125ei499ei316ed21:encodings.rgb_formatsl4:RGBX4:RGBAeee

So the configure window uses dimensions that are not whole numbers, and that crashes the server bdecoder because that's invalid.

Also saw these in the websockify server log (not sure if they are related): 

code 400, message Bad request syntax ('\x88\x8f\\\xb9\xac\xc6_Q\xf8\xa7.\xde\xc9\xb2|\xda\xc0\xa9/\xdc\xc8')
code 400, message Bad HTTP/0.9 request type ('\x88\x8f\x0e\x06\xd6I')
code 400, message Bad HTTP/0.9 request type ('\x88\x8f#\xaf\x85\x15')

Wed, 17 Sep 2014 20:51:31 GMT - Josh:

I think r7677 should fix sending window dimensions that are not whole numbers

Wed, 17 Sep 2014 20:55:50 GMT - Josh:

@aradtech I like the idea of having a switch to full screen an app with no borders, should be trivial to implement but how would you handle for example, popup windows? Draw them with no chrome but in the position they requested?

Mon, 22 Sep 2014 08:14:31 GMT - Antoine Martin:

I've made it easier to setup the html client in #689.

It would be nice if the host and port fields could be populated with the host and port from the http request. Even better would be if it could auto-connect by default, and present a password box if the server requires authentication.

Tue, 28 Oct 2014 22:59:30 GMT - Josh:

r7994 adds auto connect to host and port from http request, authentication coming soon.

Fri, 31 Oct 2014 08:02:51 GMT - Antoine Martin:

I don't know how to help debug these issues: I am getting no windows about 30% of the time (just force reload the page), and sometimes I get a window but its contents do not get painted until I move it, the rest of the time it works as expected..

Tue, 27 Jan 2015 08:34:10 GMT - aradtech:

joshiggins , could you please contact me via email at aradtech@…, we are interested in having you do some more work on the html5 client.

Sat, 28 Feb 2015 01:04:52 GMT - Josh:

r8710 and r8712 makes the client side protocol handling more stable.

It's more stable now. Next up, window drawing will be rationalised to iron out inconsistent behaviour (like sometimes not immediately painting contents), so that we have a solid foundation to start adding more encodings.

Tue, 03 Mar 2015 18:59:36 GMT - extasic:

Replying to totaam:

Hi, I couldn't get the html5 client to work properly. Can you please check if I did something wrong, or if it might be an error?

In order to reproduce the behavior, I created a Dockerfile (http://paste.ubuntu.com/10518212/). You can use Docker on Linux or Boot2Docker on Windows or OS X to create an Xpra container.

The image can be build with docker build -t xpratest . and a new container be started with docker run --rm -p 8001:8001 xpratest.

When I connect to the web, I would expect both, xeyes and xterm to be rendered properly. Instead, the pupils of xeyes are not moved but painted over, when a movement occurs (the mouse is moved), and the xterm window stays empty. The result looks like https://www.dropbox.com/s/7g8xojwce3z1x3t/Screenshot%202015-03-03%2019.38.29.png?dl=0.

I used the current SVN revision 8743 as well as 8696, which has once already worked before. If I am using an Ubuntu image instead of Debian Jessie, the result is the same.

Do you have any idea what the problem might be?

Tue, 03 Mar 2015 20:51:45 GMT - Josh:

I couldn't build the supplied Dockerfile correctly. I had to put the packages on a single line and run the Xpra server as root, for some reason the created user wanted to use / as it's home directory.

However, you are missing the python-imaging package.

I use a container for development and I can share mine if you are interested.

Wed, 04 Mar 2015 18:06:20 GMT - extasic:

Replying to joshiggins:

I couldn't build the supplied Dockerfile correctly. I had to put the packages on a single line and run the Xpra server as root, for some reason the created user wanted to use / as it's home directory.

Maybe you were using an older Docker version? I think I remember some changes on 1.5.

However, you are missing the python-imaging package.

Great, that solved my problem, thank you!

I use a container for development and I can share mine if you are interested.

That would be great, maybe I made some other mistakes.

Thu, 05 Mar 2015 05:02:56 GMT - Antoine Martin:

However, you are missing the python-imaging package.


Great, that solved my problem, thank you!


In theory, the server should be able to work without python-imaging.. Not well, but it should be able to work using plain rgb encoding. (assuming that the html client handles that)

Thu, 05 Mar 2015 12:50:54 GMT - Josh:

In theory, the server should be able to work without python-imaging.. Not well, but it should be able to work using plain rgb encoding. (assuming that the html client handles that)

Is there any difference in handling between rgb24 and rgb? the HTML5 client asks for rgb24...

Sat, 07 Mar 2015 20:44:28 GMT - Josh:

r8746 + r8747 completes refactoring (and makes it the default) so that the client is more javascript-y. The implementation detail is largely unchanged but the following is notable

Sun, 08 Mar 2015 20:27:13 GMT - Josh:

r8767 fixes a bug in the protocol handling that has evaded me for months, that usually manifests itself in windows not being drawn immediately.

Sat, 14 Mar 2015 16:02:17 GMT - Josh:

r8781 adds jpeg and png support to the HTML5 client.

Right now we use the native Image() object which needs the data base64 encoded - seems like a faff but even so it turned out considerably faster than png.js.

Sun, 22 Mar 2015 16:53:18 GMT - extasic:

First of all, thank you for your great job!

When using your latest changeset (r8781), there is a strange issue with some applications. If I run a QT application using wine, all menu elements (file menu, dropdowns, ...) open below the current window, not in front of it as expected. This only occurs when using the html5 client.

http://i.imgur.com/q2m73Jw.png

Do you have any idea what the problem might be? What information should I provide you in order to locate the issue?

Another issue is about the keyboard. When using a German keyboard layout, I can type all letters and numbers. There are some special characters like the exclamation mark I can type as well, but there is e.g. no key mapped to the period, so I'm unable to type it. Is it a known issue? Is there a way to configure the keyboard usage to include all German keys, maybe even including German "Umlauts"?

Sat, 18 Apr 2015 14:53:52 GMT - Josh:

r9046 adds a style class for the window type MENU (we only had DROPDOWN_MENU and POPUP_MENU). Hopefully there shouldn't be any more menu elements appearing behind windows.

I've created a ticket for the keyboard problem #837.

Sat, 18 Apr 2015 22:12:01 GMT - Josh:

H.264 decoding support is stabilising in r9045. It will start to be selected if mutually available since r9049.

Use the new connection interface introduced in r9050 to change the preferred encoding by navigating to http://server:port/connect.html

The default behaviour without specifying /connect.html is still to connect automatically. If the connection is not successful, it will redirect to the connection interface.

Mon, 20 Apr 2015 17:10:06 GMT - extasic:

Replying to joshiggins:

r9046 adds a style class for the window type MENU (we only had DROPDOWN_MENU and POPUP_MENU). Hopefully there shouldn't be any more menu elements appearing behind windows.

Thank you for your update! Unfortunately the problem still persists for my useless. I'd provide a Dockerfile, but this didn't work the last time. All I did was install a current wine version and ran few Qt Applications. None of them (like VLC in the pictures above) worked properly: They displayed all menu items, dropdowns and tooltips behind the window. I had to move the window away in order to see them, and be able to select an entry.

Do you have any other idea what I might try, or what debug / log information I should provide?

I've created a ticket for the keyboard problem #837.

Great, thank you!

Mon, 20 Apr 2015 17:39:48 GMT - Josh:

@extasic I can't seem to reproduce this with the latest revision.

The wine applications use the DIALOG type hint for popups but these are still correctly placed above the WINDOW type.....

If you can tell me which window class is applied to your popup windows it would be useful. This is easy in chrome just by right clicking and selecting inspect element. It will show the html markup which should specify a class="window-WINDOW_TYPE" for the popup window.

Failing that I can take another stab at the Dockerfile.

Thu, 23 Apr 2015 09:13:50 GMT - Antoine Martin:

Not had time to look into it, but with the latest trunk I get: 

disconnect: invalid packet format, not an xpra client?

It was fine just hours/days ago.

Thu, 23 Apr 2015 14:18:59 GMT - extasic:

Replying to joshiggins:

@extasic I can't seem to reproduce this with the latest revision.

The wine applications use the DIALOG type hint for popups but these are still correctly placed above the WINDOW type.....

If you can tell me which window class is applied to your popup windows it would be useful. This is easy in chrome just by right clicking and selecting inspect element. It will show the html markup which should specify a class="window-WINDOW_TYPE" for the popup window.

This is how the resulting HTML looks in Chrome. The highlighted Canvas represents a file menu (just like in the VLC picture I posted above). The div above the dialog one represents the main window. So there might be an error in the window type, right? But shouldn't there be something like a z-index anyway? Any workaround for this?

http://i.imgur.com/QAkGYBN.png

Just to mention it again: This error occurs for me only when running any Qt application in wine. If I run a non-Qt application in wine, or a Qt application directly in Linux, I couldn't reproduce the error. If I use the native xpra client, the error doesn't occur, even when running a Qt application under wine.

Thu, 23 Apr 2015 14:22:49 GMT - Antoine Martin:

My guess is that those applications use the wrong window type, it probably should not be NORMAL (sadly, lots of applications do that - ie: chrome does). We have some code in the xpra client to try to workaround such broken behaviour. @joshiggins: maybe the window is "transient-for" or "override-redirect" and you can use that?

Thu, 23 Apr 2015 15:08:35 GMT - Josh:

Added a workaround in r9138. I was able to reproduce with VLC under Wine, it creates NORMAL override-redirect windows for the menus.

Thu, 23 Apr 2015 15:12:28 GMT - Antoine Martin:

You can ignore comment:45, I was hitting the wrong test server - I need sleep!

Thu, 23 Apr 2015 15:41:24 GMT - Josh:

I've uploaded a short video clip demonstrating the current state of H.264 decoding as a result of changes referred to in comment:42

https://www.youtube.com/watch?v=3yOoGWGyvgM

It's actually pretty good and remains usable.

Thu, 23 Apr 2015 16:06:08 GMT - aradtech:

I can confirm Josh's changes work on F21.

Thu, 23 Apr 2015 16:07:32 GMT - aradtech:

And would like to say thank you to Josh for your awesome work on the Xpra Html5 client and our requests.

Fri, 24 Apr 2015 10:52:49 GMT - extasic:

Replying to joshiggins:

Added a workaround in r9138. I was able to reproduce with VLC under Wine, it creates NORMAL override-redirect windows for the menus.

Works for me too, thank you!

Sun, 26 Apr 2015 13:15:22 GMT - Josh:

r9148 + r9149 adds support for LZ4 packet compressor

Thu, 30 Apr 2015 15:42:02 GMT - Antoine Martin: status changed; resolution set

It's looking good for the release!

I have tagged v0.15.x, so let's move new work to a new ticket: #850

It will be easier to know what was delivered in this release and what remains to be done.

If you are hitting this ticket wanting to report a bug, please open a new ticket instead.

Fri, 01 May 2015 04:45:03 GMT - Antoine Martin: milestone changed

(update milestone)

Sun, 31 May 2015 09:42:34 GMT - Antoine Martin:

There is now a wiki page for the HTML5 client: wiki/Clients/HTML5.

Sat, 23 Jan 2021 04:56:39 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/473