#473 closed task (fixed)
html5 client
Reported by: | Antoine Martin | Owned by: | Josh |
---|---|---|---|
Priority: | major | Milestone: | 0.15 |
Component: | platforms | Version: | |
Keywords: | Cc: | Josh |
Description (last modified by )
Similar to noVNC, and probably re-using some of its code (websockets, etc)
Made much easier by #474
I can start a websockets proxy (on port 8080) that points back to xpra (on port 10000):
./websockify.py --web ./Xpra/trunk/html5/ 8080 localhost:10000
And if I use the new tcp-proxy code:
xpra start :10 --bind-tcp=0.0.0.0:10000 --tcp-proxy=127.0.0.1:8080
When we make websocket request to the xpra port (10000), the http traffic goes through xpra to the proxy then back to xpra as tcp!
Eventually, the websockets proxy code should be moved into xpra, but this is good enough for experimenting and testing.
Attachments (6)
Change History (64)
comment:1 Changed 7 years ago by
Description: | modified (diff) |
---|---|
Owner: | changed from Antoine Martin to Antoine Martin |
Status: | new → assigned |
comment:2 Changed 7 years ago by
Description: | modified (diff) |
---|
Changed 7 years ago by
Attachment: | xpra-html5-PoC-client.png added |
---|
Changed 7 years ago by
Attachment: | bencode.js added |
---|
modified bencoder that can talk to an xpra server and handle byte buffers efficiently (original code is MIT licensed)
comment:3 Changed 7 years ago by
What I've used:
- websockify (many licenses for all the different parts: LGPL version 3, MPL 2, BSD... see: LICENSE.txt)
- bencode.js (modified version - MIT licensed)
- The canvas code is based on shapesdemo, itself based on these tutorials: A Gentle Introduction to Making HTML5 Canvas Interactive, Selectable Shapes Part 2: Resizable, Movable Shapes on HTML5 Canvas.
The license reads: Free to use and distribute at will, so long as you are nice to people, etc
. May need clarification... as free licenses are unrestricted, and asking for people to "be nice" may be incompatible with other free licenses!
- For zlib, I've used imaya's zlib.js (english link), which is MIT licensed.
I will attach some code once it is cleaned up.
What still needs to be done:
- move the network code to a worker (see: Using webworkers)
- use FileReaderSync so packets get delivered in the same order as they are received!
- use Transferable objects from the worker thread to the ui thread
- instead of invalidating the whole canvas, repaint just the area we have updated using CanvasRenderingContext2D.drawImage or CanvasRenderingContext2D.putImageData
- handle png, jpeg and webp picture formats (ie: receiving image through websocket and Display image from blob using javascript and websockets)
- fix compatibility with other browsers (only works on Firefox for now!)
- handle browser window resizing (fit to page)
- handle window decorations using stylesheets
- tell server about the size of our decorations so it can start sending pixel data immediately without having to wait for the client
map-event
packet which is unnecessary using HTML5 (we can place things precisely wherever we like) - better UI with connection dialog, options, etc (and add no decorations mode option for full-window content)
- handle focus, clicks and keyboard
And maybe further down the line:
- handle notifications (floating divs?), custom cursors, bell events (blink? bell icon?), clipboard, etc..
- use MediaSource API to pass video data to a video element to support h264/vp8/etc
- use webgl for rendering and csc
- use int protocol aliases to make packets shorter
- move websockify calls inside the xpra process to avoid all the back and forth, extra sockets, etc
- lz4 support (after removing the need for node.js from the module)
- implement rencode in js
comment:4 Changed 7 years ago by
Initial code merged in r5028: sort of works in chrome and firefox (partial window updates don't work, partial keyboard support, no mouse support, ..)
comment:5 Changed 7 years ago by
comment:6 Changed 7 years ago by
Description: | modified (diff) |
---|
comment:7 Changed 7 years ago by
Description: | modified (diff) |
---|
comment:8 Changed 7 years ago by
Description: | modified (diff) |
---|
comment:9 Changed 7 years ago by
Many more changes in r5034 to r5066, it looks a lot better:
- we capture all key and mouse events
- keyboard works well enough (including
BackSpace
) - added server version check
- focus sort of works - most of the time...
- we can disconnect and restore the page to its original state
- some style effort to make it look less barren
- canvas now resizes to fit browser window size, and we tell the server
- popup mode (launch in window with minimal decorations)
- better debugging (only via javascript console now)
- handle window raise
- allow all params to be set via query string, ie:
http://localhost:8080/index.html?host=localhost&port=8080&connect=true
(and validate all values to prevent XSS) - auto-connect option (see example URL above)
- use javascript strict mode
- option to show dialog or not (canvas then uses up all the window space)
Changed 7 years ago by
Attachment: | xpra-html5.png added |
---|
updated screenshot of html5 client running in full-window mode
comment:10 Changed 7 years ago by
r5076 + r5077 + r5078 + r5079 + r5080 add a cython version of the bencoder, which makes it substantially faster (important since the html5 version uses the bencoder rather than rencode)
Running the updated bencode test shows the difference in performance:
test_compare_cython() results: {'python': 1903, 'cython': 1255} (in milliseconds)
comment:11 Changed 7 years ago by
Many more improvements in r5081 to r5089, including support for fullscreen and maximized windows, window top bar icons, etc..
Sadly, I've come to realize that the way I have used the canvas is all wrong: each window should be its own DIV
instead, and we can manage:
- decorations using regular CSS
- focus using CSS z-index
- window contents using a canvas
- maximizing/fullscreen using CSS
- buttons and events using regular javascript events instead of re-wiring clicks and events ourselves (then we can more easily have hover, etc)
- redraws are then managed by the browser (much faster)
etc
But I have to leave this for now... it is what it is.
comment:12 Changed 7 years ago by
Milestone: | 0.12 → 1.0 |
---|
Needs more time than I can afford to spend on it. Re-scheduling.
Using a CSS decorations on a canvas-per-window is a must though.
Changed 7 years ago by
Attachment: | xprahtml5.patch added |
---|
work by Joshua Higgins to make this use a canvas within a div for each window (much much better!)
Changed 7 years ago by
Attachment: | 3junexprahtml5.patch added |
---|
superseding my previous patch, cleaned up, many improvements
comment:13 Changed 7 years ago by
Attached updated patch
- better CSS handling with styles for window types
- basic z-index stacking
- edge resizing
- shape.js is no longer required
comment:14 Changed 7 years ago by
Cc: | Josh added |
---|
The patch above makes the following changes on your patch:
- ignored changes in
util.js
andwebsock.js
- deleted some old commented out code
- removed empty
function pass()
(not referenced anywhere?) - re-indented form section in
index.html
since it is mostly unchanged and removed extra closing div - I've kept the early exit in
do_send_new_screen_size
(I seem to remember it avoided errors when closing the connection?)
I did this as I was reviewing the code.
I then tried to run it but it must have got stuck... will try again later.
comment:15 follow-up: 16 Changed 7 years ago by
comment:16 Changed 7 years ago by
Replying to totaam:
@joshiggins: Any HTML5 updates?
Adding missing window functions for icon, maximise and actually destroying windows when connection is closed (currently leaves the last drawn windows floating around). Started moving websockets code into a worker. Maybe it would be good to start SVN revisions to make it easier tracking bugs in these patches/?
There is a bug that affects even the unpatched html5 client which causes an invalid packet header error to be thrown with a large number of screen updates, which looks like the buffer in protocol.js to be at fault but nailing it has been proving difficult!
comment:17 Changed 7 years ago by
Adding missing...
Good stuff!
Maybe it would be good to start SVN revisions
svn repo info sent separately
which causes an invalid packet header error to be thrown with a large number of screen updates
Just a guess, but it might be that those are rgb
updates compressed with zlib, try turning that off and see if it helps. (see #614)
comment:19 Changed 7 years ago by
Some updates in r7228 + r7230 + r7255 + r7271
- Draw window updates off screen and use requestAnimationFrame to redraw browser canvas
- Maximise and restore windows, destroy client windows on connection close
- Invalid packet header bug fixed by using websock.js more robust receive queue buffer
- r7271 moves communication and packet decoding code into a Webworker
However, using a Websocket within a Webworker is not supported in Firefox and it doesn't look like it's coming anytime soon (see https://bugzilla.mozilla.org/show_bug.cgi?id=504553) so the web worker method will only be used if WebKit? is detected.
It's usable at this stage with not so bad performance. In addition to some bits in comment 3 also needs:
- Separate client initialisation code and window styles from index.html
- Fullscreen / override windows need handling properly with CSS
- Send new screen size on browser resize
- Add lz4 support (https://github.com/pierrec/node-lz4)
- Still needs to draw window icon
Seems to work well in Chrome, Safari and Firefox. Reports from Internet Explorer would be welcome.
comment:20 Changed 7 years ago by
I'd very much like to try out the recent changes to the html5 client but I'm having an issue with the 0.14.x release (I was able to run the client with 0.13.x). The message I'm seeing in the log file is "no matching packet encoder found!". Looking at the code for 0.14.x I'm not sure the following logic is correct:
svn annotate http://xpra.org/svn/Xpra/tags/v0.14.x/src/xpra/net/protocol.py | less -N
352 6964 antoine def enable_encoder_from_caps(self, caps): 353 6983 antoine opts = packet_encoding.get_enabled_encoders(order=packet_encoding.PERFORMANCE_ORDER) 354 6982 antoine for e in opts: 355 6982 antoine if caps.boolget("rencode"): 356 6982 antoine self.enable_encoder(e) 357 6982 antoine return True 358 6982 antoine log.error("no matching packet encoder found!") 359 6982 antoine return False
Especially when comparing it to a prior version of the same method:
svn annotate -r 6981 http://xpra.org/svn/Xpra/trunk/src/xpra/net/protocol.py
6964 antoine def enable_encoder_from_caps(self, caps): 6965 antoine if packet_encoding.use_rencode and caps.boolget("rencode"): 5076 antoine self.enable_rencode() 6965 antoine elif packet_encoding.use_yaml and caps.boolget("yaml"): 6964 antoine self.enable_yaml() 6965 antoine elif packet_encoding.use_bencode and caps.boolget("bencode", True): 6964 antoine self.enable_bencode() 6964 antoine else: 6969 antoine log.error("no matching packet encoder found!") 6969 antoine return False 6969 antoine return True
However both methods are pretty different from how it was coded in 0.13.x. I'm not quite sure what to try next if this isn't the cause of my connection issue. Is the 0.14.x logic correct? Is there something else I should look at? Thanks - Sam
comment:21 Changed 7 years ago by
Owner: | changed from Antoine Martin to Sam F Beroz |
---|---|
Status: | assigned → new |
comment:23 Changed 7 years ago by
Tried it out and it works much better then before , great work! I was wondering if we could get a couple of switches to say make it so app starts full screen with no min/close window perhaps if we are only gonna use it for one app that will use the entire canvas space of the browser area.
comment:24 Changed 7 years ago by
Owner: | changed from Sam F Beroz to Josh |
---|
Yeah, just tried it and it works great!
I've mirrored the html code here for easy access: http://xpra.org/html5/
I managed to hit this bug though:
ValueError: invalid literal for long() with base 10: '563.991455078125' 2014-09-17 10:20:33,210 failed to parse bencode packet: 6c31363a63...
Which decodes to:
$ python -c 'import binascii;print(binascii.unhexlify("6c31363a63..."))' l16:configure-windowi1ei563.991455078125ei487.991455078125ei499ei316ed21:encodings.rgb_formatsl4:RGBX4:RGBAeee
So the configure window uses dimensions that are not whole numbers, and that crashes the server bdecoder because that's invalid.
Also saw these in the websockify server log (not sure if they are related):
code 400, message Bad request syntax ('\x88\x8f\\\xb9\xac\xc6_Q\xf8\xa7.\xde\xc9\xb2|\xda\xc0\xa9/\xdc\xc8') code 400, message Bad HTTP/0.9 request type ('\x88\x8f\x0e\x06\xd6I') code 400, message Bad HTTP/0.9 request type ('\x88\x8f#\xaf\x85\x15')
comment:25 Changed 7 years ago by
I think r7677 should fix sending window dimensions that are not whole numbers
comment:26 Changed 7 years ago by
@aradtech I like the idea of having a switch to full screen an app with no borders, should be trivial to implement but how would you handle for example, popup windows? Draw them with no chrome but in the position they requested?
comment:27 Changed 7 years ago by
I've made it easier to setup the html client in #689.
It would be nice if the host and port fields could be populated with the host and port from the http request.
Even better would be if it could auto-connect by default, and present a password box if the server requires authentication.
comment:28 Changed 6 years ago by
r7994 adds auto connect to host and port from http request, authentication coming soon.
comment:29 Changed 6 years ago by
I don't know how to help debug these issues: I am getting no windows about 30% of the time (just force reload the page), and sometimes I get a window but its contents do not get painted until I move it, the rest of the time it works as expected..
comment:30 Changed 6 years ago by
joshiggins , could you please contact me via email at aradtech@…, we are interested in having you do some more work on the html5 client.
comment:31 Changed 6 years ago by
r8710 and r8712 makes the client side protocol handling more stable.
- packets were shifted from the buffer too early, when unlucky this caused random invalid packet errors
- we can now detect websocket in webworker feature support instead of picking up browser version cues
- this is required because Firefox 37+ will get support for this feature and assuming non-webkit browsers did not support it caused the functionality in Chrome to regress when they changed to Blink
It's more stable now. Next up, window drawing will be rationalised to iron out inconsistent behaviour (like sometimes not immediately painting contents), so that we have a solid foundation to start adding more encodings.
comment:32 Changed 6 years ago by
Replying to totaam:
Hi, I couldn't get the html5 client to work properly. Can you please check if I did something wrong, or if it might be an error?
In order to reproduce the behavior, I created a Dockerfile (http://paste.ubuntu.com/10518212/). You can use Docker on Linux or Boot2Docker on Windows or OS X to create an Xpra container.
The image can be build with docker build -t xpratest .
and a new container be started with docker run --rm -p 8001:8001 xpratest
.
When I connect to the web, I would expect both, xeyes and xterm to be rendered properly. Instead, the pupils of xeyes are not moved but painted over, when a movement occurs (the mouse is moved), and the xterm window stays empty. The result looks like https://www.dropbox.com/s/7g8xojwce3z1x3t/Screenshot%202015-03-03%2019.38.29.png?dl=0.
I used the current SVN revision 8743 as well as 8696, which has once already worked before. If I am using an Ubuntu image instead of Debian Jessie, the result is the same.
Do you have any idea what the problem might be?
comment:33 follow-up: 34 Changed 6 years ago by
I couldn't build the supplied Dockerfile correctly. I had to put the packages on a single line and run the Xpra server as root, for some reason the created user wanted to use / as it's home directory.
However, you are missing the python-imaging package.
I use a container for development and I can share mine if you are interested.
comment:34 Changed 6 years ago by
Replying to joshiggins:
I couldn't build the supplied Dockerfile correctly. I had to put the packages on a single line and run the Xpra server as root, for some reason the created user wanted to use / as it's home directory.
Maybe you were using an older Docker version? I think I remember some changes on 1.5.
However, you are missing the python-imaging package.
Great, that solved my problem, thank you!
I use a container for development and I can share mine if you are interested.
That would be great, maybe I made some other mistakes.
comment:35 follow-up: 36 Changed 6 years ago by
However, you are missing the python-imaging package.
Great, that solved my problem, thank you!
In theory, the server should be able to work without python-imaging.. Not well, but it should be able to work using plain rgb encoding. (assuming that the html client handles that)
comment:36 Changed 6 years ago by
In theory, the server should be able to work without python-imaging.. Not well, but it should be able to work using plain rgb encoding. (assuming that the html client handles that)
Is there any difference in handling between rgb24 and rgb? the HTML5 client asks for rgb24...
comment:37 Changed 6 years ago by
r8746 + r8747 completes refactoring (and makes it the default) so that the client is more javascript-y. The implementation detail is largely unchanged but the following is notable
- r8719 separates client code from the HTML page
- r8745 is a slight improvement in
XpraWindow.paint()
so that we avoid painting twice - The connection interface needs to be re-implemented based on callbacks from the new
XpraClient
object. Right now the behaviour is to connect automatically to the host and port from the HTTP request.
comment:38 Changed 6 years ago by
r8767 fixes a bug in the protocol handling that has evaded me for months, that usually manifests itself in windows not being drawn immediately.
comment:39 Changed 6 years ago by
comment:40 Changed 6 years ago by
First of all, thank you for your great job!
When using your latest changeset (r8781), there is a strange issue with some applications. If I run a QT application using wine, all menu elements (file menu, dropdowns, ...) open below the current window, not in front of it as expected. This only occurs when using the html5 client.
Do you have any idea what the problem might be? What information should I provide you in order to locate the issue?
Another issue is about the keyboard. When using a German keyboard layout, I can type all letters and numbers. There are some special characters like the exclamation mark I can type as well, but there is e.g. no key mapped to the period, so I'm unable to type it. Is it a known issue? Is there a way to configure the keyboard usage to include all German keys, maybe even including German "Umlauts"?
comment:41 follow-up: 43 Changed 6 years ago by
comment:42 Changed 6 years ago by
H.264 decoding support is stabilising in r9045. It will start to be selected if mutually available since r9049.
Use the new connection interface introduced in r9050 to change the preferred encoding by navigating to http://server:port/connect.html
The default behaviour without specifying /connect.html is still to connect automatically. If the connection is not successful, it will redirect to the connection interface.
comment:43 Changed 6 years ago by
Replying to joshiggins:
r9046 adds a style class for the window type MENU (we only had DROPDOWN_MENU and POPUP_MENU). Hopefully there shouldn't be any more menu elements appearing behind windows.
Thank you for your update! Unfortunately the problem still persists for my useless. I'd provide a Dockerfile, but this didn't work the last time. All I did was install a current wine version and ran few Qt Applications. None of them (like VLC in the pictures above) worked properly: They displayed all menu items, dropdowns and tooltips behind the window. I had to move the window away in order to see them, and be able to select an entry.
Do you have any other idea what I might try, or what debug / log information I should provide?
I've created a ticket for the keyboard problem #837.
Great, thank you!
comment:44 follow-up: 46 Changed 6 years ago by
@extasic I can't seem to reproduce this with the latest revision.
The wine applications use the DIALOG type hint for popups but these are still correctly placed above the WINDOW type.....
If you can tell me which window class is applied to your popup windows it would be useful. This is easy in chrome just by right clicking and selecting inspect element. It will show the html markup which should specify a class="window-WINDOW_TYPE" for the popup window.
Failing that I can take another stab at the Dockerfile.
comment:45 Changed 6 years ago by
Not had time to look into it, but with the latest trunk I get:
disconnect: invalid packet format, not an xpra client?
It was fine just hours/days ago.
comment:46 Changed 6 years ago by
Replying to joshiggins:
@extasic I can't seem to reproduce this with the latest revision.
The wine applications use the DIALOG type hint for popups but these are still correctly placed above the WINDOW type.....
If you can tell me which window class is applied to your popup windows it would be useful. This is easy in chrome just by right clicking and selecting inspect element. It will show the html markup which should specify a class="window-WINDOW_TYPE" for the popup window.
This is how the resulting HTML looks in Chrome. The highlighted Canvas represents a file menu (just like in the VLC picture I posted above). The div above the dialog one represents the main window. So there might be an error in the window type, right? But shouldn't there be something like a z-index anyway? Any workaround for this?
Just to mention it again: This error occurs for me only when running any Qt application in wine. If I run a non-Qt application in wine, or a Qt application directly in Linux, I couldn't reproduce the error. If I use the native xpra client, the error doesn't occur, even when running a Qt application under wine.
comment:47 Changed 6 years ago by
My guess is that those applications use the wrong window type, it probably should not be NORMAL (sadly, lots of applications do that - ie: chrome does). We have some code in the xpra client to try to workaround such broken behaviour.
@joshiggins: maybe the window is "transient-for" or "override-redirect" and you can use that?
comment:48 follow-up: 53 Changed 6 years ago by
Added a workaround in r9138. I was able to reproduce with VLC under Wine, it creates NORMAL override-redirect windows for the menus.
comment:49 Changed 6 years ago by
You can ignore comment:45, I was hitting the wrong test server - I need sleep!
comment:50 Changed 6 years ago by
I've uploaded a short video clip demonstrating the current state of H.264 decoding as a result of changes referred to in comment:42
https://www.youtube.com/watch?v=3yOoGWGyvgM
It's actually pretty good and remains usable.
comment:52 Changed 6 years ago by
And would like to say thank you to Josh for your awesome work on the Xpra Html5 client and our requests.
comment:53 Changed 6 years ago by
Replying to joshiggins:
Added a workaround in r9138. I was able to reproduce with VLC under Wine, it creates NORMAL override-redirect windows for the menus.
Works for me too, thank you!
comment:55 Changed 6 years ago by
Resolution: | → fixed |
---|---|
Status: | new → closed |
It's looking good for the release!
I have tagged v0.15.x, so let's move new work to a new ticket: #850
It will be easier to know what was delivered in this release and what remains to be done.
If you are hitting this ticket wanting to report a bug, please open a new ticket instead.
comment:57 Changed 6 years ago by
There is now a wiki page for the HTML5 client: wiki/Clients/HTML5.
comment:58 Changed 3 months ago by
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/473
experimental html5 client is now capable of showing an xterm window!