xpra icon
Bug tracker and wiki

Opened 17 months ago

Last modified 6 weeks ago

#1844 new enhancement

async clipboard api

Reported by: Antoine Martin Owned by: alas
Priority: critical Milestone: 3.0
Component: html5 Version: 2.3.x
Keywords: Cc:

Description (last modified by Antoine Martin)

See Unblocking Clipboard Access: It's a replacement for execCommand-based copy & paste that has a well-defined permissions model and doesn't block the page.

MDN: Navigator clipboard.

Original html5 clipboard tickets: #842, #1461

Change History (13)

comment:1 Changed 6 months ago by Antoine Martin

See also related work in #812

comment:2 Changed 5 months ago by Antoine Martin

Status: newassigned

For images, see #2312

comment:3 Changed 5 months ago by Antoine Martin

Priority: majorcritical

Blocker for #2312

comment:4 Changed 5 months ago by Antoine Martin

Description: modified (diff)

Updates:

  • r22834 new async api calls
  • r22837 answer clipboard requests
  • r22838 workaround PRIMARY + CLIPBOARD overwriting values
  • r22839 don't bother synchronizing PRIMARY if we have the new async API
  • r22840 stricter test for navigator.clipboard API availability

Important: just found information on clipboardchange event in w3c: Async Clipboard API. This is not supported in any browser yet.
This means we would not necessarily have to claim the clipboard every time we get a token, we could rely on those events to notify the server.
For chrome: onClipboardDataChanged.

Maybe we should query the Permissions API rather than just checking if the method exists?
See Asynchronous Clipboard API Sample

Tested:

  • Chrome 75.0.3770.51 (Official Build) beta (64-bit) Linux: OK (connecting to localhost using websocket)
  • Chrome 74 MS Windows: OK via https, otherwise legacy mode
  • Firefox 67 Linux: legacy mode
  • Firefox 67 MS Windows: legacy mode

The Firefox results are surprising since they claim to support the async API since V63, but this matches the results from this test.
The spec says: Firefox only supports reading the clipboard in browser extensions, using the "clipboardRead" extension permission.

Then there are also issues with "secure contexts": works for localhost, otherwise https is required, etc..

Last edited 5 months ago by Antoine Martin (previous) (diff)

comment:5 Changed 4 months ago by Antoine Martin

IE fixes in r22862. (no arrow functions)

The problem with the permissions API is that if the user denies the request, we cannot ask again and we're left with a non-functional API..
To restore things, one has to visit chrome://settings/content/siteDetails?site=https%3A%2F%2Fgooglechrome.github.io%2F (chrome), or follow steps similar to How do I make Chrome forget a “no” to geolocation on a site?

Last edited 4 months ago by Antoine Martin (previous) (diff)

comment:6 Changed 4 months ago by Antoine Martin

Also fixes #2292

comment:7 Changed 4 months ago by Antoine Martin

Owner: changed from Antoine Martin to zaveri
Status: assignednew

For testing: the html5 client will print a message to the console indicating if it's using the old or new (async) clipboard code. (message actually fixed in r22911)

Tweaks and better compatibility for legacy mode in r22918.
IE compatibility fixes in r22919 + r22920.

The new code saves having to synchronize the PRIMARY selection constantly, we do that with the old code just in case we get a clipboard copy request from the browser so that we can provide a response instantly. (this is the synchronous clipboard)
With the new async code, we populate the clipboard only when needed.

comment:8 Changed 8 weeks ago by Antoine Martin

Owner: changed from zaveri to alas

@afarr: can you take over zaveri?
This ticket goes with #2312.

comment:9 Changed 7 weeks ago by alas

Well, as a start I did a test of the clipboard with the windows client (3.0r23707) against a 3.0-r23689 server on Fedora 30 - and none of my old tricks was able to find a problem.
Will update with results from html5 client tests.

comment:10 Changed 6 weeks ago by alas

Testing with the same 3.0-r23689 server on Fedora 30 with Chrome html5 client, it looks like nothing copied from a local application can be pasted into the html5 client session with a right-click menu paste option (not into a browser input field, or a gedit field) - but the contents are being sync'd with the server in some way, because immediately following a right-click paste (which will paste the last contents copied/pasted server-side) failure an immediate use of keyboard shortcut (control-v in this case) will paste the contents last copied locally.

Repro (that got confusing just writing it).

  • Copy something in a session.
  • Paste inside the session (right-click menu or keyboard shortcut) just to be sure what's there (should succeed).
  • Paste into a local application (should succeed, right-click of keyboard shortcut).
  • Copy something else in local application (right-click or keyboard shortcut). (Optionally then paste it again locally just to be sure nothing super-weird is happening.)
  • Return to application in the session (browser, gedit or other editor, not xterm - they behave uniquely) & use right-click to paste (should fail, pasting whatever was copied inside the session last, rather than what was copied from local application).
  • Immediately use keyboard shortcut to paste same contents into session application (now it should work, pasting different contents than what was just pasted with the right-click).

As an added bonus, the issue of #2403 (more than 30 clipboard requests per second easily triggered with gedit) is equally easy to trigger with the Chrome html5 client (and equally easy to disable clipboard entirely, requiring a disconnect and reconnect of the session to re-enable the clipboard).

Additionally (and I'm not sure if this should be a separate ticket), after triggering the warning for about 544 ms I somehow triggered a keyboard layout error?

2019-09-05 13:35:36,696 client   2 broadway decoder initialized
2019-09-05 13:35:43,892 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:43,909 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:43,909 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:43,925 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:43,942 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:43,958 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:43,978 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:43,992 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,009 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,025 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,042 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,074 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,074 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,103 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,109 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,159 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,160 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,247 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,264 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,335 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,351 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,369 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,384 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,402 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:44,436 Warning: more than 30 clipboard requests per second!
2019-09-05 13:35:46,720 setting keyboard layout to 'us'
The XKEYBOARD keymap compiler (xkbcomp) reports:
> Internal error:   Could not resolve keysym XF86MonBrightnessCycle
> Internal error:   Could not resolve keysym XF86RotationLockToggle
Errors from xkbcomp are not fatal to the X server

Will check with another html5 client when next I get a chance.

comment:11 Changed 6 weeks ago by Antoine Martin

Owner: changed from alas to Antoine Martin
Status: newassigned

... looks like nothing copied from a local application can be pasted into the html5 client session with a right-click menu paste option ...

Here's what's happening here: when you copy something using a local application, we currently have no way of knowing that something has happened to the local clipboard and that some new data is available.. (unless we use some sort of polling)
So when you right click in the server side application and use the popup menu to do something with the clipboard, we're still using the old value.
Control-V on the keyboard is handled differently: we detect this specific key combination (and some others, Shift+Insert, macos ones) and proactively query the local clipboard to send the new clipboard contents before sending the key event, so the server will have the latest clipboard contents before handling the clipboard shortcut.
I don't think we can use a similar strategy for clicks as there is no way of knowing what the click is meant to do: the click could be on a "cut" or "copy" button or menu entry, just as well as a "paste" button. There's no way to tell.
And proactively copying the data might just override the application's intended action for this event.

I'm experimenting with a polling solution, which is tricky.

As an added bonus, the issue of #2403 (more than 30 clipboard requests per second easily triggered with gedit) is equally easy to trigger with the Chrome html5 client

It shouldn't be. We're not supposed to synchronize the PRIMARY selection at all when the async clipboard code is used. And the CLIPBOARD selection shouldn't be changing that often.
I'll look into it.

(and equally easy to disable clipboard entirely, requiring a disconnect and reconnect of the session to re-enable the clipboard).

That's no longer an issue: r23723, ticket:2403#comment:1.

comment:12 Changed 6 weeks ago by Antoine Martin

Owner: changed from Antoine Martin to alas
Status: assignednew

Try r23724: whenever we get a click event, we now poll the local clipboard for changes.

Notes and caveats:

  • may trigger clipboard warnings with some browser versions and security settings - so maybe this new feature should be made optional?
  • tested on Chrome only for now - Firefox isn't working (will look into it - maybe not allowed from click events?)
  • ideally the code should be modified to use an asynchronous call, to guarantee that the clipboard contents are sent before the click - meh: difficult to read, and may delay the clicks too much
  • if this approach works well enough, maybe we should use it more so that we can catch weird keyboard shortcuts? (ie: fire the clipboard polling code on every key event? probably not with Firefox..)
  • I still need to look into the PRIMARY 30 requests per second issue
Last edited 6 weeks ago by Antoine Martin (previous) (diff)

comment:13 Changed 6 weeks ago by Antoine Martin

Updates:

  • Firefox is unlikely to work: readText: Firefox only supports reading the clipboard in browser extensions, using the "clipboardRead" extension permission.
  • MS Edge and IE11 don't have any support for the new clipboard API, but using the legacy API seems to work for polling (needed fixes from r23726 + r23727) - tested various combinations downloaded from microsoft.com : Download virtual machines
  • I am unable to reproduce any problems with the PRIMARY selection: although I do see events firing on the server-side with -d clipboard, those events never get sent to the browser as it doesn't enable this selection:
    $ xpra info | grep PRIMARY.enabled
    clipboard.PRIMARY.enabled=False
    
Last edited 6 weeks ago by Antoine Martin (previous) (diff)
Note: See TracTickets for help on using tickets.