xpra icon
Bug tracker and wiki

Opened 5 months ago

Last modified 4 weeks ago

#2312 new task

clipboard images with html5 client

Reported by: Antoine Martin Owned by: alas
Priority: major Milestone: 3.0
Component: html5 Version: 2.5.x
Keywords: Cc:

Description (last modified by Antoine Martin)

We want to transfer the pixel data as image/png and set it as best we can using setData or newer methods (#1844).

See THE DEFINITIVE GUIDE TO COPYING AND PASTING IN JAVASCRIPT

The difficulty here is that because the html5 clibpoard is greedy, we need to choose png as soon as we get the clipboard event so we can send the binary data with the token.

Some pointers in ticket:812#comment:29

Attachments (3)

clipboard.html (2.5 KB) - added by Antoine Martin 5 months ago.
test page that crashes the chrome tab
clipboard-html5-images.patch (11.3 KB) - added by Antoine Martin 5 months ago.
support for images in and out with latest chrome - almost working
clipboard-fixed.html (2.5 KB) - added by Antoine Martin 3 months ago.
working clipboard image copy

Download all attachments as: .zip

Change History (16)

comment:1 Changed 5 months ago by Antoine Martin

Description: modified (diff)
Status: newassigned

comment:2 Changed 5 months ago by Antoine Martin

More pointers:

Raising #1844.

comment:3 Changed 5 months ago by Antoine Martin

As per Chrome Platform Status: Asynchronous Clipboard API: Support for images and other data types is still under development.

comment:4 Changed 5 months ago by Antoine Martin

MDN: Clipboard: only Firefox has support for read() and write. (since Firefox 63)

comment:5 Changed 5 months ago by Antoine Martin

Firefox only: clipboard.setImageData The image is provided as an ArrayBuffer? containing the encoded image. JPEG and PNG formats are supported.

Chrome: chrome.clipboard setImageData : Since Chrome 76. Warning: this is the current Dev channel.

comment:6 Changed 5 months ago by Antoine Martin

The image API is now available in chrome beta: https://bugs.chromium.org/p/chromium/issues/detail?id=150835#c118 This can also be enabled in M75 (released in stable tomorrow) through enabling the experimental flag here:
chrome://flags/#enable-experimental-web-platform-features
(use this as a link)

Last edited 4 months ago by Antoine Martin (previous) (diff)

Changed 5 months ago by Antoine Martin

Attachment: clipboard.html added

test page that crashes the chrome tab

Changed 5 months ago by Antoine Martin

support for images in and out with latest chrome - almost working

comment:7 Changed 5 months ago by Antoine Martin

The patch above almost works, but there is a problem with chrome: the tabs crash... reproducer in attachment/ticket/2312/clipboard.html.

With the new read() and write() API calls, we could modify the new clipboard code to work more like the regular X11 asynchronous clipboard and return the list of data types it supports, without bundling the data with it. Saving lots of bandwidth and making it more reliable when there are multiple options: the client application can then choose the format most suitable instead of us picking one.
This would require a reversible mapping from clipboard data types to the wire ones. Tricky?

It would be useful to collect the list of mimetypes we can get from the async API so we can handle them properly rather than assuming we just get UTF8 for text and image/png for images.


Related links:

Last edited 5 months ago by Antoine Martin (previous) (diff)

comment:8 Changed 5 months ago by Antoine Martin

Merged the image transfer clipboard code (disabled by default) in r22857.
(so it won't bitrot)

comment:9 Changed 4 months ago by Antoine Martin

chrome bug reported here: clipboard.write can crash the tab

comment:10 Changed 3 months ago by Antoine Martin

They claim that the bug has been fixed, but it's still present in the latest beta builds.
Version 76.0.3809.80 (Official Build) beta (64-bit)

Also, the png data from the example I have given is clearly valid, so it should not be throwing a DOMException in the first place: bug 981006: comment 22

Changed 3 months ago by Antoine Martin

Attachment: clipboard-fixed.html added

working clipboard image copy

comment:11 Changed 3 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas
Status: assignednew

Updates:

  • r23310 fixes the Blob instantiation, copying images to the clipboard works!
  • r23311 (+rr23312 fixup) enables it by default
  • r23313 same for server-to-client

Tested by:

  • starting a TCP server
  • connecting with chrome 76 and enabling the feature (see comment:6)
  • within the session, using eog to browse some local images
  • then control-C to copy the image to the clipboard
  • on the client, run gimp and create a new picture
  • paste the image into gimp

Things to consider:

  • the copy is asynchronous and if you alt-tab away from the browser window quickly after copying the image, chrome will fail and complain that the document is not active (as a log message only..)
  • size limit: clipboard transfers are not rate-limited, so maybe we should re-use the file-transfer code for chunking them? or downsize the image if we find that the data is too big? (16MB is now the limit - which is pretty big for one packet!)
  • we don't test for the "png" clipboard feature in the browser, only for the new API availability - though in practice the end result is almost the same: if copying images is not supported then it will fail but transferring an image as plain text is unlikely to be useful anyway (ie: with eog, you get the file path as text, which is not useful)
  • having clipit running can still cause problem, even though we try to ignore it
  • some clipboard attributes should be refactored: #2370
  • when sharing the same session with multiple clients, the type of data transferred for clipboard copying will change depending on what the client is capable of handling (should work - not tested)
  • could add "jpeg" support, looks like chrome has this already
  • by parsing image data on the server side (adding the image timestamp and optional overlay), we ensure that the image data is valid png as produced by python-pillow, but this may also strip metadata from the file

@afarr: this should be of interest to you

Last edited 3 months ago by Antoine Martin (previous) (diff)

comment:12 Changed 4 weeks ago by alas

Just as an FYI - Chrome 76 now seems to be stable (76.0.3809.132), but testing without nocicing the enabling detail of comment6... it is still failing.

comment:13 Changed 4 weeks ago by Antoine Martin

it is still failing.

Does it work if you enable the experimental features as per comment:6?

Note: See TracTickets for help on using tickets.