xpra icon
Bug tracker and wiki

Opened 19 months ago

Closed 18 months ago

Last modified 17 months ago

#988 closed defect (fixed)

fix screen capture tools when used with xpra

Reported by: Dweller Owned by: alas
Priority: major Milestone: 0.16
Component: core Version: 0.15.x
Keywords: Cc:

Description

This bit of java should save a screengrab.. but all it saves is black rectangles where the Xpra windows are..

I suspect the underlying cause is the same one why my java based instant messaging client is unable to capture part of my screen to send to other participants. And likely also why screen casting in java based meeting software isn't functional.

import java.awt.AWTException;
import java.awt.GraphicsDevice;
import java.awt.GraphicsEnvironment;
import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import javax.imageio.ImageIO;

public class Test {

	public static void main(String[] args) throws AWTException, IOException {
		GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment();  
		GraphicsDevice[] screens = ge.getScreenDevices();       
		Rectangle allScreenBounds = new Rectangle();  
		for (GraphicsDevice screen : screens) {  
		       Rectangle screenBounds = screen.getDefaultConfiguration().getBounds();        
		       allScreenBounds.width += screenBounds.width;  
		       allScreenBounds.height = Math.max(allScreenBounds.height, screenBounds.height);
		       allScreenBounds.x=Math.min(allScreenBounds.x, screenBounds.x);
		       allScreenBounds.y=Math.min(allScreenBounds.y, screenBounds.y);
		      } 
		Robot robot = new Robot();
		System.out.println("Grabbing screen using awt robot.");
		BufferedImage bufferedImage = robot.createScreenCapture(allScreenBounds);
		File file = new File("screengrab.png");
		if(!file.exists())
		    file.createNewFile();
		FileOutputStream fos = new FileOutputStream(file);
		ImageIO.write( bufferedImage, "png", fos );
		fos.close();
		System.out.println("Done. File has been written to "+file.getAbsolutePath());
	}
}

Attachments (2)

paint-root-overlay.patch (1.3 KB) - added by Antoine Martin 19 months ago.
PoC
paint-root-overlay-v2.patch (7.2 KB) - added by Antoine Martin 19 months ago.
much better implementation: with timer, command line option, etc..

Download all attachments as: .zip

Change History (16)

comment:1 Changed 19 months ago by Antoine Martin

Resolution: wontfix
Status: newclosed

We never composite the windows onto the virtual display, we send them to the client instead.

If you want the capture, run it where things are being displayed: on the client, not the server.

Not going to fix this, sorry.

comment:2 Changed 19 months ago by Dweller

Running the capture on the client OS isn't an option, because the chat client is running on the machine with the xpra server. So although yes, the capture code is running on the xpra server, it's doing so as part of an app being rendered via an xpra client.

I'm using xpra to display apps running on my remote server, one of those apps is an instant messaging client written in java, and one of it's features is to allow capturing part of the screen to send to a chat participant.

That feature of the chat client is non functional when the client is run via xpra, because all it can grab is black rectangles.

The same issue occurs when using e-meeting tools that allow sharing of the desktop, or part of the desktop.

I'm not expecting it to be able to grab windows that are not rendered by xpra, but did kinda expect it to be able to grab images of the ones that were.

I'm working round this today by launching an additional XServer on the client OS, and killing off & retargetting apps to that x-display when I need to share images from them.. which is less than ideal.

comment:3 Changed 19 months ago by Antoine Martin

Milestone: 0.16future
Resolution: wontfix
Status: closedreopened

We could add an option to paint onto the vfb.
This would need to be off by default, but I guess it could also be useful for debugging (the rare cases where "xpra screenshot" is not a good solution?)

The right place to do this is probably in the x11 server's damage function: call window.get_image and paint onto the root window at the window's coordinates.
Then we also need to handle window moves, resizes and destroy events: we have to repaint the vfb, initially we can just repaint everything in window z-order when that happens - and maybe do it via a timer so we don't overload the UI thread. The new option could be something like sync-vfb=off|delay-in-ms.

Last edited 19 months ago by Antoine Martin (previous) (diff)

Changed 19 months ago by Antoine Martin

Attachment: paint-root-overlay.patch added

PoC

comment:4 Changed 19 months ago by Antoine Martin

Milestone: future0.16

Turns out not to be terribly difficult I think.
r10681 adds the bindings we need, and the patch above paints the root overlay window. It's leaky and inefficient, doesn't deal with window positions or resizes or moves... but screengrabs show the updated window contents.

This is yet another reason for dealing with the geometry bits from #907.

Came across this ticket: JDK-6903034 : java.awt.Robot.createScreenCapture() doesn't work for translucent windows

Last edited 19 months ago by Antoine Martin (previous) (diff)

Changed 19 months ago by Antoine Martin

Attachment: paint-root-overlay-v2.patch added

much better implementation: with timer, command line option, etc..

comment:5 Changed 19 months ago by Antoine Martin

Owner: changed from Antoine Martin to Dweller
Status: reopenednew

@Dweller: apply the v2 patch on top of trunk and run the server with:

xpra start --sync-vfb=50

You should then be able to do screengrabs.

If that works for you then re-assign to me and I'll fix it up.
(ie: the blue and red colours are swapped because we paint via gdk which expects RGBX and we feed it BGRX from the xshm image.. ideally we should be painting with xshm too for better performance, meh)

comment:6 Changed 19 months ago by Dweller

Owner: changed from Dweller to Antoine Martin

Yes, it worked.. as far as I could test it..

A side effect of the patch seemed to be to kill off the ability for me to click on things.. I can click to activate windows, but not click on buttons, mouseover seemed to stop triggering.. and generally apps seemed to be missing my mouse messages.

So I could test with my sample code, as that's all pure command line, and yes, that grabs a screenshot that shows the xpra windows all present. But I couldn't actually test with the real chat client / emeeting stuff, because I although I could trigger their screen grab from a menu via the keyboard, I couldn't then dragselect the area of screen to capture as it ignored the mouse clicks.

Backported to 0.15.6, in case the lost mouse clicks were related to my build of trunk, but had the same behavior there. Tried setting the flag value to 0 and the function disables as expected, and then mouse clicks work just fine.

Intruigingly, the background for wbar changed from it's default grey to black while the patch was active.. (if wbar is run via a normal xserver, it's background is transparent) .. so looks like there could be ui effects beyond just screen grabs.

Sooo close !!

comment:7 Changed 19 months ago by Antoine Martin

Owner: changed from Antoine Martin to Dweller

Merged in r10682. Note: the flag has been renamed to sync-xvfb to make it more consistent with the rest of the options and docs.

It is far from perfect, it is still very slow for a start, but it is usable and as good as it is going to get for now.

I have added a one pixel border around the windows to make it a bit clearer. If we really wanted to, we could also paint a window top bar since we have the client's frame extents now #919 (meh, we don't have the top bar style or buttons to paint anyway).


A side effect of the patch seemed to be to kill off the ability for me to click on things


Fixed thanks to the excellent page so you want to build a compositor.


Intruigingly, the background for wbar changed from it's default grey to black while the patch was active..


Fixed in r10686, which also fixes the z-ordering of windows by keeping track of focus events. (the ones that had focus more recently are painted on top of others)

Potential future work (feel free to create a new ticket):

  • performance: should be using xshm / xcopy and not roundtrip via GTK which is terribly slow
  • could repaint only what has changed (very hard to do - meh)
  • this could be enabled / disabled at runtime via control channel

@Dweller: please test and close if this works for you.

comment:8 Changed 19 months ago by Antoine Martin

Owner: changed from Dweller to alas

@afarr: FYI as this may be of interest to you, we can now capture the virtual screen contents using regular capture tools and screen sharing tools should now work. Please close.

comment:9 Changed 19 months ago by Dweller

Did some testing.. it does function as far as screen grabbing IS possible when this patch is active.. but it also impacts the stability of the xpra server.. some apps that use screen grabbing seem capable of crashing the Xvfb process entirely.. (ok, not strictly xpra.. but same effect when all your apps are killed off) after limited investigation it would seem the kernel is killing off the Xvfb process due to memory constraints.

I'll try to find time to test again with a modified version of the test code that grabs in a loop.. if there's a leak somewhere maybe that can drive it enough to debug where..

Or maybe I should try and put Xdummy onto ubuntu instead of Xvfb ?

comment:10 Changed 19 months ago by Antoine Martin

@Dweller: Xvfb is only used as a workaround on Ubuntu, and since the leak seems to come from this process... this really does not look like an xpra problem.

comment:11 Changed 19 months ago by Dweller

I'll try the instructions at wiki/Xdummy for ubuntu, and see how I get on =)

Last edited 17 months ago by Antoine Martin (previous) (diff)

comment:12 Changed 19 months ago by Antoine Martin

Summary: Java cannot capture window contents when windows are displayed via Xprafix screen capture tools when used with xpra

Some updates: r10797 (fix for trays), r10798 (paints workarea and monitor bounds)

comment:13 Changed 18 months ago by alas

Resolution: fixed
Status: newclosed

Yup, tested with 0.16.0 r10922 (OSX) client against a 0.16.0 r10916 server (using scrot, which was a new trick for me).

Worked as expected on the second try, but the first try only captured one of two xterms as well as a chromium window that was started from the xterm that wasn't captured.

I'm going to guess that this is a matter of the "It is far from perfect" stipulation of comment:4, and close.

comment:14 Changed 18 months ago by Antoine Martin

I can't reproduce the "second xterm not captured" - which should have worked, so leaving this closed.

Note: See TracTickets for help on using tickets.