Xpra: Ticket #389: ms windows shadow server improvements

The current shadow server for win32 sort of works but we need to improve:



Sun, 18 Aug 2013 04:15:43 GMT - Antoine Martin: owner, status changed

Screenscraping may be the only feasible way to get 3D apps to work: Since the hardware-accelerated 3D rendering commands typically bypass the GDI layer, most screen scrapers (WinVNC, LiveMeeting?, GoToMeeting?, etc.) cannot pick up the 3D rendering area of Windows applications, or they will not detect when the 3D area has been updated.

(from DRC @virtualgl-users)


Tue, 19 Aug 2014 04:06:04 GMT - Antoine Martin: milestone changed

Scheduling for 0.16


Wed, 24 Sep 2014 15:07:30 GMT - Antoine Martin:

Some fixes:


Sun, 28 Jun 2015 09:52:00 GMT - Antoine Martin:

Generic Window API links:

Window Event Hooks:

Example code:


Wed, 22 Jul 2015 11:58:43 GMT - Antoine Martin:

r10002 uses Pillow's ImageGrab module to capture the screen on win32, which is already much faster.

It is still quite slow: timeit clocks it at about 100ms for 1080p on a mid-range system, which is a lot better than GTK, but has room for improvement:

Some links:


Fri, 24 Jul 2015 09:38:13 GMT - Antoine Martin:

The code in r10002 only captured one screen, this is fixed in r10026, see ticket:637#comment:11 for details. Will backport.

r10028 should help speed things up a bit by re-using the same bitmap and context objects until the display settings change. But this is still not good enough: it takes about 100ms for copying 4 megapixels (2 x 1080p screens). About 65% of this time is spent in GetBitmapBits. The BitBlt performance with Aero enabled is just poor. Disabling aero is a temporary workaround.

We want to stay on the GPU if we can (#365), especially if we're going to use the GPU for encoding (ie: nvenc) - the difference is in the order of a factor 100!

More info:

This will have to do for now. (nvenc needs fixing first anyway... see #389)


Fri, 23 Oct 2015 07:05:04 GMT - Antoine Martin: milestone changed

Re-scheduling.


Sat, 26 Dec 2015 17:41:37 GMT - Antoine Martin:

r11482 added a basic systray for the shadow server which only shows: "about", "exit" and "close menu".


Sun, 27 Dec 2015 08:11:04 GMT - Antoine Martin: attachment set

work in progress named pipes support


Fri, 25 Mar 2016 13:51:25 GMT - Antoine Martin:

Moving the named pipes to #1150.


Sat, 26 Mar 2016 11:59:32 GMT - Antoine Martin:

Keyboard issues in #1099.

Stumbled upon this info python-win32: Global Window Messages: What you probably want is a WH_CBT hook so that you can catch WM_ACTIVATE and WM_DEACTIVATE. There is some code that does something similar in pyAA apparently.


Tue, 29 Mar 2016 04:51:17 GMT - Antoine Martin: milestone changed

Re-scheduling.

See my original question to python-win32 mailing list: https://mail.python.org/pipermail/python-win32/2009-September/009584.html. And KeyboardHookProc in DLL doesn't do anything when called from python

This is going to take some effort and will require Windows 8 or later and probably building things with MSVC 2013 (not the "free" express version..) for the newer API: IDXGIOutputDuplication: The IDXGIOutputDuplication interface accesses and manipulates the duplicated desktop image.

Printing may be possible using redmon: The RedMon? port monitor redirects a special printer port to a program


Thu, 07 Apr 2016 11:37:31 GMT - Antoine Martin:

For sound see: wasapisink

r12394 will use directsoundsrc by default for shadow servers


Sun, 22 May 2016 11:20:34 GMT - Antoine Martin:


Fri, 17 Jun 2016 09:16:22 GMT - Antoine Martin:


Fri, 17 Jun 2016 13:28:41 GMT - Antoine Martin:

r12850 disables composition on vista and later, r12852 + r12853 + r12854 improve error handling - which fires a lot more with vista onwards when the UAC prompt shows up.


Tue, 12 Jul 2016 16:52:22 GMT - Antoine Martin: milestone changed

Milestone renamed


Wed, 07 Sep 2016 10:58:22 GMT - Antoine Martin: milestone changed


Tue, 20 Sep 2016 12:36:32 GMT - Antoine Martin:

A good solution for solving shadow performance issues is #1317 (requires an nvidia card...)


Thu, 09 Feb 2017 05:21:35 GMT - Antoine Martin:

If we go down the directx route, with mingw #678 these may be helpful / needed:

So, by injecting the DLL into every process, we could get individual windows as surfaces.

If we are willing to require windows 8 or later, the solution is likely to be the Desktop Duplication API:

Some links on the logon stuff, which we need for running as a service:

For audio:


Sun, 19 Feb 2017 06:36:41 GMT - Antoine Martin: milestone changed


Thu, 20 Apr 2017 16:59:47 GMT - Antoine Martin:

Added basic NVIDIA Capture SDK support using basic "copy to system memory" API. Screen capturing + downloading to system RAM at 4K takes less than 30ms, so we should be able to do 30fps at 4K (given enough bandwidth / fast encoder).

Much more can be done with this API:


Thu, 27 Apr 2017 17:20:25 GMT - Antoine Martin:

Minor updates in r15727 + r15730. Helped uncover a core imagewrapper bug: r15728.

The NVFBC progress should now be tracked here: #1317.


Tue, 23 May 2017 09:47:04 GMT - Antoine Martin:

Some scripts that could be useful - and not just for shadow servers:

Here's some ctypes code to access the data:

from comtypes.client import CreateObject
o=CreateObject('WbemScripting.SWbemLocator')
s=o.ConnectServer('.', 'root\\cimv2')
res=s.ExecQuery('SELECT * FROM Win32_DisplayConfiguration')
print([int(x.Properties_['BitsPerPel'].Value) for x in res])

Could also be useful for detecting virtualbox:

print([(x.Properties_['DeviceName'].Value) for x in res])
[u'VirtualBox Graphics Adapter']

Now for making a service:


Tue, 23 May 2017 11:00:46 GMT - Antoine Martin:

Moving the system service to #1527.


Mon, 10 Jul 2017 14:19:06 GMT - Antoine Martin: milestone changed

re-scheduling


Wed, 20 Sep 2017 06:40:35 GMT - Antoine Martin: milestone changed

Python3 #1568 is making progress, this may help.


Sat, 03 Feb 2018 05:42:00 GMT - Antoine Martin:

For capturing the screen with directx, a good example is found in the gstreamer source: https://github.com/GStreamer/gst-plugins-bad/tree/master/sys/winscreencap.

See also multi-monitor handling as windows: #1805


Mon, 07 May 2018 05:40:06 GMT - Antoine Martin: milestone changed


Fri, 05 Oct 2018 06:51:38 GMT - Antoine Martin: milestone changed


Tue, 25 Jun 2019 15:54:57 GMT - Antoine Martin:

NVFBC is now deprecated on windows-10: ticket:1317#comment:11. DDA example: nvEncDXGIOutputDuplicationSample


Sun, 04 Aug 2019 03:59:19 GMT - Antoine Martin: milestone changed


Mon, 02 Dec 2019 08:58:33 GMT - Antoine Martin:

Links:

See also:


Mon, 24 Feb 2020 15:39:21 GMT - Antoine Martin: milestone changed


Sun, 04 Oct 2020 14:34:26 GMT - Antoine Martin: milestone changed


Sat, 23 Jan 2021 04:53:56 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/389