xpra icon
Bug tracker and wiki

Opened 4 years ago

Last modified 4 weeks ago

#389 assigned enhancement

ms windows shadow server improvements

Reported by: Antoine Martin Owned by: Antoine Martin
Priority: minor Milestone: 2.3
Component: server Version:
Keywords: win32 Cc:

Description

The current shadow server for win32 sort of works but we need to improve:

Attachments (1)

namedpipes.patch (12.7 KB) - added by Antoine Martin 22 months ago.
work in progress named pipes support

Download all attachments as: .zip

Change History (27)

comment:1 Changed 4 years ago by Antoine Martin

Owner: changed from Antoine Martin to Antoine Martin
Status: newassigned

Screenscraping may be the only feasible way to get 3D apps to work:
Since the hardware-accelerated 3D rendering commands typically bypass the GDI layer, most screen scrapers (WinVNC, LiveMeeting?, GoToMeeting?, etc.) cannot pick up the 3D rendering area of Windows applications, or they will not detect when the 3D area has been updated.

(from DRC @virtualgl-users)

comment:2 Changed 3 years ago by Antoine Martin

Milestone: future0.16

Scheduling for 0.16

comment:3 Changed 3 years ago by Antoine Martin

Some fixes:

  • bug which caused 100% cpu on win32 shadow servers #691
  • r7794 keyboard is almost usable (but completely hackish: missing modifiers, no key repeat, etc)
  • + some minor tweaks: r7793, etc
Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:4 Changed 2 years ago by Antoine Martin

  • generic shadow improvements are tracked in #899
  • clipboard: makes sense to re-do the using native code as per #812.
  • keyboard: better modifiers handling?
  • nvenc: #558
  • use window list to split into distinct window sources? (see #501)
  • service? tray menu to enable sharing?
  • named pipes support so we don't have to require TCP: win32pipe

Generic Window API links:

Window Event Hooks:

  • Hooks: A global hook monitors messages for all threads in the same desktop as the calling thread. A thread-specific hook monitors messages for only an individual thread. A global hook procedure can be called in the context of any application in the same desktop as the calling thread, so the procedure must be in a separate DLL module.
  • RegisterShellHookWindow: This function is not intended for general use. It may be altered or unavailable in subsequent versions of Windows.
  • SetWindowsHookEx or SetWindowsHook: (needs the DLL to be injected? for all but mouse events)
    • with WH_SHELL (The function receives notifications of Shell events from the system.): ShellProc callback function, in particular:
      • HSHELL_REDRAW
      • maybe also: HSHELL_WINDOWACTIVATED, HSHELL_WINDOWCREATED, HSHELL_WINDOWDESTROYED and HSHELL_WINDOWREPLACED, which we could tie with the list of rectangles if we keep the hwnd at hand.
    • with WH_CBT (The system calls this function before activating, creating, destroying, minimizing, maximizing, moving, or sizing a window):
  • SetWinEventHook: Sets an event hook function for a range of events

Example code:

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:5 Changed 2 years ago by Antoine Martin

r10002 uses Pillow's ImageGrab module to capture the screen on win32, which is already much faster.

It is still quite slow: timeit clocks it at about 100ms for 1080p on a mid-range system, which is a lot better than GTK, but has room for improvement:

  • we call tobytes - which makes a copy, we could avoid this by going pure native via pywin32 and managing the target buffer's lifetime ourselves (and ideally, straight into nvenc's input buffer - tall order!)
  • we use the BGRX format so that we can use nvenc... (we could use whatever RGB format is native to win32 and just add the necessary kernel to nvenc instead, see #558)

Some links:

comment:6 Changed 2 years ago by Antoine Martin

The code in r10002 only captured one screen, this is fixed in r10026, see ticket:637#comment:11 for details. Will backport.

r10028 should help speed things up a bit by re-using the same bitmap and context objects until the display settings change.
But this is still not good enough: it takes about 100ms for copying 4 megapixels (2 x 1080p screens).
About 65% of this time is spent in BitBlt, and the rest is in GetBitmapBits.
The BitBlt performance with Aero enabled is just poor. Disabling aero is a temporary workaround.

We want to stay on the GPU if we can (#365), especially if we're going to use the GPU for encoding (ie: nvenc) - the difference is in the order of a factor 100!

More info:

This will have to do for now. (nvenc needs fixing first anyway... see #389)

Last edited 19 months ago by Antoine Martin (previous) (diff)

comment:7 Changed 2 years ago by Antoine Martin

Milestone: 0.160.17

Re-scheduling.

comment:8 Changed 22 months ago by Antoine Martin

r11482 added a basic systray for the shadow server which only shows: "about", "exit" and "close menu".

Changed 22 months ago by Antoine Martin

Attachment: namedpipes.patch added

work in progress named pipes support

comment:9 Changed 19 months ago by Antoine Martin

Moving the named pipes to #1150.

comment:10 Changed 19 months ago by Antoine Martin

Keyboard issues in #1099.

Stumbled upon this info python-win32: Global Window Messages: What you probably want is a WH_CBT hook so that you can catch WM_ACTIVATE and WM_DEACTIVATE.
There is some code that does something similar in pyAA apparently.

comment:11 Changed 19 months ago by Antoine Martin

Milestone: 0.170.18

Re-scheduling.

See my original question to python-win32 mailing list: https://mail.python.org/pipermail/python-win32/2009-September/009584.html.
And Win32 WH_CBT Hook - close window before it is created, WH_CBT hook, using SetWindowsHookEx to monitor windows events, KeyboardHookProc in DLL doesn't do anything when called from python

This is going to take some effort and will require Windows 8 or later and probably building things with MSVC 2013 (not the "free" express version..) for the newer API: IDXGIOutputDuplication: The IDXGIOutputDuplication interface accesses and manipulates the duplicated desktop image.

Printing may be possible using redmon: The RedMon? port monitor redirects a special printer port to a program

Last edited 19 months ago by Antoine Martin (previous) (diff)

comment:12 Changed 19 months ago by Antoine Martin

For sound see: wasapisink

r12394 will use directsoundsrc by default for shadow servers

Last edited 18 months ago by Antoine Martin (previous) (diff)

comment:13 Changed 17 months ago by Antoine Martin

  • r12647 should fix sound shadowing on win32 (untested - virtualbox is having a fit)
  • r12707 (skip warnings)
Last edited 17 months ago by Antoine Martin (previous) (diff)

comment:14 Changed 16 months ago by Antoine Martin

Last edited 16 months ago by Antoine Martin (previous) (diff)

comment:15 Changed 16 months ago by Antoine Martin

r12850 disables composition on vista and later, r12852 + r12853 + r12854 improve error handling - which fires a lot more with vista onwards when the UAC prompt shows up.

comment:16 Changed 15 months ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

comment:17 Changed 14 months ago by Antoine Martin

Milestone: 1.03.0

comment:18 Changed 13 months ago by Antoine Martin

A good solution for solving shadow performance issues is #1317 (requires an nvidia card...)

comment:19 Changed 8 months ago by Antoine Martin

If we go down the directx route, with mingw #678 these may be helpful / needed:

So, by injecting the DLL into every process, we could get individual windows as surfaces.

If we are willing to require windows 8 or later, the solution is likely to be the Desktop Duplication API:

Some links on the logon stuff, which we need for running as a service:

For audio:

Last edited 6 months ago by Antoine Martin (previous) (diff)

comment:20 Changed 8 months ago by Antoine Martin

Milestone: 3.02.1

comment:21 Changed 6 months ago by Antoine Martin

Added basic NVIDIA Capture SDK support using basic "copy to system memory" API.
Screen capturing + downloading to system RAM at 4K takes less than 30ms, so we should be able to do 30fps at 4K (given enough bandwidth / fast encoder).

Much more can be done with this API:

  • TODO: error recovery, re-instantiate context on recoverable failures, bubble up if unrecoverable
  • download to RGB since we don't use the alpha channel (will save memory bandwidth and CPU time)
  • integrate with NVENC - via CUDA?
  • port to Linux - the examples refuse to run on Fedora 26 (could be glvnd related?)
  • meh: support 32-bit mode?
  • ARGB10 10-bit colour modes
  • download as YUV420 when quality is set lower? (NVENC is probably better in all cases)

comment:22 Changed 6 months ago by Antoine Martin

Minor updates in r15727 + r15730. Helped uncover a core imagewrapper bug: r15728.

The NVFBC progress should now be tracked here: #1317.

Last edited 6 months ago by Antoine Martin (previous) (diff)

comment:23 Changed 5 months ago by Antoine Martin

Some scripts that could be useful - and not just for shadow servers:

Here's some ctypes code to access the data:

from comtypes.client import CreateObject
o=CreateObject('WbemScripting.SWbemLocator')
s=o.ConnectServer('.', 'root\\cimv2')
res=s.ExecQuery('SELECT * FROM Win32_DisplayConfiguration')
print([int(x.Properties_['BitsPerPel'].Value) for x in res])

Could also be useful for detecting virtualbox:

print([(x.Properties_['DeviceName'].Value) for x in res])
[u'VirtualBox Graphics Adapter']

Now for making a service:

Last edited 5 months ago by Antoine Martin (previous) (diff)

comment:24 Changed 5 months ago by Antoine Martin

Moving the system service to #1527.

comment:25 Changed 3 months ago by Antoine Martin

Milestone: 2.12.2

re-scheduling

comment:26 Changed 4 weeks ago by Antoine Martin

Milestone: 2.22.3

Python3 #1568 is making progress, this may help.

Note: See TracTickets for help on using tickets.