The current shadow server for win32 sort of works but we need to improve:
Screenscraping may be the only feasible way to get 3D apps to work: Since the hardware-accelerated 3D rendering commands typically bypass the GDI layer, most screen scrapers (WinVNC, LiveMeeting?, GoToMeeting?, etc.) cannot pick up the 3D rendering area of Windows applications, or they will not detect when the 3D area has been updated.
(from DRC @virtualgl-users)
Scheduling for 0.16
Some fixes:
Generic Window API links:
Window Event Hooks:
WH_SHELL (The function receives notifications of Shell events from the system.): ShellProc callback function, in particular:
HSHELL_REDRAW
HSHELL_WINDOWACTIVATED, HSHELL_WINDOWCREATED, HSHELL_WINDOWDESTROYED and HSHELL_WINDOWREPLACED, which we could tie with the list of rectangles if we keep the hwnd at hand.
WH_CBT (The system calls this function before activating, creating, destroying, minimizing, maximizing, moving, or sizing a window):
Example code:
r10002 uses Pillow's ImageGrab module to capture the screen on win32, which is already much faster.
It is still quite slow: timeit clocks it at about 100ms for 1080p on a mid-range system, which is a lot better than GTK, but has room for improvement:
tobytes - which makes a copy, we could avoid this by going pure native via pywin32 and managing the target buffer's lifetime ourselves (and ideally, straight into nvenc's input buffer - tall order!)
BGRX format so that we can use nvenc... (we could use whatever RGB format is native to win32 and just add the necessary kernel to nvenc instead, see #558)
Some links:
The code in r10002 only captured one screen, this is fixed in r10026, see ticket:637#comment:11 for details. Will backport.
r10028 should help speed things up a bit by re-using the same bitmap and context objects until the display settings change. But this is still not good enough: it takes about 100ms for copying 4 megapixels (2 x 1080p screens). About 65% of this time is spent in GetBitmapBits. The BitBlt performance with Aero enabled is just poor. Disabling aero is a temporary workaround.
We want to stay on the GPU if we can (#365), especially if we're going to use the GPU for encoding (ie: nvenc) - the difference is in the order of a factor 100!
More info:
This will have to do for now. (nvenc needs fixing first anyway... see #389)
Re-scheduling.
r11482 added a basic systray for the shadow server which only shows: "about", "exit" and "close menu".
work in progress named pipes support
Moving the named pipes to #1150.
Keyboard issues in #1099.
Stumbled upon this info python-win32: Global Window Messages: What you probably want is a WH_CBT hook so that you can catch WM_ACTIVATE and WM_DEACTIVATE. There is some code that does something similar in pyAA apparently.
Re-scheduling.
See my original question to python-win32 mailing list: https://mail.python.org/pipermail/python-win32/2009-September/009584.html. And KeyboardHookProc in DLL doesn't do anything when called from python
This is going to take some effort and will require Windows 8 or later and probably building things with MSVC 2013 (not the "free" express version..) for the newer API: IDXGIOutputDuplication: The IDXGIOutputDuplication interface accesses and manipulates the duplicated desktop image.
Printing may be possible using redmon: The RedMon? port monitor redirects a special printer port to a program
For sound see: wasapisink
r12394 will use directsoundsrc by default for shadow servers
r12850 disables composition on vista and later, r12852 + r12853 + r12854 improve error handling - which fires a lot more with vista onwards when the UAC prompt shows up.
Milestone renamed
A good solution for solving shadow performance issues is #1317 (requires an nvidia card...)
If we go down the directx route, with mingw #678 these may be helpful / needed:
g++ dxtest.cpp -o dxtest.exe -I/dxsdk/include -L/dxsdk/lib -DUNICODE -ld3d9
SetWindowsHookEx.
GetFrontBufferData...
So, by injecting the DLL into every process, we could get individual windows as surfaces.
If we are willing to require windows 8 or later, the solution is likely to be the Desktop Duplication API:
Some links on the logon stuff, which we need for running as a service:
For audio:
Added basic NVIDIA Capture SDK support using basic "copy to system memory" API. Screen capturing + downloading to system RAM at 4K takes less than 30ms, so we should be able to do 30fps at 4K (given enough bandwidth / fast encoder).
Much more can be done with this API:
Minor updates in r15727 + r15730. Helped uncover a core imagewrapper bug: r15728.
The NVFBC progress should now be tracked here: #1317.
Some scripts that could be useful - and not just for shadow servers:
Here's some ctypes code to access the data:
from comtypes.client import CreateObject
o=CreateObject('WbemScripting.SWbemLocator')
s=o.ConnectServer('.', 'root\\cimv2')
res=s.ExecQuery('SELECT * FROM Win32_DisplayConfiguration')
print([int(x.Properties_['BitsPerPel'].Value) for x in res])
Could also be useful for detecting virtualbox:
print([(x.Properties_['DeviceName'].Value) for x in res]) [u'VirtualBox Graphics Adapter']
Now for making a service:
Moving the system service to #1527.
re-scheduling
Python3 #1568 is making progress, this may help.
For capturing the screen with directx, a good example is found in the gstreamer source: https://github.com/GStreamer/gst-plugins-bad/tree/master/sys/winscreencap.
See also multi-monitor handling as windows: #1805
NVFBC is now deprecated on windows-10: ticket:1317#comment:11. DDA example: nvEncDXGIOutputDuplicationSample
Links:
See also:
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/389