Opened 9 years ago
Last modified 16 months ago
#389 assigned enhancement
ms windows shadow server improvements
Reported by: | Antoine Martin | Owned by: | Antoine Martin |
---|---|---|---|
Priority: | minor | Milestone: | 4.2 |
Component: | server | Version: | |
Keywords: | win32 | Cc: |
Description
The current shadow server for win32 sort of works but we need to improve:
- keyboard mapping is not done: we must generate a lookup map for keycodes
- we may be able to export windows individually: Image.frombuffer with 16-bit image data (grabs from a specific hwnd)
- if we are going to screenscrape, then we should do it faster: Fastest method of screen capturing
- capture the cursor: How to make a bitmap of the current mouse pointer?
- fixed multi-monitor code? win32ui vs wxPy screen capture multi monitor
- Various methods for capturing the screen
- there is no way to connect to the server without tcp mode, maybe we should add a simple named pipe listener: createNamedPipe in python (also similar code in winswitch)
Attachments (1)
Change History (36)
comment:1 Changed 9 years ago by
Owner: | changed from Antoine Martin to Antoine Martin |
---|---|
Status: | new → assigned |
comment:4 Changed 7 years ago by
- generic shadow improvements are tracked in #899
- clipboard: makes sense to re-do the using native code as per #812.
- keyboard: better modifiers handling?
- nvenc: #558
- use window list to split into distinct window sources? (see #501)
- service? tray menu to enable sharing?
- named pipes support so we don't have to require TCP: win32pipe
Generic Window API links:
- EnumWindows (used by seamless mode)
- GetWindowInfo to get WINDOWINFO (size, border, type, status..)
- GetWindowRect
- GetWindowText
- IsWindowUnicode
Window Event Hooks:
- Hooks: A global hook monitors messages for all threads in the same desktop as the calling thread. A thread-specific hook monitors messages for only an individual thread. A global hook procedure can be called in the context of any application in the same desktop as the calling thread, so the procedure must be in a separate DLL module.
- RegisterShellHookWindow: This function is not intended for general use. It may be altered or unavailable in subsequent versions of Windows.
- SetWindowsHookEx or SetWindowsHook: (needs the DLL to be injected? for all but mouse events)
- with
WH_SHELL
(The function receives notifications of Shell events from the system.): ShellProc callback function, in particular:HSHELL_REDRAW
- maybe also:
HSHELL_WINDOWACTIVATED
,HSHELL_WINDOWCREATED
,HSHELL_WINDOWDESTROYED
andHSHELL_WINDOWREPLACED
, which we could tie with the list of rectangles if we keep the hwnd at hand.
- with
WH_CBT
(The system calls this function before activating, creating, destroying, minimizing, maximizing, moving, or sizing a window):
- with
- SetWinEventHook: Sets an event hook function for a range of events
Example code:
comment:5 Changed 7 years ago by
r10002 uses Pillow's ImageGrab module to capture the screen on win32, which is already much faster.
It is still quite slow: timeit clocks it at about 100ms for 1080p on a mid-range system, which is a lot better than GTK, but has room for improvement:
- we call
tobytes
- which makes a copy, we could avoid this by going pure native via pywin32 and managing the target buffer's lifetime ourselves (and ideally, straight into nvenc's input buffer - tall order!) - we use the
BGRX
format so that we can use nvenc... (we could use whatever RGB format is native to win32 and just add the necessary kernel to nvenc instead, see #558)
Some links:
comment:6 Changed 7 years ago by
The code in r10002 only captured one screen, this is fixed in r10026, see ticket:637#comment:11 for details. Will backport.
r10028 should help speed things up a bit by re-using the same bitmap and context objects until the display settings change.
But this is still not good enough: it takes about 100ms for copying 4 megapixels (2 x 1080p screens).
About 65% of this time is spent in BitBlt, and the rest is in GetBitmapBits.
The BitBlt performance with Aero enabled is just poor. Disabling aero is a temporary workaround.
We want to stay on the GPU if we can (#365), especially if we're going to use the GPU for encoding (ie: nvenc) - the difference is in the order of a factor 100!
More info:
- Ways to capture the screen, in particular with direct3d9: To do this with Direct3D, you could use GetRenderTargetData to copy the rendering surface (retrieved with GetRenderTarget) to an off-screen surface (which can be created with CreateOffscreenPlainSurface).
- Desktop Duplication API (Windows 8 onwards), looks like a proper solution to shadowing. We could also handle the rectangle "move" messages it sends, which is a lot better than plain "damage" events.
This will have to do for now. (nvenc needs fixing first anyway... see #389)
comment:8 Changed 6 years ago by
r11482 added a basic systray for the shadow server which only shows: "about", "exit" and "close menu".
comment:10 Changed 6 years ago by
Keyboard issues in #1099.
Stumbled upon this info python-win32: Global Window Messages: What you probably want is a WH_CBT hook so that you can catch WM_ACTIVATE and WM_DEACTIVATE.
There is some code that does something similar in pyAA apparently.
comment:11 Changed 6 years ago by
Milestone: | 0.17 → 0.18 |
---|
Re-scheduling.
See my original question to python-win32 mailing list: https://mail.python.org/pipermail/python-win32/2009-September/009584.html.
And Win32 WH_CBT Hook - close window before it is created, WH_CBT hook, using SetWindowsHookEx to monitor windows events, KeyboardHookProc in DLL doesn't do anything when called from python
This is going to take some effort and will require Windows 8 or later and probably building things with MSVC 2013 (not the "free" express version..) for the newer API: IDXGIOutputDuplication: The IDXGIOutputDuplication interface accesses and manipulates the duplicated desktop image.
Printing may be possible using redmon: The RedMon? port monitor redirects a special printer port to a program
comment:12 Changed 6 years ago by
For sound see: wasapisink
r12394 will use directsoundsrc by default for shadow servers
comment:14 Changed 6 years ago by
- we can disable compositing by calling: DwmEnableComposition. This will fix screen capture on Vista and later.
- maybe we should switch to using GetDIBits instead of GetBitmapBits, as per: Desktopmagic screengrab_win32 (ctypes), GetDIBits to PIL image example
- good directx example: Doing readback from Direct3D textures and surfaces, this is probably the right way of hooking up NVENC #558
- good IDXGIOutput1::DuplicateOutput example: RPI-GPU-rdpClient wddm.h
- ffmpeg option: gdigrab, directshow
comment:15 Changed 6 years ago by
comment:17 Changed 6 years ago by
Milestone: | 1.0 → 3.0 |
---|
comment:18 Changed 6 years ago by
A good solution for solving shadow performance issues is #1317 (requires an nvidia card...)
comment:19 Changed 5 years ago by
If we go down the directx route, with mingw #678 these may be helpful / needed:
- w32api-directx-standalone: Alternative DirectX headers for MinGW w32api headers
- setting up mingw for directx, ie:
g++ dxtest.cpp -o dxtest.exe -I/dxsdk/include -L/dxsdk/lib -DUNICODE -ld3d9
- Universal DirectX SDK: The goal of this open source project is build an universal DirectX SDK. It can work with multiple versions of Visual Studio, MinGW, and other compilers
- How to capture the screen in DirectX 9 to a raw bitmap in memory without using D3DXSaveSurfaceToFile
- Hooking DirectX EndScene from an injected DLL and in particular: Case study: Fraps and how it uses
SetWindowsHookEx
. - Capture screen using DirectX - but this uses the slow
GetFrontBufferData
... - Fastest method of screen capturing
- Various methods for capturing the screen
- Resource Management Best Practices
- D3D hooking
- C# – SCREEN CAPTURE AND OVERLAYS FOR DIRECT3D 9, 10 AND 11 USING API HOOKS
- Direct3DHook
- msdn: ways to capture the screen
- Execution time: GetFrontBufferData, GetBackBuffer, GetRenderTargetData
So, by injecting the DLL into every process, we could get individual windows as surfaces.
If we are willing to require windows 8 or later, the solution is likely to be the Desktop Duplication API:
- Desktop Duplication Sample
- DXGICaptureSample
- Very Fast Screen Capture With Windows 8, C# and DirectX11
Some links on the logon stuff, which we need for running as a service:
- LogonUser function
- Service functions
- Service Logon Accounts
- pywinrm: Python library for Windows Remote Management (WinRM)
- Impersonating Principals on Windows: You need to authenticate a thread temporarily as another principal on a Windows machine—for example, to make something run with the appropriate administrative rights.
For audio:
comment:20 Changed 5 years ago by
Milestone: | 3.0 → 2.1 |
---|
comment:21 Changed 5 years ago by
Added basic NVIDIA Capture SDK support using basic "copy to system memory" API.
Screen capturing + downloading to system RAM at 4K takes less than 30ms, so we should be able to do 30fps at 4K (given enough bandwidth / fast encoder).
Much more can be done with this API:
- TODO: error recovery, re-instantiate context on recoverable failures, bubble up if unrecoverable
- download to RGB since we don't use the alpha channel (will save memory bandwidth and CPU time)
- integrate with NVENC - via CUDA?
- port to Linux - the examples refuse to run on Fedora 26 (could be glvnd related?)
- meh: support 32-bit mode?
- ARGB10 10-bit colour modes
- download as YUV420 when quality is set lower? (NVENC is probably better in all cases)
comment:22 Changed 5 years ago by
comment:23 Changed 5 years ago by
Some scripts that could be useful - and not just for shadow servers:
- List Print Job Information - if we could somehow get a jobid from the print command, ADDJOB_INFO_1 has one, but we're not calling AddJob ourselves..
- List All Possible Video Controller Resolutions - for randr-like support
- List Current Display Configuration Values: could also be useful in the client, so we know the real display depth?
Here's some ctypes code to access the data:
from comtypes.client import CreateObject o=CreateObject('WbemScripting.SWbemLocator') s=o.ConnectServer('.', 'root\\cimv2') res=s.ExecQuery('SELECT * FROM Win32_DisplayConfiguration') print([int(x.Properties_['BitsPerPel'].Value) for x in res])
Could also be useful for detecting virtualbox:
print([(x.Properties_['DeviceName'].Value) for x in res]) [u'VirtualBox Graphics Adapter']
- List Sound Card Properties - so we can select the right one (but how to reconcile the deviceid with gstreamer's device number?)
- List Printer Capabilities / List Printer Information: export more printer information (orientation, etc)
Now for making a service:
comment:26 Changed 5 years ago by
Milestone: | 2.2 → 2.3 |
---|
Python3 #1568 is making progress, this may help.
comment:27 Changed 4 years ago by
For capturing the screen with directx, a good example is found in the gstreamer source: https://github.com/GStreamer/gst-plugins-bad/tree/master/sys/winscreencap.
See also multi-monitor handling as windows: #1805
comment:28 Changed 4 years ago by
Milestone: | 2.3 → 2.4 |
---|
comment:29 Changed 4 years ago by
Milestone: | 2.4 → 3.0 |
---|
comment:30 Changed 3 years ago by
NVFBC is now deprecated on windows-10: ticket:1317#comment:11.
DDA example: nvEncDXGIOutputDuplicationSample
comment:31 Changed 3 years ago by
Milestone: | 3.0 → 4.0 |
---|
comment:32 Changed 2 years ago by
Links:
- OBS has Windows Graphics Capture support: Windows Graphics Capture, which is new to Windows 10
- gstreamer: dxgiscreencapsrc: Add screen capture plug-in using Desktop Duplication API
See also:
comment:33 Changed 2 years ago by
Milestone: | 4.0 → 4.1 |
---|
comment:34 Changed 20 months ago by
Milestone: | 4.1 → 4.2 |
---|
comment:35 Changed 16 months ago by
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/389
Screenscraping may be the only feasible way to get 3D apps to work:
Since the hardware-accelerated 3D rendering commands typically bypass the GDI layer, most screen scrapers (WinVNC, LiveMeeting?, GoToMeeting?, etc.) cannot pick up the 3D rendering area of Windows applications, or they will not detect when the 3D area has been updated.
(from DRC @virtualgl-users)