improve webcam support

[Antoine Martin]

Follow up from #1030. See wiki/Webcam.

• there is no systray menu for it
• framerate is low
• black and white only, low resolution (hardcoded csc dependencies)
• no support for multiple sessions per server..
• detecting when devices are added or removed? (dbus?)
• it uses too much bandwidth (mostly wasted too)
• Fedora dkms packaging for v4l2loopback?
• investigate win32 and osx support (shadow server / remote webcam)

[Antoine Martin]

We should also add support for native H264: ​Using the Logitech C920 webcam with Gstreamer

[Antoine Martin]

Updates:

• finally found ​the fix for the offset hack: r12822 (will backport)
• update webcam tray menu when devices are added or removed (linux only via python-inotify): r12819, packaging updates in r12821, fixup in r12820
• r12823 should now support multiple users: load the kernel module with modprobe v4l2loopback devices=4 and the xpra server will try all devices until it finds one it can use (not fully tested)
• r12825 makes sure we filter out video devices that cannot do capture

The webcam tool now shows a lot more device information.

Maybe too late for us, but I've just found a python ctypes interface for v4l2: ​https://pypi.python.org/pypi/v4l2

As for ​v4l2loopback, packaging it for dkms is too hard (dkms always seem to cause more problems that it solves for me too). Why isn't this merged upstream? Can we help push it? The code is stable and does not change much.

Remaining issues:

• framerate and hardcoded csc dependency
• win32 and osx support:
  • enumerate devices on win32: ​How to get a list of video capture devices (web cameras) on windows? (C++), ​win32: Selecting a Capture Device, ​DirectShow, ​Microsoft Media Foundation, ​Audio/Video Capture, ​ctypes enumeration code, using WMI? ​Win32_USBHub
  • enumerate devices on osx: ​How to get a list of video capture devices (web cameras) on Mac OS? (C++)
  • shadow servers (low priority)

---

This is what the systray now looks like:

[Antoine Martin]

• r12827 detects video capture devices on win32 using ​comtypes. (maybe we could capture to a PIL image more or less directly, and drop the dependency on opencv: ​http://pydoc.net/Python/jaraco.video/1.2/jaraco.video.capture/ using ​ISampleGrabber)
• r12828 detects device change events using ​WM_DEVICECHANGE (not specific to video devices... but seems to work nonetheless)

More links on win32 device events API:

• ​Device Events
• ​Registering for Device Notification
• ​USB Insertion Extrinsic Event exists?

OSX:

• ​Introduction to USB Device Interface Guide
• ​IOKit device adding/removal notifications - only fire once?
• ​IOKit.py

As for framerate, without going as far as refactoring the window-source code so we can use it for sending webcam frames, maybe we can send frames based on the line latency. (ie: 200ms round trip -> allow 4 frames in flight for a 20fps target)

[Antoine Martin]

new systray menu shows my webcam as option

[Antoine Martin]

try to get notifications when USB devices are added or removed

[Antoine Martin]

The patch above attempts to use ​IOKit.py to get device notifications for OSX, but it has a number of problems:

• can't re-use the same main loop as GTK, so it uses its own thread and main loop - which may be the cause of other problems..
• not very stable
• had to modify IOKit.py
• can't get the device path (that part crashes if you try to use it)
• webcams don't work on OSX anyway since we cannot build opencv..

Similar / related code:

• ​USBNotifier.m
• ​How can I find Device File Name of an USB?
• ​Can't find bsd name of usb device - bug in 10.11.x?
• ​iodisplayregistry.c traverses and dumps all services, also: ​GPUModelVRAMInfo.c
• ​IOKitLib header reference

Webcam API links:

• ​Still and Video Media Capture
• ​Image Capture Applications Programming Guide
• ​capturing image from webcam
• ​python Barcode Scanner (and update code for new pyobjc..)
• ​ffmpeg webcam - probably easiest to implement, 10.7 onwards...

[Antoine Martin]

r12834 solves the framerate issue over slower links by calculating how many frames-in-flight we should have and try to keep to that number.
It adds XPRA_WEBCAM_FPS means we try to send that many frames per second (ie: the default value of 20 will send a frame every 50 milliseconds), unless the ack packets start coming in late.
Debug output:

may_send_webcam_frame() latency=307, not acked=6, target=6

Not dealing with shadow servers, because I don't see how this would work.

This will do for this release, will follow up in #1231.

[Antoine Martin]

Milestone renamed

[Antoine Martin]

• r13832: improved webcam timers management, etc (see commit message)
• see also #1268

[Antoine Martin]

Not heard back, closing.

[Antoine Martin]

For win32 and macos see #1231

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/1113