xpra icon
Bug tracker and wiki

Opened 2 years ago

Closed 19 months ago

Last modified 18 months ago

#1030 closed enhancement (fixed)

webcam and scanner forwarding

Reported by: Antoine Martin Owned by: Antoine Martin
Priority: major Milestone: 0.17
Component: core Version: trunk
Keywords: Cc:

Description (last modified by Antoine Martin)

So that applications that want to access the webcam can be made to work (ie: skype, etc). Scanners use the same framework as webcams, so maybe we can handle those too? (means potentially more than one device...)

We will need a simplified "window video source" class that the client can use to send pixel data with the same adaptability as the server (but maybe simplified a bit when it comes to batch delay, speed and quality tuning). It would be good if we could know when frames are being consumed from the server device so that we don't bother updating it when no-one is watching.
It will only deal with full frames, and only using a video encoder (if one is available) and no mmap.
It should have the same controls as sound: on, off or disabled.

The server should only need to pass the frames to a virtual video device for this to work:

The initial version should not worry about:

  • switching resolutions and formats: just one format will do
  • controls such as showing the whole desktop instead of the video input
  • sound sync!

etc..

Attachments (1)

webcam-forwarding.patch (35.4 KB) - added by Antoine Martin 21 months ago.
almost working patch using opencv, png encoding only for now

Download all attachments as: .zip

Change History (16)

comment:1 Changed 2 years ago by Antoine Martin

Description: modified (diff)
Status: newassigned
Summary: webcam forwardingwebcam and scanner forwarding

Changed 21 months ago by Antoine Martin

Attachment: webcam-forwarding.patch added

almost working patch using opencv, png encoding only for now

comment:2 Changed 21 months ago by Antoine Martin

Proof of concept code merged in r11808 with some severe limitations:

  • we feed the wrong image format so the picture looks messed up
  • png and jpeg only via PIL
  • framerate is rate limited by packet roundtrip speed, and it may still cause bandwidth issues...
  • should be able to specify the virtual device to use on the server using the command line option
  • we should have a tray menu to enable/disable the feature and allow the user to select the device
  • detect hardware changes (both on client and server): hotplug events

Build updates.

These things will need to be added to wiki/Dependencies.
We don't actually use or need some of those, but installing them anyway is easier than excising them out of matplotlib / opencv.

Almost gets you there, except it fails on directx.. (why on earth is it even trying to build directx on osx??)

  • win32:
    • C:\Python27\Scripts\pip.exe install E:\matplotlib-1.5.1-cp27-none-win32.whl
    • C:\Python27\Scripts\pip.exe install E:\opencv-python-3.1.0-cp27-none-win32.whl

Still TODO:

  • new wiki page
  • documenting the installation of the virtual video device (rpmfusion has it, debian has it)
  • documenting the configuration of video devices (groups and permissions)
  • detecting device changes: dbus hotplug events for linux?
  • packaging: add recommends opencv for Debian (no depends for rpms?)
Last edited 21 months ago by Antoine Martin (previous) (diff)

comment:3 Changed 21 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas
Status: assignednew

Added a wiki page with details: wiki/Webcam.
Will follow up in #1113.
This is good enough as a proof-of-concept only, it will need more work - but not for this release.

@afarr: ready for an initial round of testing. Please re-assign to me for bugs or to smo so that he can deal with the required win32 / osx / Fedora build updates.

comment:4 Changed 21 months ago by J. Max Mena

  • Built and installed v4l2loopback
  • Upped client and server to r11861

Attempting --webcam=yes outputs the following error:

2016-02-05 13:40:09,266 Warning: virtual video directory /sys/devices/virtual/video4linux not found
2016-02-05 13:40:09,266  webcam forwarding disabled

And, obviously, the webcam forwarding does not work. Am I missing something?

I'll keep looking at it to see if I need to do anything else, but I'm pretty sure all the bits are there and ready.

comment:5 Changed 21 months ago by J. Max Mena

Update:

  • one must first load the device modprobe v4l2loopback

No more error.

However whatever I try, I cannot get it to detect a webcam on the client side and successfully forward it (even using a Win8.1 r11850 client).

comment:6 Changed 21 months ago by J. Max Mena

Oh, also some logs:

Client:

C:\Program Files (x86)\Xpra>Xpra_cmd.exe attach tcp:10.0.32.189:2200 --webcam=ye
s -d webcam
2016-02-05 14:19:57,086 Xpra gtk2 client version 0.17.0-r11850
2016-02-05 14:19:57,086  running on Microsoft Windows 8.1
2016-02-05 14:19:57,086 webcam forwarding: yes
2016-02-05 14:19:57,275 GStreamer version 1.4 for Python 3.4
2016-02-05 14:19:57,634 OpenGL_accelerate module loaded
2016-02-05 14:19:57,711  detected keyboard: layout=us
2016-02-05 14:19:57,727  desktop size is 2560x1440 with 1 screen:
2016-02-05 14:19:57,727   Default (677x381 mm - DPI: 96x96) workarea: 2560x1400
2016-02-05 14:19:57,727     DISPLAY1 (597x336 mm - DPI: 108x108)
2016-02-05 14:19:57,727  upscaled by 125%, virtual screen size: 2048x1152
2016-02-05 14:19:57,727   Default (677x381 mm - DPI: 76x76) workarea: 2048x1120
2016-02-05 14:19:57,727     DISPLAY1 (597x336 mm - DPI: 87x87)
2016-02-05 14:19:57,775 Xpra X11 server version 0.17.0-r11382
2016-02-05 14:19:57,775  running on Linux Fedora 23 Twenty Three
2016-02-05 14:19:57,775 webcam server support: True (1 devices)
2016-02-05 14:19:57,789 Attached to tcp:10.0.32.189:2200 (press Control-C to det
ach)

2016-02-05 14:19:57,789 webcam_state() webcam forwarding=True, server virtual vi
deo devices=1
2016-02-05 14:20:00,556 sound output using audio codec: Vorbis
2016-02-05 14:20:14,983 setting scaling to 100%:
2016-02-05 14:20:14,983 sending updated screen size to server: 2560x1440 with 1
screens
2016-02-05 14:20:14,983   Default (677x381 mm - DPI: 96x96) workarea: 2560x1400
2016-02-05 14:20:14,983     DISPLAY1 (597x336 mm - DPI: 108x108)

Server:

2016-02-05 13:57:41,569 init_virtual_video_devices
2016-02-05 13:57:41,570 found 1 virtual video device

(the only webcam related print)

comment:7 Changed 21 months ago by Antoine Martin

Owner: changed from alas to J. Max Mena

one must first load the device modprobe v4l2loopback


Yes, it is on the wiki page wiki/Webcam.
r11862 tries to make it clearer in the server warning message.


C:\Program Files (x86)\Xpra>Xpra_cmd.exe attach tcp:10.0.32.189:2200 --webcam=yes -d webcam


As per the wiki page.. use the value "on" if you want it to start when you connect.
Unless you disable webcam completely (values "no", "false"), the webcam support will be available but needs to be started from the systray.


r11863 and r11864 improve logging, command line parsing and error handling.
New beta builds with those changes are available.

comment:8 Changed 21 months ago by J. Max Mena

Upped client to r11892 Win8.1 and Server to trunk r11892:

  • Managed to get a random menu asking which camera to use on the laptop (two webcams, crappy Apple internal and a Logitech camera via USB)
    • Not before realizing that I was running an old version(r11850)...
  • Now with r11892: Unable to get that menu again. Clicking the webcam button on the tray only prints the following on the client side:
    2016-02-09 09:15:54,519 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
    tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=True
    2016-02-09 09:15:54,519 do_start_sending_webcam(auto)
    2016-02-09 09:15:54,519 do_start_sending_webcam(auto) device=0
    2016-02-09 09:15:56,131 test capture using <VideoCapture 04597E60>: True, True
    2016-02-09 09:15:56,131 webcam started
    2016-02-09 09:15:56,137 webcam_changed(gtk2.client,) active=True, menu=True
    2016-02-09 09:15:56,137 stop_sending_webcam()
    2016-02-09 09:15:56,137 do_stop_sending_webcam() device=<VideoCapture 04597E60>
    2016-02-09 09:15:56,263 webcam_changed(gtk2.client,) active=False, menu=True
    2016-02-09 09:15:56,263 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
    tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=False
    2016-02-09 09:16:01,857 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
    tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=True
    2016-02-09 09:16:01,857 do_start_sending_webcam(auto)
    2016-02-09 09:16:01,857 do_start_sending_webcam(auto) device=0
    2016-02-09 09:16:19,670 test capture using <VideoCapture 04597FF0>: True, True
    2016-02-09 09:16:19,670 webcam started
    2016-02-09 09:16:19,671 check_acks: webcam_last_ack=-1
    2016-02-09 09:16:19,671 Warning: no acknowledgements received from the server, s
    topping webcam
    2016-02-09 09:16:19,671 stop_sending_webcam()
    2016-02-09 09:16:19,671 do_stop_sending_webcam() device=<VideoCapture 04597FF0>
    2016-02-09 09:16:19,750 webcam_changed(gtk2.client,) active=False, menu=True
    2016-02-09 09:16:19,750 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
    tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=False
    2016-02-09 09:16:19,750 webcam_changed(gtk2.client,) active=False, menu=False
    2016-02-09 09:16:29,683 check_acks: webcam_last_ack=-1
    2016-02-09 09:16:29,683 Warning: no acknowledgements received from the server, s
    topping webcam
    2016-02-09 09:16:29,683 stop_sending_webcam()
    2016-02-09 09:16:29,683 do_stop_sending_webcam() device=None
    2016-02-09 09:16:33,137 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
    tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=True
    2016-02-09 09:16:33,137 do_start_sending_webcam(auto)
    2016-02-09 09:16:33,137 do_start_sending_webcam(auto) device=0
    2016-02-09 09:16:34,720 test capture using <VideoCapture 04597EC0>: True, True
    2016-02-09 09:16:34,720 webcam started
    2016-02-09 09:16:34,721 webcam_changed(gtk2.client,) active=True, menu=True
    2016-02-09 09:16:34,726 stop_sending_webcam()
    2016-02-09 09:16:34,726 do_stop_sending_webcam() device=<VideoCapture 04597EC0>
    2016-02-09 09:16:34,898 webcam_changed(gtk2.client,) active=False, menu=True
    2016-02-09 09:16:34,898 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
    tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=False
    2016-02-09 09:16:44,723 check_acks: webcam_last_ack=-1
    2016-02-09 09:16:44,723 Warning: no acknowledgements received from the server, s
    topping webcam
    2016-02-09 09:16:44,723 stop_sending_webcam()
    2016-02-09 09:16:44,723 do_stop_sending_webcam() device=None
    

(Clicked it a few times hoping to maybe get that menu)

Will put server side prints in a second (have to restart session to get logs)

Last edited 21 months ago by Antoine Martin (previous) (diff)

comment:9 Changed 21 months ago by J. Max Mena

Server side -d webcam print:

2016-02-09 09:20:50,363 v4l2 using device /dev/video0
2016-02-09 09:20:50,363 Error setting up webcam forwarding:
2016-02-09 09:20:50,363  [Errno 13] Permission denied: '/dev/video0'
2016-02-09 09:20:50,386 stopping webcam device 0:
2016-02-09 09:20:50,386 Warning: cannot stop webcam device 0: no such context!

Looks like it can't access the camera? Is there a group in which I should be a member?

comment:10 Changed 21 months ago by J. Max Mena

Update:

  • added myself to the video and audio groups. (one must also log out and back in)
  • it now sends video frames to the server (also takes a massive performance hit)
  • However, applications on the system do not see the virtual camera.(Cheese, Google Chrome, will see about a few more)

comment:11 Changed 21 months ago by J. Max Mena

Owner: changed from J. Max Mena to Antoine Martin

Another update:

Turns out it just doesn't like cheese or Chrome. If you install camorama and launch it with camorama -d /dev/$cameradeviceid (usually is video0), then it works. (greyscale and chopped oddly, but 30fps)

Passing back to you, antoine.

comment:12 Changed 21 months ago by Antoine Martin

Status: newassigned

Specifying the device with camorama should not be necessary as long as the device is the first one (video0).

The fps will be lower, much lower, when running on a slower connection as we wait for the server ack before sending the next frame. ie: on a 100ms link, that's going to limit things to 10fps maximum.

As for compatibility with other applications like cheese and chrome, I will try to take a look but this will not be considered a blocker for the 0.17 release as this is mostly a feature preview at this point. This can be dealt with in the next release: #1113.

comment:13 Changed 20 months ago by Antoine Martin

Note we will need a newer gcc to build webcam support on OSX because of this compiler crash: Internal compiler error in directx.cpp on Mac OS X 10.6.
Getting around this by hacking the source files only leads to more gcc pain down the road. (tbb and gcd don't build, etc)

comment:14 Changed 19 months ago by Antoine Martin

Resolution: fixed
Status: assignedclosed

I think this will do for this release.

comment:15 Changed 18 months ago by Antoine Martin

Milestone: 1.00.17
Note: See TracTickets for help on using tickets.