uinput virtual pointer device

[Antoine Martin]

split from #173 with a more limited scope: just implement the pointer events using uinput instead of xtest.

This can help us with:

• getting away from xtest which is too limited
• workaround server grabs: #1565 (though it may be difficult to move everything out of the UI thread)
• better support for fine-grained scrolling when this gets added to the kernel and userspace: ​input: add an EV_REL event for high-res vertical wheel - discussion: ​Re: Support for Logitech MX Anywhere 2
• later on: emulate touch devices, etc

[Antoine Martin]

Based on ​Bug 92772 - RFE: extra option to define the vdist/hdist for scrolling event, we might be able to generate "smooth-scrolling" / small scrolling deltas by creating our uinput device with a very small MOUSE_WHEEL_CLICK_ANGLE (or better, use a high MOUSE_WHEEL_CLICK_COUNT) then synthesizing events using the correct multiplier.

This should be doable using ​libinput: Static device configuration via udev.
Similar to ​Change scroll speed with libinput.

This looks also interesting for touch devices: ​Create a virtual multitouch device using uinput driver.

Problem is that no matter what I specify for the uinput device or udev rules, the properties are not set. (and not even the built-in rules are? system bug?)

[Antoine Martin]

Code added in r16572.

Still TODO:

• modprobe uinput to ensure we can use it?
• bug: server shutdown no longer cleans up the sockets?
• when we upgrade an existing server... we don't have the device details any more - should we save those somewhere?
• we should verify that the uinput device works before using it? (send motion, wait a little, verify that the core pointer has moved)
• the udev race is annoying and any call to udevadm trigger can reset the device permissions to something that prevents the Xorg process from accessing our uinput device.. we cannot write a udev rule because we don't know the uid in advance, so how can we tell udev to just leave it alone? Do it the hard way and just mknod + chown a private one?
• scroll speed as per comment:1

[Antoine Martin]

udev meanderings, trying to use hwdb to force the click angle / count:

cat > /lib/udev/hwdb.d/71-mouse-xpra.hwdb <<EOF
# allow xpra to use smooth scrolling
mouse:usb:*:name:Xpra Virtual Pointer:
 MOUSE_WHEEL_CLICK_ANGLE=1
 MOUSE_WHEEL_CLICK_COUNT=360

mouse:*:name:Xpra Virtual Pointer:
 MOUSE_WHEEL_CLICK_ANGLE=1
 MOUSE_WHEEL_CLICK_COUNT=360

mouse:usb:v0088p0088:*
 MOUSE_WHEEL_CLICK_ANGLE=1
 MOUSE_WHEEL_CLICK_COUNT=360
EOF
udevadm hwdb --update

Then starting a server:

sudo xpra start-desktop --uid=1000 --gid=1000 :100 --no-daemon \
    --start=xterm --bind-tcp=0.0.0.0:10000 -d mouse --input-devices=uinput
(..)
pointer device emulation using UInput device /dev/input/event20
(..)

Shows no new udev attributes:

$ sudo udevadm info -a --name /dev/input/event20
  looking at device '/devices/virtual/input/input44/event20':
    KERNEL=="event20"
    SUBSYSTEM=="input"
    DRIVER==""

  looking at parent device '/devices/virtual/input/input44':
    KERNELS=="input44"
    SUBSYSTEMS=="input"
    DRIVERS==""
    ATTRS{name}=="Xpra Virtual Pointer"
    ATTRS{phys}==""
    ATTRS{properties}=="0"
    ATTRS{uniq}==""

Trying with a plain udev rule:

cat > /lib/udev/rules.d/71-xpra-mouse.rules << EOF
ACTION=="add|change", SUBSYSTEM=="input", ATTRS{name}=="Xpra Virtual Pointer", ENV{MOUSE_WHEEL_CLICK_ANGLE}="1"
EOF
udevadm control --reload-rules && udevadm trigger

Same result.

$ libinput list-devices
(...)
Device:           Xpra Virtual Pointer
Kernel:           /dev/input/event20
Group:            10
Seat:             seat0, default
Capabilities:     pointer 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      disabled
Nat.scrolling:    disabled
Middle emulation: disabled
Calibration:      n/a
Scroll methods:   button
Click methods:    none
Disable-w-typing: n/a
Accel profiles:   flat *adaptive
Rotation:         n/a

Did the flat acceleration option not work? (could cause problems when we use relative motion to go where we want)

Or maybe it does do something (the plain udev rule), but not on the correct device? (/devices/virtual/input/input46 but not on /devices/virtual/input/input46/event20)

$ udevadm monitor -ukp

KERNEL[21719.101574] add      /devices/virtual/input/input46 (input)
ACTION=add
DEVPATH=/devices/virtual/input/input46
EV=7
KEY=7f0000 0 0 0 0
MODALIAS=input:b0003v0000p0000e0000-e0,1,2,k110,111,112,113,114,115,116,r0,1,8,amlsfw
NAME="Xpra Virtual Pointer"
PRODUCT=3/0/0/0
PROP=0
REL=103
SEQNUM=11953
SUBSYSTEM=input

KERNEL[21719.101695] add      /devices/virtual/input/input46/mouse2 (input)
ACTION=add
DEVNAME=/dev/input/mouse2
DEVPATH=/devices/virtual/input/input46/mouse2
MAJOR=13
MINOR=34
SEQNUM=11954
SUBSYSTEM=input

KERNEL[21719.101768] add      /devices/virtual/input/input46/event20 (input)
ACTION=add
DEVNAME=/dev/input/event20
DEVPATH=/devices/virtual/input/input46/event20
MAJOR=13
MINOR=84
SEQNUM=11955
SUBSYSTEM=input

UDEV  [21719.102443] add      /devices/virtual/input/input46 (input)
.INPUT_CLASS=mouse
ACTION=add
DEVPATH=/devices/virtual/input/input46
EV=7
ID_INPUT=1
ID_INPUT_MOUSE=1
ID_SERIAL=noserial
KEY=7f0000 0 0 0 0
MODALIAS=input:b0003v0000p0000e0000-e0,1,2,k110,111,112,113,114,115,116,r0,1,8,amlsfw
MOUSE_WHEEL_CLICK_ANGLE=1
NAME="Xpra Virtual Pointer"
PRODUCT=3/0/0/0
PROP=0
REL=103
SEQNUM=11953
SUBSYSTEM=input
TAGS=:seat:
USEC_INITIALIZED=21719102303

UDEV  [21719.103088] add      /devices/virtual/input/input46/mouse2 (input)
.INPUT_CLASS=mouse
ACTION=add
DEVNAME=/dev/input/mouse2
DEVPATH=/devices/virtual/input/input46/mouse2
ID_INPUT=1
ID_INPUT_MOUSE=1
ID_SERIAL=noserial
MAJOR=13
MINOR=34
SEQNUM=11954
SUBSYSTEM=input
USEC_INITIALIZED=21719102990

UDEV  [21719.129999] add      /devices/virtual/input/input46/event20 (input)
.INPUT_CLASS=mouse
ACTION=add
DEVNAME=/dev/input/event20
DEVPATH=/devices/virtual/input/input46/event20
ID_INPUT=1
ID_INPUT_MOUSE=1
ID_SERIAL=noserial
MAJOR=13
MINOR=84
SEQNUM=11955
SUBSYSTEM=input
USEC_INITIALIZED=21719129875

[Antoine Martin]

Actually, the udev variant does do something - the scrolling is slower than usual with these values:

ACTION=="add|change", ATTRS{name}=="Xpra Virtual Pointer", \
ENV{MOUSE_WHEEL_CLICK_ANGLE}="1", ENV{MOUSE_WHEEL_CLICK_COUNT}="360"

Success!

[Antoine Martin]

New issue: Xorg version 1.17.x as found in centos7 doesn't support the xorg.conf options we need (caught by the rpmbuild unit tests):

Parse error on line 17 of section InputClass in file /usr/src/rpmbuild/BUILD/xpra-2.2-python2/unittests/../etc/xpra/xorg.conf
        "NoMatchProduct" is not a valid keyword in this section.

Does it support MatchProduct? Can we get away with not specifying NoMatchProduct and have Ignore=True for all devices, then override for the devices we do want to use?

If not, we'll need a different xorg.conf for centos7, and disable uinput. (we could also rely on the selftests in case someone enables uinput on centos7)
Also looks like centos 7.4 will support this: ​https://bugzilla.redhat.com/show_bug.cgi?id=1401641 (updated to xorg 1.19.x)

[Antoine Martin]

Updates:

• r16648 + r16649: install uinput xorg config, and switch to this file if we find --input-devices=uinput
• r16650: verify uinput device is being received by the X11 server, fallback to XTest if it isn't
• r16651: make sure we match the device path exactly (oops!)
• r16652: ship udev rule, XPRA_MOUSE_WHEEL_CLICK_MULTIPLIER=30 env var, logging

With all this in place, we have the ability to send fine grained scrolling events. This is what libinput sees (event3 is the real pointer, event19 our uinput virtual one - which is a high precision one and uses higher values):

-event3   POINTER_MOTION   +14.46s	  1.08/  0.00
 event3   POINTER_MOTION   +14.46s	  0.00/  1.08
-event19  POINTER_MOTION   +14.46s	  0.00/  0.82
-event3   POINTER_MOTION   +14.46s	  0.00/  1.07
-event19  POINTER_MOTION   +14.46s	  0.83/  0.83
 event19  POINTER_MOTION   +14.47s	  0.00/  0.85
-event3   POINTER_MOTION   +14.47s	  1.04/  0.00
-event19  POINTER_MOTION   +14.47s	  0.86/  0.86
 event19  POINTER_MOTION   +14.47s	  0.00/  0.87
 event19  POINTER_MOTION   +14.48s	  0.87/  0.00
-event3   POINTER_AXIS     +14.55s	vert -15.00* horiz 0.00 (wheel)
-event19  POINTER_AXIS     +14.55s	vert -450.00* horiz 0.00 (wheel)
-event3   POINTER_AXIS     +14.57s	vert -15.00* horiz 0.00 (wheel)
-event19  POINTER_AXIS     +14.58s	vert -450.00* horiz 0.00 (wheel)
-event3   POINTER_AXIS     +14.60s	vert -15.00* horiz 0.00 (wheel)
-event19  POINTER_AXIS     +14.61s	vert -450.00* horiz 0.00 (wheel)
-event3   POINTER_AXIS     +14.71s	vert 15.00* horiz 0.00 (wheel)
-event19  POINTER_AXIS     +14.71s	vert 450.00* horiz 0.00 (wheel)
-event3   POINTER_AXIS     +14.73s	vert 15.00* horiz 0.00 (wheel)
-event19  POINTER_AXIS     +14.73s	vert 450.00* horiz 0.00 (wheel)
-event3   POINTER_AXIS     +14.75s	vert 15.00* horiz 0.00 (wheel)
-event19  POINTER_AXIS     +14.75s	vert 450.00* horiz 0.00 (wheel)
-event3   POINTER_MOTION   +14.75s	  0.00/  0.96
-event19  POINTER_MOTION   +14.75s	  0.00/  0.73
-event3   POINTER_MOTION   +14.78s	  0.00/  0.71
-event19  POINTER_MOTION   +14.78s	  0.00/  0.60

[Antoine Martin]

Client side support added in r16654 which also cleans up the code nicely (+ minor fixups in r16655).
This can be tested without a precise wheel or tablet by using environment variables to adjust the deltas we send to the server, then using an application like Firefox which should be listening for the precise scroll events rather than the old chunky ones. (tools like xev do not, but libinput or xinput can show the precise wheel deltas)

Start the server as root with (your actual uid/gid may vary):

modprobe uinput
xpra start --uid=1000 --gid=1000 --start=firefox --input-devices=uinput -d mouse

And verify that the server does use the uinput virtual device, it should print something like this:

pointer device emulation using UInput device /dev/input/eventNNN

• testing with a linux client (maybe we should make input-devices=xi the default now):

  XPRA_XINPUT_WHEEL_DIV=15 xpra attach --input-devices=xi -d mouse
  

Values greater than 15 will scroll less, lower values will scroll more.

• testing with a MS Windows client (see ​WM_MOUSEWHEEL for the value of 120):

  set XPRA_WIN32_WHEEL_DELTA=120
  xpra_cmd attach ...
  

• with macos (see ticket:1157#comment:24):

  XPRA_OSX_WHEEL_MULTIPLIER=100 XPRA_OSX_WHEEL_PRECISE_MULTIPLIER=1 xpra attach ...
  

Since those are multipliers, increase the value(s) to scroll more.
(those multiplier values may not be correct.. as they're just guesses since the apple documentation is as bad as their API - TBC with hardware testing in #1157)

Some useful debugging tools:

• udevadm monitor -ukp (as root): shows devices being created when starting the server, with their settings
• libinput list-devices and libinput debug-events
• xinput test-xi2 DEVICENO xinput2 events for a particular device

---

@maxmylyn: with the default scroll values things should work exactly as before, input devices with non-standard scroll values should now be handled better.

Will follow up in #1615 for touch devices (most useful for supporting tablets and phones).

[J. Max Mena]

Trying to start my trunk Fedora 25 server as root(using the command in comment:7 and changing it to the correct uid and gid values) it errors out with the following:

Failed to create bus connection: Operation not permitted

I'm sure it's something simple, but I'm stuck at this. Earlier today it was erroring out with a much longer traceback, but I can't seem to get that far anymore.

[Antoine Martin]

Failed to create bus connection: Operation not permitted

Try with sudo xpra ... or add --systemd-run=no

[Antoine Martin]

r16661 recommends python-uinput in the RPM spec file, but there is a problem with that: it seems that the Fedora RPM package is fundamentally broken at the moment: ​python-uinput-0.11.2 is available.

[J. Max Mena]

Okay that makes sense why I was seeing tracebacks. I'm not at home today, so I'll try that tomorrow morning.

[J. Max Mena]

Alright I got my server to finally work nicely with starting as root. Is there a workaround for the broken uinput module?

I'll try downloading the tarball from the upstream source and building it, maybe that'll do it.

[Antoine Martin]

I'll try downloading the tarball from the upstream source and building it, maybe that'll do it.

You can do that, but this is easier:

easy_install -U -Z python-uinput

To make it even easier, I've added python2-uinput rpm to the stable Fedora and centos repositories: r16678.
So dnf upgrade --refresh should be all you need now.

[J. Max Mena]

Okay, I finally got the time to actually test this very thoroughly. And, it works fantastically. One thing I've noted is that you have to change the XPRA_XINPUT_WHEEL_DIV=15 flag by large numbers before you see a notable affect on scroll speed. Setting it to 60 doesn't seem to notably affect scrolling (except it takes an extra wheel "click" to get it to start scrolling), but setting it to something extreme like 500 makes scrolling painfully slow (intended).

It also doesn't appear to have broken anything in OSX, surprisingly (r16657 client). So that's a win. However, changing the XPRA_OSX_WHEEL_MULTIPLIER=100 doesn't appear to have any effect at all. But, leaving it at the default works fine.

The Windows client also doesn't seem to have broken anything. However I'm noting that changing the XPRA_WIN32_WHEEL_DELTA doesn't appear to have any effect at all. BUT, leaving it at the default works just as fine as before.

---

Tempted to close this and open a new ticket to follow up the OSX and Win32 variables not having any effect. Unless that's intended? (probably not but I feel like asking just in case)

[J. Max Mena]

Okay I should have double checked the buttons were working earlier, but I was focused on scrolling.

Right now right and middle click are reversed in all the clients.

[Antoine Martin]

Changing the env settings on macos and win32 should have a noticeable effect:

• MS Windows: the value for XPRA_WIN32_WHEEL_DELTA should be printed as WHEEL_DELTA with "-d win32" and also with NativeGUI_info.exe -v. It will be shown on mouse wheel events with "-d mouse":

  win32 mousewheel: orientation=%s, distance=%i, wheel-delta=%s, ...
  

• macos: the multiplier (either XPRA_OSX_WHEEL_MULTIPLIER or XPRA_OSX_WHEEL_PRECISE_MULTIPLIER for precise events) will be shown with "-d mouse" as:

  normalize_precision(%.3f)=%.3f (multiplier=%i)
  

XPRA_OSX_WHEEL_MULTIPLIER=100 is the default value, so that's not expected to change anything, try 1000 or 10 instead.

As for the middle and right clicks being reversed: try r16685 (tiny server side fix).

[J. Max Mena]

Okay the reversed mouse buttons was fixed by that update. That being said, I'll follow up on the OSX/Win32 things after lunch.

[J. Max Mena]

So "after lunch" turned into two weeks later. Long story short I was reassigned to other things around lunch time, and then the next two weeks I was given a list of other things to do, that was a higher priority than this.

---

Anyways, back to this ticket.

I updated my server to r16705, but left the client versions the same.

I double checked the Windows logs and the XPRA_WIN32_WHEEL_DELTA is indeed getting changed. Same thing for OSX, the normalize_precision prints are off by about a factor of 100 from each other when setting XPRA_OSX_WHEEL_MULTIPLIER to 10 and 1000.

As far as I can tell, that was the only outstanding thing left in this ticket; everything else seems to be working fine with uinput (been using it for the last few hours and everything plays nicely for now, and all buttons seem to work properly on my Logitech Performance MX mouse). Do you want to go ahead and close it, or is there still any more work to be done?

I do need to get ahold of a special Apple mouse (that weird Magic Mouse) to double check that, but there's no reason we can't follow up with that in a new ticket if something is broken. However, I did test with my Apple Laptop's (A 2014ish Macbook Pro something or other) trackpad and it behaved just fine.

[Antoine Martin]

Will follow up in #1615 and #1631

[Antoine Martin]

Minor fixes in r17315, r17110

[Antoine Martin]

Minor fix in r17631: check uinput device permissions early, so we don't use the xorg-uinput config unless uinput is actually likely to be used.

[Antoine Martin]

As of r18445 the custom udev script is now called "xpra_udev_product_version".

[J. Max Mena]

This revision (r18445) caused a build to fail on both my Fedora machines with the following error:

error: file '/root/trunk/src/scripts/xpra_udev_product_version' does not exist

r18444 builds fine

[Antoine Martin]

This revision (r18445) caused a build to fail ...

Try r18446.

[J. Max Mena]

r18446 and subsequent builds no longer fail - closing again (must have forgot to hit submit on Friday)

[Antoine Martin]

Important fixes in r18916, see ticket:1615#comment:2.

For html5 client support, see #1797.
Follow up: #1615 touch device support.

[Antoine Martin]

See also ​Reporting high-resolution scroll events - almost exactly the same findings, with one major difference: apparently SDL has problems so they ("Chromium OS Touch / Input Team") are proposing REL_WHEEL_HI_RES and REL_HWHEEL_HI_RES to report high resolution wheel events. We can do that.

[Antoine Martin]

Update: newer kernels may include this updated patch: ​MS and Logitech high-resolution scroll wheel support

[Antoine Martin]

New issues:

• wayland does not ignore our device, we need the equivallent of this file: browser/xpra/trunk/src/etc/X11/xorg.conf.d/90-xpra-virtual.conf
• r21192 fix for wheel-motion packets with precise-wheel (uinput) mode
• r21194 + r21196: fix for uinput horizontal scrolling
• something broke and pointer location is out of sync, the delta between where we want to move to and where we are does not decrease (should be near 0):

  UInputPointerDevice.move_pointer(-1, 442, 256)
  X11Keyboard.query_pointer=464, 546
  delta(464, 546)=-22, -290
  

• with firefox, the scrolling is not very smooth..

[Antoine Martin]

Re REL_WHEEL_HI_RES, see also: ​High resolution wheel scrolling on Linux v4.21