Xpra: Ticket #1060: allow client applications to specify the video region

Follow up from #410: some applications may know where the region of interest is, and they should be able to pass on that information to us. It should also be possible to turn on or off the video region code.



Sun, 20 Dec 2015 16:07:01 GMT - Antoine Martin: owner changed

Implemented in r11448 (trunk). It is a bit late for merging features, but if needed, we could still apply this to the 0.16.x branch.

This adds 3 public methods both via xpra control and the dbus interface (#904).

Here are a few examples covering each one using the control interface:

The control interface is great for testing, but for an application, the dbus interface may be more practical.

@afarr: re-assigning to you so you can bug Idris to claim this ticket.


Tue, 09 Aug 2016 15:07:30 GMT - Antoine Martin: priority changed

8 months... does this work or not?


Tue, 16 Aug 2016 23:05:12 GMT - alas: status changed; resolution set

Uhhh... sorry. Yes this does work, using dbus interface.

Hopefully, one day, the details will be worked out to get it to me in a form I can play with... but idris has, erhhm, left the building - so not sure when that's liable to happen.

In the meantime, I'll close this and find something to try to work on in a timely fashion.


Tue, 30 Aug 2016 11:21:04 GMT - Antoine Martin: status changed; resolution deleted

Really?

r13491 is a fix for this API (backported to v0.17.x in r13509), either you weren't using it, or you weren't seeing the bug in the server log, or the fix is going to break it???


Fri, 02 Sep 2016 02:18:59 GMT - alas:

So... it looks like I was rather incorrect. We were, in fact, not using it.

It turns out that, rather than using the dbus channel (or control channel, obviously) to handle video-region enabling/detection... we had done a spot of customization in the video_subregion code, setting self.detection=false ... and we were then detecting the video regions with our application, and using dbus to communicate that region to the server. (I'm not really sure what that might've been doing to the behavior when our application wasn't detecting and communicating/specifying a video region... which, I suppose, might be the reason we had to re-open #967?, but I won't say that out loud...)

Since this was allowing our application to specify a video region, and since it seemed to be working... I guess I was under the impression that this was working - and I am now under the impression that we'd just taken a short cut to enable our application to set the video region. (The short cut seems to work though, for whatever that's worth.)

Doing a spot of testing with 1.0 r13520 both on win32 client and fedora 23 server (to help clarify what's going on for my own edification, if nothing else), it looks like the control channel commands you mentioned above are behaving oddly.

Repeating with -d regiondetect enabled on the server, with the same video region specified, after scrolling down I saw the server pick what seemed to be an even more arbitrary rectangle to designate as video.

2016-09-01 18:04:14,419 scoreinout(rectangle[35, 435, 194, 132], 1466834, 21874694) inregion=0.063, inwindow=0.022, ratio=2.846, sizeboost=1.022
2016-09-01 18:04:14,419 testing      current video region       rectangle[35, 435, 194, 132]:   6% in,  93% out,   2% of window, score=296
2016-09-01 18:04:14,420 keeping existing video region rectangle[35, 435, 194, 132] with score 296

Unfortunately - I still don't have a system with dbus hooks set up to pass the video regions detected by our application to servers that I can build/update/etc... so I'm not sure about testing these controls against an application that can actually specify video regions.

I can do some more in-depth testing of this in the meantime (or at least try) if there are any more specific behaviors we should be looking for when testing with the control channel? (Though, I suppose this should maybe be pushed off until I'm able to get something together to better test an application more dynamically enabling and disabling video region detection v. specification?)


Fri, 02 Sep 2016 15:39:02 GMT - Antoine Martin:

1) region detection bugs do not belong here, that's what #410 or even #967 are for, but I'll take a look at the heuristics to ensure we don't favour regions that are too small to matter. 2) the fact that you're not actually using the API in this ticket, but hacking the code... is just shocking, a complete and utter waste of my time (9 months) - the process is not working, that needs fixing

So, I'm going to work on other things until you've had a chance to close some more tickets for this release, and just hope that those actually get tested properly.


Back to the subject of this ticket:


Sat, 10 Sep 2016 01:40:09 GMT - alas: owner, status changed

Ok, spent a long while testing with win32 1.0 r13637 client against 1.0 r13641 fedora 23 server (making sure not to open Session Info, #1301).

Trying to set with control channel seemed to fail.

[jimador@jimador ~]$ xpra control :13 video-region-enabled 2 false
video region disabled for window 2
[jimador@jimador ~]$ xpra info :13 | grep video_subregion.enabled
window.1.video_subregion.enabled=True
window.2.video_subregion.enabled=True
window.15.video_subregion.enabled=True
window.27.video_subregion.enabled=True
window.29.video_subregion.enabled=True

Also see no indications of success with opengl paint boxes.

Trying to re-enable seemed to produce odd and inconsistent results, with paint boxes indicating that attempts to set video regions after video-region-enabled was set to false seeming to indicate failure, but xpra info | grep video_subregion seeming to indicate that video regions were successfully set. Usually.

Meanwhile, I had no luck whatsoever passing a boolean argument via d-feet. Trying to use the SetVideoDetection? with parameters of 2, 1 got me a result of:

'g-io-error-quark: GDBus.Error:org.freedesktop.DBus.Python.AssertionError: Traceback (most recent call last):\n  File "/usr/lib64/python2.7/site-packages/dbus/service.py", line 707, in _message_cb\n    retval = candidate_method(self, *args, **keywords)\n  File "/usr/lib64/python2.7/site-packages/xpra/server/dbus/dbus_server.py", line 185, in SetVideoRegionDetection\n    self.server.control_command_video_region_detection(wid, detection)\n  File "/usr/lib64/python2.7/site-packages/xpra/server/server_base.py", line 1688, in control_command_video_region_detection\n    assert type(wid)==int, "window id is not an int"\nAssertionError: window id is not an int\n (36)'

Other parameter formats that I could think of (or find or guess at after searches about python dbus-interface syntax) generally warned me that strings weren't viable parameters... ("name 'true' is not defined".

So I had no luck with d-feet with this one (and calvinko also apparently had no luck with this with the dbus-interface, while not using d-feet as a GUI).

Since this one also involves a boolean, had no luck testing with d-feet/d-bus.

Setting with the control channel seems to work, according to info:

[jimador@jimador ~]$ xpra control :13 video-region-detection 2 false
video region detection disabled for window 2
[jimador@jimador ~]$ xpra info :13 | grep detection
window.1.video_subregion.detection=True
window.2.video_subregion.detection=False
window.15.video_subregion.detection=True
window.27.video_subregion.detection=True
window.29.video_subregion.detection=True

Opengl paint boxes still seem to indicate the video region, as does xpra info:

window.2.video_subregion.rectangle=(391, 749, 460, 103)
window.2.video_subregion.refresh_region[0]=(391, 749, 460, 102)
window.2.video_subregion.score=136
window.2.video_subregion.scores=0
window.2.video_subregion.scores.rectangle(391, 749, 460, 103)=136

But after scrolling until the video region is in a different region of the page, the the xpra info don't seem to recognize a difference (expected?):

window.2.video_subregion.rectangle=(391, 749, 460, 103)
window.2.video_subregion.score=136
window.2.video_subregion.scores=0
window.2.video_subregion.scores.rectangle(391, 749, 460, 103)=136

... and those values don't change regardless of how much time the page has to "sort itself out". But the paint boxes seem to indicate the regions are being encoded in a mix of rgb (orange) and jpeg (purple)... which seems expected with this set of parameters, but I figured it should be mentioned.

Re-enabling the video-region-detection...

[jimador@jimador ~]$ xpra control :13 video-region-detection 2 true
video region detection enabled for window 2

.. seems to work as hoped, with the new region being caught relatively quickly ( a second and change?).

window.2.video_subregion.rectangle=(391, 104, 460, 259)
window.2.video_subregion.refresh_region[0]=(391, 104, 460, 258)
window.2.video_subregion.score=175
window.2.video_subregion.scores=0
window.2.video_subregion.scores.rectangle(391, 104, 460, 259)=175
window.2.video_subregion.scores.rectangle(391, 749, 460, 103)=0

... and then, scrolling back, nearly, to the previous video location.

window.2.video_subregion.rectangle=(391, 715, 460, 137)
window.2.video_subregion.refresh_region[0]=(391, 715, 460, 136)
window.2.video_subregion.score=152
window.2.video_subregion.scores=0
window.2.video_subregion.scores.rectangle(391, 715, 460, 137)=152

Likewise, setting (re-setting) video-region-detection to false, then setting a video region, all via the control channel, seems to work as hoped.

[jimador@jimador ~]$ xpra control :13 video-region-detection 2 false
video region detection disabled for window 2
[jimador@jimador ~]$ xpra control :13 video-region 2 191 749 460 111
video region set to (191, 749, 460, 111) for window 2

and an xpra info | grep video_subregion:

window.2.video_subregion.rectangle=(191, 749, 460, 111)
window.2.video_subregion.score=152
window.2.video_subregion.scores=0
window.2.video_subregion.scores.rectangle(391, 715, 460, 137)=152

... though, there's no indication via paintpoxes of an h264 region in this case (though xpra info | grep detection shows wid 2 with video_subregion.detection=false in this case, as hoped). This might've been a result of google-chrome chewing up all the memory of the vm I was using for testing though... I can re-test this if needed.

Again, couldn't successfully pass a boolean parameter via d-feet to test the d-bus interface.

Seemed to work as expected with both d-feet/d-bus & xpra control channel. I can throw out a lot of logs of commands passed and info queries, but you'll be sick of those details by now if they're just gonna show things working.

Only detail I'd point out from this part of the testing is that the video regions set via this api call seem to be lost/discarded after any new heuristics calculations are made... so a video region thusly declared will be discarded and, apparently, re-calculated according to video region detection heuristics, whenever a mouse is rolled over anything. Does this mean that applications that mean to use the api to set the video region will need to continuously update the region specified in order to cope with heuristics override potentials to accompany mousing?

Anyway, that should be enough details to chew on, let me know if anything is unclear or so unexpected that it ought to be re-tested. Passing to you in the meantime.


Sat, 10 Sep 2016 09:22:02 GMT - Antoine Martin: owner changed

Also:

I may try to backport this ugly mess.


Tue, 13 Sep 2016 00:42:43 GMT - alas: owner changed

Ok, testing some more with 1.0 r13673 fedora 23 server and win32 client.

[jimador@jimador ~]$ xpra control :13 video-region-enabled 2 false
video region disabled for window 2
[jimador@jimador ~]$ xpra info :13 | grep video_subregion
window.2.video_subregion.counter=0
window.2.video_subregion.detection=True
window.2.video_subregion.enabled=False
window.2.video_subregion.supported=1
window.2.video_subregion.video-mode=False

Using d-feet SetVideoRegionEnabled (2, 1) (after having disabled detection in a separate test, so that false is expected)...

window.2.video_subregion.counter=0
window.2.video_subregion.detection=False
window.2.video_subregion.enabled=True
window.2.video_subregion.supported=1
window.2.video_subregion.video-mode=False

With the above settings though (subregion.detection=false & subregion.enabled=true), I didn't seem to be able to successfully set an arbitrary video region... at least according to the opengl paint boxes (the info says that I have successfully done so).

While running the following (with window.2.geometry=(2183, 172, 2168, 1164)), I get no blue h264 region displayed on the window (and no green vp8 region either, after testing #1104).

[jimador@jimador ~]$ xpra control :13 video-region 2 2355 550 300 300
video region set to (2355, 550, 300, 300) for window 2
window.2.video_subregion.detection=False
window.2.video_subregion.enabled=True
window.2.video_subregion.exclusion-zones=()
window.2.video_subregion.fps=0
window.2.video_subregion.height=300
window.2.video_subregion.in-out=(0, 0)
window.2.video_subregion.non-max-wait=150
window.2.video_subregion.rectangle=(2355, 550, 300, 300)
window.2.video_subregion.score=0
window.2.video_subregion.set-at=0
window.2.video_subregion.supported=1
window.2.video_subregion.time=0
window.2.video_subregion.video-mode=False
window.2.video_subregion.width=300
window.2.video_subregion.x=2355
window.2.video_subregion.y=550

Even more wondering, in my first try to set up an arbitrary video region, I was afraid the d-feet window (which I don't seem to be able to move on the desktop) was "covering" my arbitrary video region, so I moved the window, and ran the video region setting control channel command again... only to notice that I'd used "invalid" x,y coordinates... but that the xpra info seemed to suggest that the bad coordinates had been set nonetheless.

I ran the following:

[jimador@jimador ~]$ xpra control :13 video-region 2 1855 550 300 300
video region set to (1855, 550, 300, 300) for window 2
[jimador@jimador ~]$ xpra info :13 | grep subregion
window.2.video_subregion.counter=0
window.2.video_subregion.damaged=0
window.2.video_subregion.detection=False
window.2.video_subregion.enabled=True
window.2.video_subregion.exclusion-zones=()
window.2.video_subregion.fps=0
window.2.video_subregion.height=300
window.2.video_subregion.in-out=(0, 0)
window.2.video_subregion.non-max-wait=150
window.2.video_subregion.rectangle=(1855, 550, 300, 300)
window.2.video_subregion.score=0
window.2.video_subregion.set-at=0
window.2.video_subregion.supported=1
window.2.video_subregion.time=0
window.2.video_subregion.video-mode=False
window.2.video_subregion.width=300
window.2.video_subregion.x=1855
window.2.video_subregion.y=550

Then moved the window, then realized that was a foolish thing to do, checked on the new geometry: window.2.geometry=(2183, 172, 2168, 1164) ( had earlier been window.2.geometry=(1497, 170, 2168, 1164)).

I then ran the xpra info again, only to notice that the invalid video region was still officially set.

window.2.video_subregion.in-out=(0, 0)
window.2.video_subregion.non-max-wait=150
window.2.video_subregion.rectangle=(1855, 550, 300, 300)
window.2.video_subregion.score=0
window.2.video_subregion.set-at=0
window.2.video_subregion.supported=1
window.2.video_subregion.time=0
window.2.video_subregion.video-mode=False
window.2.video_subregion.width=300
window.2.video_subregion.x=1855
window.2.video_subregion.y=550

... all told though, I think "this ugly mess" is very very close.


Tue, 13 Sep 2016 03:53:22 GMT - Antoine Martin: owner changed

I get no blue h264 region displayed on the window


The fps is 0, we don't actually use the video encoders unless the framerate requires it. (10 fps or more)


I was afraid the d-feet window (which I don't seem to be able to move on the desktop)


Sounds like a bug, maybe #772? (affects osx and win32 only)


but that the xpra info seemed to suggest that the bad coordinates had been set nonetheless.


We don't validate any coordinates given by the API against the window dimensions, they could be valid one second and invalid the next: X11 is asynchronous and so is the command channel.

More importantly: these coordinates are all relative to the window, and you seem to be trying to set absolute coordinates.


Mon, 19 Sep 2016 19:24:15 GMT - alas: attachment set

setting: video-region 3 100 100 700 700 with detection off, works as expected


Mon, 19 Sep 2016 19:32:52 GMT - alas: attachment set

Setting: video-region 3 100 100 250 250 seems to create a region that is oddly/partially co-existing with previous 700 700 region


Mon, 19 Sep 2016 19:35:25 GMT - alas: attachment set

Setting:video-region 3 100 100 250 250 after shifting focus away from window then back - expected region seems to be set


Mon, 19 Sep 2016 19:36:39 GMT - alas: attachment set

Setting: video-region 3 100 100 250 250 sometimes seems to create a video region that is partially overridden by real video area


Mon, 19 Sep 2016 19:52:16 GMT - alas:

Ok, testing some more, this time with osx client 1.0 r13763 against a fedora 23 1.0 r13778 server, it looks like everything's working as expected, though with some paint box caveats (which may be expected too, but I will bring them up just in case).

First, a quick summary (because this ticket has become such a mess as I've taken too long to test cogently).

With detection enabled, setting a video region is treated as a hint - so mousing, scrolling, etc. is liable to cause the video detection to discard the region set by the api.

With detection disabled, regions set as video by the api will be respected - though only if there's enough updates in the region to justify h264 rather than other encodings. Scrolling, mousing, etc. will not override the set region (though, it looks like small regions are liable to use png or rgb, or occasionally jpeg... rather than h264... again, if there aren't enough updates being detected to justify h264).

When video-region-detection is on, this value is treated as a hint, but is liable to be discarded as "things" change.

When video-region-enabled is disabled, the setting of regions is ignored.

When video-region-detection is off, the region will be set according to the values set by the video-region call. Double checking this with opengl paint boxes, however, I'm seeing some odd behavior with small regions in particular.


-> And here I begin adding some details and xpra info for cases that seem strange.

setting: video-region 3 100 100 700 700 with detection off, works as expected

Some xpra info.

[jimador@jimador ~]$ xpra info :13 | grep -E '(fps=|subregion)'
window.1.damage.fps=0
window.1.video_subregion.counter=0
window.1.video_subregion.detection=True
window.1.video_subregion.enabled=True
window.1.video_subregion.supported=True
window.1.video_subregion.video-mode=False
window.2.damage.fps=3
window.2.video_subregion.counter=0
window.2.video_subregion.detection=True
window.2.video_subregion.enabled=True
window.2.video_subregion.supported=True
window.2.video_subregion.video-mode=False
window.3.damage.fps=80
window.3.encoder.fps=22
window.3.video_subregion.counter=0
window.3.video_subregion.damaged=0
window.3.video_subregion.detection=False
window.3.video_subregion.enabled=True
window.3.video_subregion.exclusion-zones=()
window.3.video_subregion.fps=0
window.3.video_subregion.height=700
window.3.video_subregion.in-out=(0, 0)
window.3.video_subregion.non-max-wait=150
window.3.video_subregion.rectangle=(100, 100, 700, 700)
window.3.video_subregion.refresh_region[0]=(100, 100, 700, 700)
window.3.video_subregion.score=0
window.3.video_subregion.set-at=0
window.3.video_subregion.supported=True
window.3.video_subregion.time=0
window.3.video_subregion.video-mode=False
window.3.video_subregion.width=700
window.3.video_subregion.x=100
window.3.video_subregion.y=100


Setting: video-region 3 100 100 250 250 seems to create a region that is oddly/partially co-existing with previous 700 700 region

More info.

[jimador@jimador ~]$ xpra info :13 | grep -E '(fps=|subregion)'
window.1.damage.fps=0
window.1.video_subregion.counter=0
window.1.video_subregion.detection=True
window.1.video_subregion.enabled=True
window.1.video_subregion.supported=True
window.1.video_subregion.video-mode=False
window.2.damage.fps=1
window.2.video_subregion.counter=0
window.2.video_subregion.detection=True
window.2.video_subregion.enabled=True
window.2.video_subregion.supported=True
window.2.video_subregion.video-mode=False
window.3.damage.fps=80
window.3.encoder.fps=19
window.3.video_subregion.counter=0
window.3.video_subregion.damaged=0
window.3.video_subregion.detection=False
window.3.video_subregion.enabled=True
window.3.video_subregion.exclusion-zones=()
window.3.video_subregion.fps=0
window.3.video_subregion.height=250
window.3.video_subregion.in-out=(0, 0)
window.3.video_subregion.non-max-wait=150
window.3.video_subregion.rectangle=(100, 100, 250, 250)
window.3.video_subregion.refresh_region[0]=(100, 100, 250, 250)
window.3.video_subregion.score=0
window.3.video_subregion.set-at=0
window.3.video_subregion.supported=True
window.3.video_subregion.time=0
window.3.video_subregion.video-mode=False
window.3.video_subregion.width=250
window.3.video_subregion.x=100
window.3.video_subregion.y=100


Setting:video-region 3 100 100 250 250 after shifting focus away from window then back - expected region seems to be set

More info.

[jimador@jimador ~]$ xpra info :13 | grep -E '(fps=|subregion)'
window.1.damage.fps=0
window.1.video_subregion.counter=0
window.1.video_subregion.detection=True
window.1.video_subregion.enabled=True
window.1.video_subregion.supported=True
window.1.video_subregion.video-mode=False
window.2.damage.fps=0
window.2.video_subregion.counter=0
window.2.video_subregion.detection=True
window.2.video_subregion.enabled=True
window.2.video_subregion.supported=True
window.2.video_subregion.video-mode=False
window.3.damage.fps=30
window.3.encoder.fps=0
window.3.video_subregion.counter=0
window.3.video_subregion.damaged=0
window.3.video_subregion.detection=False
window.3.video_subregion.enabled=True
window.3.video_subregion.exclusion-zones=()
window.3.video_subregion.fps=0
window.3.video_subregion.height=250
window.3.video_subregion.in-out=(0, 0)
window.3.video_subregion.non-max-wait=150
window.3.video_subregion.rectangle=(100, 100, 250, 250)
window.3.video_subregion.refresh_region[0]=(198, 226, 152, 124)
window.3.video_subregion.score=0
window.3.video_subregion.set-at=0
window.3.video_subregion.supported=True
window.3.video_subregion.time=0
window.3.video_subregion.video-mode=False
window.3.video_subregion.width=250
window.3.video_subregion.x=100
window.3.video_subregion.y=100


Setting: video-region 3 100 100 250 250 sometimes seems to create a video region that is partially overridden by real video area

And, yet more info for comparison.

window.3.damage.fps=43
window.3.encoder.fps=0
window.3.video_subregion.counter=0
window.3.video_subregion.damaged=0
window.3.video_subregion.detection=False
window.3.video_subregion.enabled=True
window.3.video_subregion.exclusion-zones=()
window.3.video_subregion.fps=0
window.3.video_subregion.height=250
window.3.video_subregion.in-out=(0, 0)
window.3.video_subregion.non-max-wait=150
window.3.video_subregion.rectangle=(100, 100, 250, 250)
window.3.video_subregion.refresh_region[0]=(204, 231, 146, 119)
window.3.video_subregion.score=0
window.3.video_subregion.set-at=0
window.3.video_subregion.supported=True
window.3.video_subregion.time=0
window.3.video_subregion.video-mode=False
window.3.video_subregion.width=250
window.3.video_subregion.x=100
window.3.video_subregion.y=100

Are these occasional indications of "additive" video regions, or "subtractive" indications, expected? Is it possible that the work to de-prioritize small region detection might be spilling over in the case of api-set regions?

If this is all expected, then I think we're good to close this ticket - though those small regions set by video-region api might be worth looking at a little bit first. Passing this back to you.


Mon, 19 Sep 2016 19:52:31 GMT - alas: owner changed


Tue, 20 Sep 2016 04:18:23 GMT - Antoine Martin: status changed; resolution set

Your small region has a zero fps, so it will not be using a video encoding. Then since it doesn't get updated, other areas will paint over it. This is working exactly as it should and even better than I expected.

Closing at last!


Sat, 23 Jan 2021 05:13:48 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1060