xpra icon
Bug tracker and wiki

Opened 3 months ago

Closed 2 months ago

Last modified 2 months ago

#2113 closed enhancement (fixed)

Improving region detect on an OpenGL canvas

Reported by: Nathan Hallquist Owned by: Antoine Martin
Priority: major Milestone: 2.5
Component: encodings Version: 2.4.x
Keywords: Cc:

Description (last modified by Antoine Martin)

This ticket pursues the issue of ticket:2022#comment:47.

The desired behavior is for the video region to coincide with the OpenGL canvas throughout the entire run:

0x1000886 (has no name): ()  1582x854+3+29  +1874+6

I've attached a log.

The first rotation operation happens from 9:31 to 9:50

The second rotation happens until from the first log after the pause to the end.

In general, with logging enabled the whole application seems to be slow, but the underlying problem remains very detectable.

Attachments (4)

region.txt (216.8 KB) - added by Nathan Hallquist 3 months ago.
log-children.patch (1.5 KB) - added by Antoine Martin 3 months ago.
regularly log the client window's children
gl_gtk_app.py (29.5 KB) - added by Antoine Martin 2 months ago.
example app with an embedded opengl subwindow
post-patch-log.txt (39.0 KB) - added by Nathan Hallquist 2 months ago.

Download all attachments as: .zip

Change History (13)

Changed 3 months ago by Nathan Hallquist

Attachment: region.txt added

Changed 3 months ago by Antoine Martin

Attachment: log-children.patch added

regularly log the client window's children

comment:1 Changed 3 months ago by Antoine Martin

Owner: changed from Antoine Martin to Nathan Hallquist

If the window has a child window just for its canvas then we could use this geometry as a strong hint that this can be used as a region of interest during video region detection.
Can you post the full xwininfo -root -tree for this application?
And / or apply the patch above and see what comes out.

Note: unfortunately, this won't work for most browsers, chromium uses a single canvas for the entire window and does its own compositing:

     0x3600001 "SomeSite - Google Chrome": ("google-chrome-unstable" "Google-chrome-unstable")  2399x2000+343+60  +343+60
        2 children:
        0x3e00003 (has no name): ()  1x1+0+0  +343+60
        0x380000d (has no name): ()  2399x2000+0+0  +343+60

comment:2 Changed 3 months ago by Nathan Hallquist

I think this is a really good idea. Too bad it doesn't work for web browsers. I'm not familiar with how browser plugins work, but it would be nifty if there were a way to hint XPRA from a plugin.

I'm providing the xwininfo. If you want the patch, I'll do that. Let me know.

Here's the highlight:


0x100088c (has no name): ()  1370x776+3+29  +832+60

Here's the whole thing:

xwininfo: Window id: 0x311 (the root window) "Xpra"

  Root window id: 0x311 (the root window) "Xpra"
  Parent window id: 0x0 (none)
     21 children:
     0x40002b "Xpra-CorralWindow-0xc00022": ()  1196x752+8+90  +8+90
        1 child:
        0xc00022 "nathan@curry:~/lsprepost4.7_centos7": ("xterm" "XTerm")  1196x752+0+0  +8+90
           1 child:
           0xc0002f (has no name): ()  1196x752+0+0  +8+90
              1 child:
              0xc00034 (has no name): ()  14x752+-1+-1  +7+89
     0x40011e "Xpra-CorralWindow-0x10005de": ()  1526x899+829+31  +829+31
        1 child:
        0x10005de "LS-PrePost(R) V4.7.0 (Beta) - 27Dec2018-64bit ../issue/1.d3plot": ("lsprepost" "Lsprepost")  1526x899+0+0  +829+31
           6 children:
           0x10005ff (has no name): ()  673x23+689+853  +1518+884
           0x1000d62 (has no name): ()  680x17+4+856  +833+887
           0x10005f5 (has no name): ()  736x32+604+883  +1433+914
           0x1000d61 (has no name): ()  595x26+4+886  +833+917
           0x100088c (has no name): ()  1370x776+3+29  +832+60
           0x10005df (has no name): ()  1x1+-1+-1  +828+30
     0x40012b "Xpra-CorralWindow-0x1000b02": ()  455x136+3037+64  +3037+64
        1 child:
        0x1000b02 "Animate": ("lsprepost" "Lsprepost")  455x136+0+0  +3037+64
           6 children:
           0x1000d67 (has no name): ()  32x26+418+20  +3455+84
           0x1000d66 (has no name): ()  52x26+317+20  +3354+84
           0x1000d65 (has no name): ()  22x26+248+20  +3285+84
           0x1000d64 (has no name): ()  32x26+180+20  +3217+84
           0x1000d63 (has no name): ()  32x26+104+20  +3141+84
           0x1000b03 (has no name): ()  1x1+-1+-1  +3036+63
     0x1004340 "Configuration Settings": ("lsprepost" "Lsprepost")  994x711+0+0  +0+0
        9 children:
        0x10044d0 (has no name): ()  32x26+272+534  +272+534
        0x10044cf (has no name): ()  102x26+336+490  +336+490
        0x10044ce (has no name): ()  96x26+716+413  +716+413
        0x10044cd (has no name): ()  152x26+681+241  +681+241
        0x10044cc (has no name): ()  52x26+546+241  +546+241
        0x10044cb (has no name): ()  195x29+660+146  +660+146
        0x10044ca (has no name): ()  102x26+299+73  +299+73
        0x10045b1 (has no name): ()  148x660+5+5  +5+5
        0x1004341 (has no name): ()  1x1+-1+-1  +-1+-1
     0x100433d "lsprepost": ("lsprepost" "Lsprepost")  200x200+0+0  +0+0
        1 child:
        0x100433e (has no name): ()  1x1+-1+-1  +-1+-1
     0x10042ed "lsprepost": ("lsprepost" "Lsprepost")  227x154+1176+56  +1176+56
        1 child:
        0x10042ee (has no name): ()  1x1+-1+-1  +1175+55
     0x1000b1e "LS-PrePost 4.x Usage": ("lsprepost" "Lsprepost")  520x191+0+0  +0+0
        2 children:
        0x1000b27 (has no name): ()  485x93+11+40  +11+40
        0x1000b1f (has no name): ()  1x1+-1+-1  +-1+-1
     0x10005f9 (has no name): ()  1x1+-1+-1  +-1+-1
     0x1000003 "lsprepost": ("lsprepost" "Lsprepost")  200x200+0+0  +0+0
        1 child:
        0x1000004 (has no name): ()  1x1+-1+-1  +-1+-1
     0x1000001 "lsprepost": ("lsprepost" "Lsprepost")  10x10+10+10  +10+10
        1 child:
        0x1000002 (has no name): ()  1x1+-1+-1  +9+9
     0xe00001 "Xpra Audio record": ("Xpra-Audio-record" "Xpra-Audio-record")  10x10+10+10  +10+10
     0x400026 (has no name): ()  1x1+-1+-1  +-1+-1
     0x400022 "Xpra-SystemTray": ()  1x1+0+0  +0+0
        1 child:
        0x400023 (has no name): ()  1x1+-1+-1  +-1+-1
     0x400020 "Xpra": ()  10x10+-100+-100  +-100+-100
     0x40001f "Xpra": ()  10x10+-100+-100  +-100+-100
     0x40001e "Xpra": ()  10x10+-100+-100  +-100+-100
     0x40001b "Xpra-WorldWindow": ("xpra" "Xpra")  3840x1080+0+0  +0+0
        1 child:
        0x40001c (has no name): ()  1x1+-1+-1  +-1+-1
     0x40001a "Xpra": ()  1x1+0+0  +0+0
     0x400003 "Xpra-ManagerSelection": ()  10x10+-100+-100  +-100+-100
     0x400001 "xpra": ("xpra" "Xpra")  10x10+10+10  +10+10
        1 child:
        0x400002 (has no name): ()  1x1+-1+-1  +9+9
     0x200001 "xpra": ("xpra" "Xpra")  10x10+10+10  +10+10
        1 child:
        0x200002 (has no name): ()  1x1+-1+-1  +9+9

comment:3 Changed 3 months ago by Antoine Martin

Owner: changed from Nathan Hallquist to Antoine Martin
Status: newassigned
Type: defectenhancement

I'll assume that xpra sees the same values as xwininfo.

Changed 2 months ago by Antoine Martin

Attachment: gl_gtk_app.py added

example app with an embedded opengl subwindow

comment:4 Changed 2 months ago by Antoine Martin

Description: modified (diff)
Resolution: fixed
Status: assignedclosed

Using the example code from Getting Python, GTK and GL to play nice, we can get a test window with a 640x480 opengl view embedded in it.
xwininfo sees:

     0x64cafe (has no name): ()  660x635+-10+26  +-10+26
        1 child:
        0x5600003 "ApplicationMainWindowDemo": ("gl_gtk_app.py" "Gl_gtk_app.py")  640x580+10+45  +0+71
           2 children:
           0x5600021 (has no name): ()  640x480+0+75  +0+146
           0x5600004 (has no name): ()  1x1+-1+-1  +-1+70

Done in r21651.
The children windows will now also be visible in xpra info | grep children=.
Using the test application attached, I see:

windows.1.children=((16777249, 0, 75, 640, 480, 0, 24),)

Which is exactly where the opengl area is, and it is correctly identified as a video area.

It was correctly identified without this code, but the new code gives it a score boost which will help other cases where the existing heuristics might not have spotted it, and it should also help when the subwindow is moved or resized.

@nathan_lstc: video region should work better out of the box for applications with an opengl sub-window.

comment:5 Changed 2 months ago by Nathan Hallquist

I started an older version of my 3D software (that doesn't talk to dbus) and set the XPRA_CONTENT_TYPE by hand with xprop.

xprop -format _XPRA_CONTENT_TYPE 8s -set _XPRA_CONTENT_TYPE video

When I did that it launches right into h264, which is desired. Then it locks and then it goes for a short time and then it locks and then it goes until I stop rotating. The screen-locking seem to coincide with sburegion recalculations in the log. I've attached a log compress,regiondetect. I will defer to you on whether to reopen this ticket or not. I have the qualitative impression that it is better. I can downgrade and test an older revision, and I will do that if you request it (but I'd rather not).

Last edited 2 months ago by Antoine Martin (previous) (diff)

Changed 2 months ago by Nathan Hallquist

Attachment: post-patch-log.txt added

comment:6 Changed 2 months ago by Antoine Martin

Resolution: fixed
Status: closedreopened

comment:7 Changed 2 months ago by Antoine Martin

Resolution: fixed
Status: reopenedclosed

The pauses that you see are caused by the nvenc re-initializing, as we saw in #2022 the first frame can take a long time to encode:

10:55:50,997 compress: 728.0ms for 1598x892  pixels at    3,29   for wid=2     using      h264 with ratio   1.4%  ( 5574KB to    80KB), sequence    86, client_options={u'speed': 94, u'frame': 0, u'pts': 0, 'csc': 'YUV444P', u'quality': 75}
10:55:51,782 compress: 450.8ms for 1598x892  pixels at    3,29   for wid=2     using      h264 with ratio   0.8%  ( 5574KB to    42KB), sequence    90, client_options={u'speed': 94, u'frame': 0, u'pts': 0, 'csc': 'YUV444P', u'quality': 79}

It starts with x264 and no video region (encoding the whole window as video), which is expected as per r21057 - #2044: we don't want to use nvenc until we are certain that the video stream is here to stay.
Then it switches to nvenc (first pause), processes just 4 video frames and then it detects the video region and has to re-initialize nvenc (second pause).

I am closing this ticket again because problem is specific to nvenc and belongs in #2048.

Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:8 Changed 2 months ago by Antoine Martin

r21670 makes the sub-window region boost configurable using an env var: XPRA_SUBWINDOW_REGION_BOOST (defaults to 20).

Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:9 Changed 2 months ago by Antoine Martin

r21671 fixes python3 compatibility and more.

Note: See TracTickets for help on using tickets.