Xpra: Ticket #385: opengl rendering improvements: handle plain RGB, scaling, transparency

See wiki/ClientRendering and #350 (which is probably needed first)

In order to bring feature parity with the regular gtk2 pixmap backing, we would need:

This will make the code more manageable (fewer special cases), should make the overall experience better (ie: we can then use RGB modes with mmap and gl, x264 can encode to/from rgb without doing csc server side) and will give the opengl more of a chance to be used.

Thu, 18 Jul 2013 05:43:50 GMT - Antoine Martin: description changed

Thu, 18 Jul 2013 05:47:02 GMT - Antoine Martin: description changed

Sun, 21 Jul 2013 05:22:33 GMT - Antoine Martin: description changed

scaling completed in r3906

Sun, 21 Jul 2013 05:25:52 GMT - Antoine Martin: description changed

Sun, 21 Jul 2013 06:17:19 GMT - Antoine Martin:

The problem with video RGB painting is that the data coming out of x264 is in planar mode ("GBRP") so we cannot re-use _do_paint_rgb24 as is.

Can we update the fbo one plane at a time? Maybe using a new texture for each plane? Or is our only option to first convert to plain RGB and re-use the _do_paint_rgb24 code?

The client-side conversion is probably almost as costly as doing YUV444 to RGB csc server-side before compression, but since we are CPU bound server-side, we are much better off shifting this burden to the client. (effectively lowering the overall latency)

Sun, 21 Jul 2013 06:18:20 GMT - Antoine Martin: description changed

Sun, 21 Jul 2013 08:04:35 GMT - ahuillet:

Theoretically we can paint planar RGB. It requires a separate shader and some more support code, however.

Sun, 21 Jul 2013 13:37:38 GMT - Antoine Martin:

Planar RGB done in r3928 + r3927 + r3926 + r3925 + r3923 + r3922 + r3921 + r3920

Sun, 28 Jul 2013 05:20:21 GMT - Antoine Martin:

Looks to me like the new buffer management code (#350) is making PyOpenGL unhappy (100% reproducible):

OpenGL paint error: ("No array-type handler for type \
    <type 'buffer'> (value: <read-only buffer ptr 0x7f394011b910, size 171904 ) registered", \
    <OpenGL.GL.images.ImageInputConverter object at 0x2277390>)
Traceback (most recent call last):
  File "xpra/client/gl/gl_window_backing.py", line 376, in gl_paint_planar
    self.update_planar_textures(x, y, enc_width, enc_height, img, pixel_format, scaling=(enc_width!=width or enc_height!=height))
  File "xpra/client/gl/gl_window_backing.py", line 431, in update_planar_textures
    glTexSubImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, x, y, width/div_w, height/div_h, GL_LUMINANCE, GL_UNSIGNED_BYTE, pixel_data)
  File "OpenGL/latebind.py", line 45, in __call__
    return self._finalCall( *args, **named )
  File "OpenGL/wrapper.py", line 781, in wrapperCall
    pyArgs = tuple( calculate_pyArgs( args ))
  File "OpenGL/wrapper.py", line 354, in calculate_pyArgs
    yield converter(args[index], self, args)
  File "OpenGL/GL/images.py", line 433, in __call__
    return arrayType.asArray( arg )
  File "OpenGL/arrays/arraydatatype.py", line 141, in asArray
    return cls.getHandler(value).asArray( value, typeCode or cls.typeConstant )
  File "OpenGL/arrays/arraydatatype.py", line 52, in __call__
    typ, repr(value)[:50]
TypeError: ("No array-type handler for type <type 'buffer'> \
    (value: <read-only buffer ptr 0x7f394011b910, size 171904 ) registered", \
    <OpenGL.GL.images.ImageInputConverter object at 0x2277390>)

(This particular trace occurred with GBRP when using quality=100) Solutions (most of which assume that the problem is to do with read-only memory - TBC):

Sun, 28 Jul 2013 06:04:41 GMT - Antoine Martin:

Damn, read-write makes no difference, so maybe PyOpenGL requires a new style buffer. PITA

This works around the issue by copying to a new string object before passing the data to gl:

--- src/xpra/client/gl/gl_window_backing.py	(revision 3977)
+++ src/xpra/client/gl/gl_window_backing.py	(working copy)
@@ -426,7 +426,7 @@
             glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textures[index])
             glPixelStorei(GL_UNPACK_ROW_LENGTH, rowstrides[index])
-            pixel_data = img_data[index]
+            pixel_data = img_data[index][:]
             debug("texture %s: div=%s, rowstride=%s, %sx%s, data=%s bytes", index, divs[index], rowstrides[index], width/div_w, height/div_h, len(pixel_data))
             glTexSubImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, x, y, width/div_w, height/div_h, GL_LUMINANCE, GL_UNSIGNED_BYTE, pixel_data)
             if index == 1:

Sun, 28 Jul 2013 11:54:33 GMT - Antoine Martin:

r3985 applies the change from comment:10, just so we can move on and use the code, but this will need a better solution eventually.

Tue, 30 Jul 2013 09:44:27 GMT - Antoine Martin:

r4017 uses a string wrapper rather than a buffer wrapper, which makes pyopengl happy and allows us to revert the extra buffer copy added in r3985.

I've left swscale alone, since we don't use it with opengl, but it may need the same treatment if we ever do csc with opengl (for whatever reason).

Items left to do:

Mon, 05 Aug 2013 05:53:14 GMT - Antoine Martin:

r4017 was "wrong": the string wrapper makes a copy... which is what we tried to avoid, so r4048 reverts that and just makes the buffer copy from the decoding thread instead of the UI thread (and only for opengl)... so we still need to find a better way to make the buffer work with zero copy and pyopengl... maybe The new-style Py_buffer struct

Thu, 17 Oct 2013 07:42:05 GMT - Antoine Martin: owner, status, milestone changed

Nice-to-have but not essential, re-scheduling.

Wed, 06 Nov 2013 11:55:22 GMT - Antoine Martin:

r4480 made the pixel buffer read/write, so we no longer have to copy the pixels before uploading them with glTexSubImage2D (see comment:9), removed copying in r4702

Sun, 08 Dec 2013 10:14:52 GMT - Antoine Martin: attachment set

working patch for posix clients (win32 broken)

Sun, 08 Dec 2013 12:24:17 GMT - Antoine Martin: summary changed

r4880 + r4881 + r4883 adds transparency support for GL windows, r4879 (server side), backported in r4894 so that older servers don't end up sending BGRA pixels without telling the client the data is in BGRA and not RGBA... (as is normally the case when we don't specify)

This change is already very useful for mmap/rgb modes where we now handle 32-bit pixel data in native BGR(A) format, so the server no longer needs to do any byteswapping and/or losing the alpha channel, it will also be very useful with vp9/h265 which both support alpha channels.

Buffer improvements task is now in its own ticket: #465

Last remaining item for this ticket is the maximum GL texture size, which is made more difficult by #263: we cannot just set a maximum window size as this also changes the behaviour of the UI..

Sun, 08 Dec 2013 14:48:04 GMT - Antoine Martin: owner, status changed

r4882 + r4883 improves texture size checking and ensures we have a fallback in case GL window setup fails. We already had a check to disable GL if the total screen space is bigger than the maximum texture supported. Ideally, we would also deal with failures during window resize, as one can resize a window and make it bigger than the screen size... but this will do for now (modern graphics card support 16k x 16k texture sizes which is more than we need).
One remaining niggle is the fact that I cannot get OpenGL to use the premultiplied alpha despite following the blending docs... (see r4902)

afarr: please test combinations I do not have access to (as of r4977, one needs to use XPRA_ALPHA=1 xpra attach ... to try to enable alpha channel on osx and win32 where it is normally disabled by default):

To make sure that there are no regressions - the performance improvements are too small to be measurable under normal usage.

The video encodings (vpx and h264) should be unchanged, rgb, png and webp should support window transparency (xpra.test_apps.transparent_colors can be used to test transparency easily) on OpenGL windows (verify window type is GL in session info).

Wed, 18 Dec 2013 08:37:08 GMT - Antoine Martin: milestone changed

This belongs in the 0.11 milestone, new ticket added for resizing flickering for 0.12: #478

Wed, 18 Dec 2013 08:44:16 GMT - Antoine Martin: milestone changed

Oooops, wrong milestone update

Thu, 09 Jan 2014 01:15:16 GMT - alas:

Testing with win client 0.11.0 r5144 (fedora 19 0.11.0 r5144): with rgb and png the transparent colors look lovely. With webp the squares are the right colors but streaked with black (not lovely).

Testing with osx client 0.11.0 r5144 produces the same results: rgb and png work as expected and webp doesn't support the transparencies.

Thu, 09 Jan 2014 06:52:46 GMT - Antoine Martin: status changed; resolution set

The webp issue is not GL related (which is what this ticket is about), moved to #487


Sat, 23 Jan 2021 04:53:48 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/385