xpra icon
Bug tracker and wiki

Opened 4 years ago

Closed 4 weeks ago

Last modified 4 weeks ago

#365 closed enhancement (fixed)

don't copy pixmap data to ram: avoid the round-trips and stay on the GPU if we can

Reported by: Antoine Martin Owned by: Antoine Martin
Priority: minor Milestone: 2.1
Component: server Version:
Keywords: Cc:

Description

EXT_texture_from_pixmap should allow us to use the pixels on the GPU for doing CSC and/or encoding without first needing to copy them to RAM (via XShmGetImage or XGetImage as we do now)


XShmGetImage is pretty fast but then we have to upload the data again to the graphics card (assuming we do csc on the GPU - which is the fastest option) and then download the results. Quite wasteful, especially at high res.

Change History (14)

comment:1 Changed 4 years ago by Antoine Martin

Milestone: future1.0
Owner: changed from Antoine Martin to Antoine Martin
Status: newassigned

Here's how I think this can work.
Note: it might be easier to test this using "xpra shadow" and a full display copy since the "root window" never goes away, and we already have code to override behaviour for the root window (see GTKRootWindowModel).


  • when we reach window.get_image(x, y, w, h) in WindowSource, we can start copying the display pixels to a PBO (maybe even asynchronously!) using glReadPixels or glCopyTexImage2D and return an image wrapper for the PBO
  • when we reach driver.memcpy_htod in nvenc, we can just skip that part and instead use pycuda's gl functions to access the GL buffer (maybe it can be done as part of the NV12 CSC step anyway - even if we have to copy it to a CUDA aligned buffer, this is no big deal)


Obviously, we'll also need fallback code for dealing with non-nvenc encoders, and lots of other little details I can't foresee..


Links:

Last edited 4 years ago by Antoine Martin (previous) (diff)

comment:2 Changed 3 years ago by Antoine Martin

Milestone: 1.00.16

Scheduling for 0.16

comment:3 Changed 3 years ago by Antoine Martin

Another API we could potentially use (maybe just on win32?) is NvIFROpenGL, for which there is zero documentation...
Only this entry in the 319.49 driver changelog:

Added the NVIDIA OpenGL-based Inband Frame Readback (NvIFROpenGL) library to the Linux driver package. This library provides a high performance, low latency interface to capture and optionally encode an individual OpenGL framebuffer. NvIFROpenGL captures pixels rendered by OpenGL only and is ideally suited to application capture and remoting.

Although DRC seems to think it's not worth it:

I determined that the IFR stuff is not any faster than using PBOs

Last edited 3 years ago by Antoine Martin (previous) (diff)

comment:4 Changed 22 months ago by Antoine Martin

Milestone: 0.160.17

Re-scheduling.

comment:5 Changed 17 months ago by Antoine Martin

Milestone: 0.171.0

comment:6 Changed 13 months ago by Antoine Martin

Milestone: 1.01.1

Milestone renamed

comment:7 Changed 12 months ago by Antoine Martin

Milestone: 1.12.0

Milestone renamed

comment:8 Changed 11 months ago by Antoine Martin

For win32 we now have an API we can use #1317 "nvidia capture sdk support"

comment:9 Changed 6 months ago by Antoine Martin

Milestone: 2.02.1

comment:10 Changed 4 months ago by Antoine Martin

Milestone: 2.12.2

comment:11 Changed 2 months ago by Antoine Martin

comment:12 Changed 4 weeks ago by Antoine Martin

Milestone: 2.22.1

Boom! Done for NVFBC (#1317) + NVENC v8 (#1552) in r16458!
Improved in r16459, made the default in r16476.

Still TODO:

  • could keep the gpu buffer active, just in case we decide to actually use a video encoder, even after doing the scrolling detection and downloading the pixels to the CPU side (and we could make the scrolling detection tighter too when the gpu buffer is present)
  • could do scrolling detection via CUDA on the GPU
  • could do lz4 / zlib / whatever on GPU for small changes
Last edited 4 weeks ago by Antoine Martin (previous) (diff)

comment:13 Changed 4 weeks ago by Antoine Martin

Resolution: fixed
Status: assignedclosed

Will follow up in #1597. Closing at last!

comment:14 Changed 4 weeks ago by Antoine Martin

Also done for Linux in r16492: needed new NVENC kernels as Linux uses a different pixel format (XRGB vs BGRX).

Note: See TracTickets for help on using tickets.