Video players will try accelerated adapters first (xvideo, etc) then fallback to painting RGB. Since we want to compress the video as a video stream, there is no point having the video players convert their stream to RGB for us to convert it back to YUV straight after for encoding!
One solution to this is a Wayland-Xpra (#387), but Wayland isn't ready for prime time yet and won't cover X11 applications well enough (I don't think), at least for now.
A solution for X11 would be to implement a simple Xv adapter in the dummy driver (or via please enable Xv support on glamor for the Intel video driver).
The flash player doesn't use libva apparently, so #510 is not an option there, but maybe we can use a technique similar to libvdpau-va-gl and capture the YUV stream even before it is scaled up.
I'm not sure how userspace would then register/grab the YUV data from the dummy-Xv adapter, but this would make #410 (better sub-window encoding: detect regions and use sub video encoder) much easier to implement.
The xv interface added to Xephyr seems simple enough: https://github.com/XQuartz/xorg-server/blob/master/hw/kdrive/ephyr/ephyr_glamor_xv.c
I'm not sure if we can add this to the dummy driver or if this needs to be added somewhere else. Will try it out with mostly empty stubs.
Probably superseded by #387
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/509