CUDA and NVENC load balancing

[Antoine Martin]

Related to #504 and #466.

When we have multiple cards and/or multiple virtual cards (GRID K1, K2 and others) in the same server, we want to ensure that the load is fairly evenly distributed amongst all the (v)GPUs.

With CUDA, this isn't a problem. But with NVENC, we have no way of knowing how many contexts are still free. What happens when we reach the limit is that creating a new context will just fail...
We cannot assume that we are the only user of the device on the system, especially with proxy encoding (#504) where each proxy instances runs in its own process space.

The code added in r5488 moves the CUDA device selection (amongst other things) to a utility module and uses the percentage of free memory to choose the device to use. Since there are normally up to 32 contexts per GPU, this should work as a cheap load balancing solution: even with 4 vGPUs per PCIE slot, things will even out before we reach 20% capacity. This won't take into account the size of the encoding contexts, but since we reserve large context buffers in all cases (see r5442 - done for supporting #410) and since the sizes should be randomly distributed anyway, this should not be too much of a problem.
We lower the NVENC codec score as we create more contexts, and we also keep track of context failures to lower the score further (taking into account how recent the failure was). This should ensure that as we get closer to the limit, we become less likely to try to use NVENC, or that when we do hit the hard limit, we have a gradual grace period until we try again.

What remains to be done:

• link the NVENC context failures to the CUDA context they occurred on: other devices may still have free contexts, we should try those first if asked to create a new NVENC context
• maybe timeout the contexts: a context that has not been used for N seconds could probably be put to better use (may depend on current load - which is difficult to estimate in a proxy encoder context..)
• in the context of proxy encoding, as we lower the NVENC codec score, we will still receive RGB frames from the server being proxied and so we need to fallback to x264 or another encoding. At the moment, we fail hard if we cannot find a fallback video encoder..

Notes:

• see r5487
• see r5483

[Antoine Martin]

• r5492 lets us fallback to any other video encoder if we can't instantiate the one we want: it will try x264 and even vpx if present. Failing that we then try to use jpeg, and as a last fallback we default to plain RGB as we received it (which is bad, but should never happen and it is still better than not sending any pixels at all!)
• r5496 will try CUDA devices that have not had recent failures ahead of those that did - this takes precedence over the "percentage of free memory" sorting, this should allow us to reach close to 100% occupancy on multi-GPU / VGPU setups.
• r5497 will timeout video encoding contexts if unused for more than 5 seconds. This code applies to the proxy instance only - not for the main server yet, since it's less likely to have as much contention for resources

(r5493 was missing from previous commits - oops)

smo: this is good enough for some testing... and I only have one card, so I cannot really test it very well.

Things to lookout for:

• #517 regressions: I believe this load balancing code should be safe from leaks, but I cannot be certain. Easiest thing to do is resize a fast updating window (and if possible, do that on two contexts that live on different cards..), which should cause many encoder re-init: destroying and creating new NVENC contexts for the new window sizes. The GPU's free memory should remain relatively constant throughout.
• utilization: can we get close to 100% of encoding contexts used? (32 contexts per card.. this will take a lot of clients and windows)
• start multiple servers or use proxy encoding (#504): does the code still manage to allocate encoding contexts properly? (and fallback/retry as needed)
• how is the initial connection delay: having to initialize the CUDA context *after* the client connects means that there will be an extra delay, even more so when there are multiple cards to probe. Is it bearable?

etc..

[Smo]

• Haven't found any leaks and was able to load balance up to 10 sessions.
• Tested with proxy encoding with 10 sessions with no issues.
• The connection delay doesn't seem to be an issue and is hardly noticeable.

closing for now will reopen if there are issues

[Antoine Martin]

See also: new CUDA load balancing feature in #2416.

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/520