Xpra: Ticket #800: allow B frames with video encoders

Tue, 21 Jul 2015 11:38:52 GMT - Antoine Martin: attachment set

work in progress to support b frames

Wed, 22 Jul 2015 12:28:08 GMT - Antoine Martin: owner, status changed

Code to flush the encoder:

while( x264_encoder_delayed_frames( h ) )
    i_frame_size = Encode_frame( h, opt->hout, NULL );
    if( i_frame_size < 0 )
        return -1;
    i_file += i_frame_size;

What we want is to keep at most N frames in the encoder at any point in time. Where N is very low without av-sync (maybe just 1), or derived from the av-sync value. We will need a timer to flush things if no new frames come in, and this belongs in the window source logic.

Fri, 13 Nov 2015 06:52:29 GMT - Antoine Martin: milestone changed


Wed, 16 Mar 2016 05:34:41 GMT - Antoine Martin: milestone changed

Fri, 17 Jun 2016 08:05:07 GMT - Antoine Martin:

For NVENC, we will have to ensure we don't exceed NV_ENC_CAPS_NUM_MAX_BFRAMES.

From the headers for NV_ENC_CONFIG.frameIntervalP: Specifies the GOP pattern as follows: frameIntervalP = 0: I, 1: IPP, 2: IBP, 3: IBBP If goplength is set to NVENC_INFINITE_GOPLENGTH frameIntervalP should be set to 1.

Sat, 18 Jun 2016 15:46:13 GMT - Antoine Martin:

Done for x264 in r12858.

Still todo:

Somewhat related: Explanation of x264 tune, maybe we could tune to "fastdecode" when we find the client is struggling with the decode speed. At the moment we just increase the batch delay which may lower the speed and increase the complexity for the decoding.. making things worse.

Sun, 19 Jun 2016 08:45:41 GMT - Antoine Martin: owner, status changed

@afarr: ready for testing. b-frames should compress better, and make better use of the delay used for av-sync. Verifying that we compress better is hard because of the usual problems: "define better". So many heuristics are intertwined that we could get higher fps, lower bandwidth, higher quality, etc...

What you can check though:

Fri, 24 Jun 2016 18:24:28 GMT - J. Max Mena: owner changed

With a trunk r12899 server and client (both Fedora 23, connected over TCP), I'm getting some really really weird painting when I enable B-Frames. I'll attach a quick screenrecord.

That being said, with B-Frames enabled video appears to be better than with B-Frames disabled. That's just my subjective opinion, we'll need some kind of performance tests edit: as mentioned in comment:5.

Fri, 24 Jun 2016 18:24:59 GMT - J. Max Mena: attachment set

Recorded with OBS - scrolling and switching tabs in Firefox

Tue, 28 Jun 2016 14:00:00 GMT - Antoine Martin: owner changed

I believe that what you are recording is caused by the delayed video frames used for the whole browser window: those arrive out of order, after the other updates.

Does that improve things?

Tue, 28 Jun 2016 16:24:49 GMT - J. Max Mena: owner changed

Upped client and server to r12942:

Somewhat. I don't get the full repaints on switching tabs, but I do still see some painting issues that I only get with B-Frames enabled. The easiest to show is collapsing and uncollapsing images on Reddit with RES enabled in Firefox. I'll attach a screenrecord to demonstrate.

Tue, 28 Jun 2016 16:25:56 GMT - J. Max Mena: attachment set

Scrolling and collapsing images on Reddit - notice the flicker at the end.

Sun, 03 Jul 2016 17:09:52 GMT - Antoine Martin:

Support for b-frames was included in the enc_ffmpeg codec, see ticket:1107#comment:10.

The issue shown on the video above is likely due to the fact that we mix video and non-video encodings on the page. The video gets delayed, but the non-video is not. This is a similar issue to #1218. The best solution is probably to only use b-frames for video regions, which is not easy... will try.

Tue, 12 Jul 2016 16:52:22 GMT - Antoine Martin: milestone changed

Milestone renamed

Fri, 15 Jul 2016 15:14:38 GMT - Antoine Martin: status changed

See r13022 and #1257.

We also need to skip some frames if we've sent them as jpeg already.

Wed, 20 Jul 2016 08:24:05 GMT - Antoine Martin:

Wed, 20 Jul 2016 19:26:13 GMT - Antoine Martin: owner, status changed

@maxmylyn: can you reproduce with the latest code? (see also: ticket:1257#comment:3)

Wed, 20 Jul 2016 19:32:45 GMT - J. Max Mena: owner changed

Yes, it's still problematic as of r13052.

Most notably, you can see it by scrolling text-sites in Firefox, and entering text in text-boxes (Trac comments, yelling at people through the internet, etc etc), and by changing tabs.

It's better, but still nowhere as good as before the B-Frames started.

Wed, 20 Jul 2016 20:00:00 GMT - Antoine Martin: owner changed

It's better, but still nowhere as good as before the B-Frames started.

Does it go away if you run the with b-frames disabled? (XPRA_B_FRAMES=0)

If so, then we're using b-frames for the main window area, and as per ticket:1257#comment:3, this should only occur if the whole window is detected as a video area. Can you confirm? Is the video region detection getting it wrong? (#410)

Using XPRA_VIDEO_SUBREGION=0 should have a similar effect: disabling video regions should now also disable b-frames.

Wed, 20 Jul 2016 22:21:04 GMT - J. Max Mena: owner changed

Yes, you're correct - it only shows up when the whole page paints as h264. I don't think the region detect is failing, but rather a corner case we're not handling properly.

And, yes, it goes away with XPRA_B_FRAMES=0

Sorry, I had this ticket all typed out and never clicked submit.

Thu, 21 Jul 2016 10:40:51 GMT - Antoine Martin: status changed; resolution set

OK, so let's keep this ticket only about b-frames, we can handle the blurriness caused by the video detection firing for the browser in #967 (re-opened). #1232 will help a lot here as we may be able to avoid using video encodings when scrolling.

Unless proven otherwise, b-frames are being used when we detect video. So I am closing this, we can follow up and test some more in #1257. See also #1261 for multiple b-frames support.

Thu, 25 Aug 2016 21:39:16 GMT - alas: status changed; resolution deleted

Re-opening... as mentioned originally in #1290, when mousing "vigorously" over a video region for several seconds, frames occasionally seem to display out of order. This behavior isn't seen when running with XPRA_B_FRAMES=0.

Testing with 1.0 r13101 win32 client against 1.0 r13410 fedora 23 server, with -d regionrefresh,mouse I got the following logs (migrated from #1290): attachment/ticket/800/800-regionrefresh-mouse.log

The frame rate drops I left details about in #1290, but the out-of-order frames don't start to be noticed until the frame rate starts to drop.

Using watch -n 1 "xpra info | grep fps=" to watch the frame rate, I am seeing a video_subregion.fps value, which suggests that the region is still painting as video (from the reduced framerate):


... but the -d paint logs from the client seem to be indicating a lot of RGB paint state -though, looking through some more, I do see that it is also periodically switching back to a YUV paint state (video?).

Looking at the -d paint logs with b-frames enabled vs. disabled I'm not seeing much difference, but I'll paste a sample of each for you to look at.

Mousing over video region with XPRA_B_FRAMES=0: attachment/ticket/800/800-nobframes.log

Doing the same without disabling the b-frames (seems to take a lot longer to switch back to YUV): attachment/ticket/800/800-bframes.log

Fri, 26 Aug 2016 09:50:56 GMT - Antoine Martin: attachment set

with b-frames

Fri, 26 Aug 2016 09:52:28 GMT - Antoine Martin: attachment set

without b-frames

Fri, 26 Aug 2016 09:53:51 GMT - Antoine Martin: attachment set

regionrefresh and mouse debug

Fri, 26 Aug 2016 09:58:19 GMT - Antoine Martin:

Wed, 07 Sep 2016 05:20:50 GMT - Antoine Martin: owner, status changed

It is possible that when the fps drops lower than 10, we switch back to a non-video encoding whilst a b-frame is still due. This would cause the non-video frame to arrive before both the delayed b-frame and its auto-refresh... r13598 should fix this by excluding the video region whenever a b-frame is still pending. There may still be a corner case: if the video region is moving on screen (ie: the page is scrolling), then we may end up excluding the region where it was rather than where it actually is... r13599 tries to prevent that.

This may be easier to test before #1299 is fixed (assuming it is our problem to fix). This issue could also be triggered by playing a youtube video (high fps) then stopping it (the last few updates will be low fps), but it is going to be hard to spot: b-frames only happen 50% of the time and those last few frames are likely identical... a better option may be to use opengl paint box, or "-d compress" (server side) or "-d paint" (client side).

Fri, 09 Sep 2016 11:31:36 GMT - Antoine Martin:

See also ticket:1232#comment:16

Fri, 16 Sep 2016 16:44:01 GMT - J. Max Mena:

With an r13767 Fedora 23 Server and Client:

(As mentioned in the scrum Tuesday - wasn't in office to make a screenrecord)

Fri, 16 Sep 2016 16:49:16 GMT - J. Max Mena:

Okay, so the video I made is too large to attach here, so I uploaded it to Google drive - I'll paste the link into the bottom of this ticket.

Also of note:


The actual link:


Sat, 17 Sep 2016 07:44:54 GMT - Antoine Martin: owner changed

This behaviour may well be unavoidable to some extent: we don't know if and when a new frame is going to come through... so we schedule a flush of the video encoder just in case nothing comes (the delay varies, it will be longer when we're quite busy). r13773 improves this by also flushing before doing any kind of screen refresh. This should make things appear smoother: you'll get another intermediate b-frame before the lossless refresh.

Another fix: prior to r13775, the video encoder would only use b-frames if we detected "true video" before (re)creating the video encoder. The new code can toggle this on or off as the results from the heuristics change. This should result in b-frames being used in more cases, which may also make some issues more apparent..

PS: #1014 may make it more difficult to test b-frames at certain screen sizes, you may want to force "encoding=h264" if that's the case.

Mon, 19 Sep 2016 23:10:45 GMT - J. Max Mena: owner changed

r13773 seems to have fixed my corner case and I'm having a hard time finding another one, despite trying very vigorously.

As farr (heh) as I can tell, it's been fixed; even after several hours testing. Even Chrome which is very liberal with its painting is behaving well.

Passing back to you to decide what to do with this ticket.

Tue, 20 Sep 2016 03:55:22 GMT - Antoine Martin: status changed; resolution set

Thanks! Closing at last.

Sat, 23 Jan 2021 05:06:20 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/800