xpra icon
Bug tracker and wiki

Opened 2 years ago

Closed 21 months ago

Last modified 21 months ago

#878 closed enhancement (fixed)

lz4: support HC and fast modes

Reported by: Antoine Martin Owned by: alas
Priority: minor Milestone: 0.16
Component: core Version: trunk
Keywords: Cc:

Description

  • compressHC is already available in 0.7.0, so we can support this already - it is not very useful (almost as slow as zlib), but I guess we can enable it for compress=9
  • LZ4_compress_fast is in trunk only, so this would require a runtime check

The mapping should go something like:

  • 0 = no compression
  • 1 = fast 50
  • 2 = fast 17
  • 3 = fast 5
  • 4 to 8 = standard
  • 9 = HC

Attachments (3)

python-lz4-0.8.0.rc1.tar.xz (23.5 KB) - added by Antoine Martin 2 years ago.
0.8.0.rc1 snapshot
python-lz4-win32.patch (9.9 KB) - added by Antoine Martin 23 months ago.
quick and dirty patch for building on win32
lz4-msvc-fix.patch (1.7 KB) - added by Antoine Martin 10 months ago.
patch for building latest python-lz4 with msvc

Download all attachments as: .zip

Change History (11)

comment:1 Changed 2 years ago by Antoine Martin

Status: newassigned
  • r9564 allows us to lower the packet size threshold above which we try to compress the data using the env var XPRA_MIN_COMPRESS_SIZE=NNNN
  • r9565 enables compressHC for level 9

The fast modes will have to wait until the python-lz4 bindings are updated with the upstream C lz4 release 129 or later, I have made a ticket: please update to r130

These changes are not useful at present: we only ever use the level 9 specified on the command line for the initial hello packet and response, as this is the only "plain" big packet we send, every other "big packets" contain compressed pixels or sound which we compress separately. And for rgb+compress, the level is capped at 5.

When we do add support for "fast", we should update the logic that calculates the compress level for rgb so that we use a slightly higher level. And maybe this would be a good time to try to also honour the compress level specified on the command line: we have logic to dynamically choose the level based on speed, this could be capped or tuned using the command line value.

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:2 Changed 2 years ago by Antoine Martin

Sent a pull request for the changes needed to support compress_fast: https://github.com/steeve/python-lz4/pull/37, ticket: https://github.com/steeve/python-lz4/issues/36

r9674 allows us to support compress fast if available.

It will be quite difficult to test until the new version is released...

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:3 Changed 2 years ago by Antoine Martin

Much better pull request: https://github.com/steeve/python-lz4/pull/41

If nothing happens, I think I will just make a tarball here so we can push it to the beta repo for testing.

Last edited 2 years ago by Antoine Martin (previous) (diff)

Changed 2 years ago by Antoine Martin

Attachment: python-lz4-0.8.0.rc1.tar.xz added

0.8.0.rc1 snapshot

comment:4 Changed 2 years ago by Antoine Martin

Owner: changed from Antoine Martin to alas
Status: assignednew

r9910 bumps the build to use the snapshot of the source taken from https://github.com/jonathanunderwood/python-lz4.

The beta area now has python-lz4 0.8.0.rc1 packages for Fedora.

Supporting centos is going to be more difficult: the new package builds against lz4-devel, which is only available in Fedora...
EPEL has it, but we don't want to require extra repository dependencies. So we would have to include lz4 in our repo.

And we're not going to bother updating win32 or osx just yet, as they're mostly used as clients and compression speed on the client side is never a bottleneck.

Now, the version stuff looks like this:

>>> import lz4
>>> lz4.LZ4_VERSION
'r130'
>>> lz4.VERSION
'0.8.0'
>>> lz4.__version__
'0.8.0'

r9911 now also checks the underlying liblz4 version (requires r119 or later).


@afarr: ready for testing.

  • install the update python-lz4 and you should see:
    $ xpra info | grep network.lz4
    network.lz4=True
    network.lz4.version=0.8.0-r130
    
  • it would be good to benchmark python-lz4 0.7.0 vs python-lz4 0.8.0 in rgb mode, looking at the pixel throughput performance and bandwidth consumption, to ensure the new compression settings are not too aggressive (which could end up using too much bandwidth). (maybe nick can help?)

See also #960, r11010.

Last edited 21 months ago by Antoine Martin (previous) (diff)

Changed 23 months ago by Antoine Martin

Attachment: python-lz4-win32.patch added

quick and dirty patch for building on win32

comment:5 Changed 21 months ago by J. Max Mena

Apparently we've been using the new version for quite some time now without issue...so I'm gonna go ahead and be confident in saying that it's working.

As for the second point...we're training one of our new employees to start writing Xpra automated test scripts, so we'll get to it "soon" (tm).

comment:6 Changed 21 months ago by alas

Checking session info with a 0.16.0 r11122 server and client (windows v. fedora21) - it looks like we're using lz4 (level 1).

Is there any point at trying out any of the other levels yet? Is there a flag to try?

comment:7 Changed 21 months ago by Antoine Martin

Resolution: fixed
Status: newclosed

The levels are chosen automatically based on the speed setting when encoding pixel data. Regular packets should be encoded using the level specified with -z.

Closing.

comment:8 Changed 21 months ago by Antoine Martin

Note: for building on win32, someone posted cleaner fixes:

Changed 10 months ago by Antoine Martin

Attachment: lz4-msvc-fix.patch added

patch for building latest python-lz4 with msvc

Note: See TracTickets for help on using tickets.