lz4: support HC and fast modes

[Antoine Martin]

• compressHC is already available in 0.7.0, so we can support this already - it is not very useful (almost as slow as zlib), but I guess we can enable it for compress=9
• LZ4_compress_fast is in trunk only, so this would require a runtime check

The mapping should go something like:

• 0 = no compression
• 1 = fast 50
• 2 = fast 17
• 3 = fast 5
• 4 to 8 = standard
• 9 = HC

[Antoine Martin]

• r9564 allows us to lower the packet size threshold above which we try to compress the data using the env var XPRA_MIN_COMPRESS_SIZE=NNNN
• r9565 enables compressHC for level 9

The fast modes will have to wait until the ​python-lz4 bindings are updated with the upstream C lz4 release 129 or later, I have made a ticket: ​please update to r130

These changes are not useful at present: we only ever use the level 9 specified on the command line for the initial hello packet and response, as this is the only "plain" big packet we send, every other "big packets" contain compressed pixels or sound which we compress separately. And for rgb+compress, the level is capped at 5.

When we do add support for "fast", we should update the logic that calculates the compress level for rgb so that we use a slightly higher level. And maybe this would be a good time to try to also honour the compress level specified on the command line: we have logic to dynamically choose the level based on speed, this could be capped or tuned using the command line value.

[Antoine Martin]

Sent a pull request for the changes needed to support compress_fast: ​https://github.com/steeve/python-lz4/pull/37, ticket: ​https://github.com/steeve/python-lz4/issues/36

r9674 allows us to support compress fast if available.

It will be quite difficult to test until the new version is released...

[Antoine Martin]

Much better pull request: ​https://github.com/steeve/python-lz4/pull/41

If nothing happens, I think I will just make a tarball here so we can push it to the beta repo for testing.

[Antoine Martin]

0.8.0.rc1 snapshot

[Antoine Martin]

r9910 bumps the build to use the snapshot of the source taken from ​https://github.com/jonathanunderwood/python-lz4.

The ​beta area now has python-lz4 0.8.0.rc1 packages for Fedora.

Supporting centos is going to be more difficult: the new package builds against lz4-devel, which is only available in Fedora...
​EPEL has it, but we don't want to require extra repository dependencies. So we would have to include lz4 in our repo.

And we're not going to bother updating win32 or osx just yet, as they're mostly used as clients and compression speed on the client side is never a bottleneck.

Now, the version stuff looks like this:

>>> import lz4
>>> lz4.LZ4_VERSION
'r130'
>>> lz4.VERSION
'0.8.0'
>>> lz4.__version__
'0.8.0'

r9911 now also checks the underlying liblz4 version (requires r119 or later).

---

@afarr: ready for testing.

• install the update python-lz4 and you should see:

  $ xpra info | grep network.lz4
  network.lz4=True
  network.lz4.version=0.8.0-r130
  

• it would be good to benchmark python-lz4 0.7.0 vs python-lz4 0.8.0 in rgb mode, looking at the pixel throughput performance and bandwidth consumption, to ensure the new compression settings are not too aggressive (which could end up using too much bandwidth). (maybe nick can help?)

---

See also #960, r11010.

[Antoine Martin]

quick and dirty patch for building on win32

[J. Max Mena]

Apparently we've been using the new version for quite some time now without issue...so I'm gonna go ahead and be confident in saying that it's working.

As for the second point...we're training one of our new employees to start writing Xpra automated test scripts, so we'll get to it "soon" (tm).

[alas]

Checking session info with a 0.16.0 r11122 server and client (windows v. fedora21) - it looks like we're using lz4 (level 1).

Is there any point at trying out any of the other levels yet? Is there a flag to try?

[Antoine Martin]

The levels are chosen automatically based on the speed setting when encoding pixel data. Regular packets should be encoded using the level specified with -z.

Closing.

[Antoine Martin]

Note: for building on win32, someone posted cleaner fixes:

• ​Fix compilation for vs2008
• ​Avoid directly including stdint.h

[Antoine Martin]

patch for building latest python-lz4 with msvc

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/878