adding lz4 encoding on win32

[Antoine Martin]

This ticket is just for reference, here's how one adds lz4 compression to the win32 builds (more info on packet encoding and compression here - lz4 is fast !) since simply doing an scripts\easy_install -Z lz4 sadly fails because of ggc-isms in the build files:

• download ​lz4
• uncompress the archive
• edit the setup.cfg file to force the build to use the mingw32/gcc compiler:

  [build]
  compiler = mingw32
  

• from a mingw32 shell, run

  C:\Python\Python.exe setup.cfg install
  

(see below for fixing the -mno-cygwin error)

• unzip the egg which has been installed (usually found in "C:\Python27\Lib\site-packages\lz4*.egg") so py2exe packaging will find it
• build the installer with py2exe

---

If you get an "Unrecognized command line option '-mno-cygwin'" compilation error, simply remove all the references to '-mno-cygwin' from "cygwinccompiler.py (usually located in C:\Python27\Lib\distutils\cygwinccompiler.py) - as per this answer: ​Compiling with cython and mingw produces gcc: error: unrecognized command line option '-mno-cygwin'

[Antoine Martin]

session info dialog showing lz4 enabled on win32 (XP 32 bit VirtualBox? VM)

[Antoine Martin]

Please confirm that those steps work.

As of r4506 you should be able to see when the lz4 compression is enabled client side via session info:

And as of r4507 via "xpra info" too:

$ xpra info | grep connection.compression=
client.connection.compression=lz4

Notes:

• lz4 is enabled by default when compression level is set to 1 and both sides support it.
• you can verify that the current client supports lz4 with:

  xpra info | grep client.features.lz4
  

• with r4885 onwards, you can verify that the server supports it with:

  xpra info | grep network.lz4
  

• with r4895 and the new xpra control feature (see #461), one can also change the encoder and compressor at runtime, ie:

  xpra control :10 compression lz4
  

  If lz4 is not present, this will return an error (beware this change takes effect whether the client supports lz4 or not..)
• with r4889, you can also see which packet encoders and compressors are available for both client and server:

[Antoine Martin]

session-info features tab now shows packet encoders and compressors

[Smo]

Tested instructions for building on win32.
Built successfully and installed.

Will reopen if there are any issues.

[Antoine Martin]

Note: r4925 adds a simple benchmark test to compare compression speed between lz4 and zlib on its fastest setting (level=1).

The results are very consistent (here on an AMD 945 CPU), and very very good:

• for regular packets:

  average gain of lz4 over zlib: 9.9 times faster
  

• for RGB pixels:

  average gain of lz4 over zlib: 18.8 times faster
  

(and that's a lower bound - most of the packets are of the type that compresses the fastest)

[Antoine Martin]

The benchmark tool also revealed that both compressors were not saving much space, and for many types of smaller packets they were actually making the packet bigger! And also costing us at the other end by adding the decompression cost.

That's because rencode is already very efficient at packing things, and the zlib encoder was designed a long time ago, before we even had picture encodings!

So r4928 skips packet compression for packets smaller than 368 bytes - which is the vast majority of packets (except for picture and cursor packets).
Which means that lz4 will only really get used for RGB areas with video encoders, or in RGB mode. See also ticket:471#comment:1

[Antoine Martin]

@smo: for 0.16 onwards, you will need to update your win32 build host with this patch: ​https://www.xpra.org/trac/browser/xpra/trunk/rpmbuild/lz4_add_version.patch
I built it for both Python 2.7.10 and Python 3.4.3 using: ​compiling using msvc
Hopefully, a new python-lz4 will be released so we can just easy-install that instead.

Verify lz4 is still enabled with Network_info.exe or the session info dialog.

[Smo]

I was attempting to update my osx build environment again today and noticed the patch from comment:5 is missing now. Did you move this somewhere else?

[Antoine Martin]

The patch was removed in r9910 since the specfile is now for 0.8.0rc, you can find the last version of the patch here: browser/xpra/trunk/rpmbuild/lz4_add_version.patch. Feel free to re-add it in the osx modulesets tree.

[Smo]

I did re-add this to the moduleset for osx.

When building lz4 0.7.0 with this patch and mingw I run into this issue however.

src/python-lz4.c:205:51: error: too many decimal points in number
src/python-lz4.c:206:55: error: 'r119' undeclared (first use in this function)
src/python-lz4.c:206:55: note: each undeclared identifier is reported only once
for each function it appears in
error: command 'C:\\MinGW\\bin\\gcc.exe' failed with exit status 1

If we change these 2 lines in python-lz4.c to

    PyModule_AddStringConstant(module, "VERSION", "0.7.0");
    PyModule_AddStringConstant(module, "LZ4_VERSION", "r119");

Then it compiles just fine.

Not sure what the best way to fix this is or I could make a patch with this fix?

Other than that this is working just fine. Any advice?

[Antoine Martin]

A patch would be nice, but as long as things are documented this is not a huge problem, we shouldn't be rebuild this too often.
If you make one, it's worth making one against 0.8.0-rc which is what we will use from now on. You can find the source here: ​http://xpra.org/src/python-lz4-0.8.0-rc1.tar.xz

[Antoine Martin]

See attachment/ticket/878/python-lz4-win32.patch

[Smo]

for mingw32 which is what i'm using add new file setup.cfg after extracting which contains

[build]
compiler = mingw32

Apply the pach python-lz4-8-mingw32.patch and python ./setup.py install afterwards extract .egg in site-packages by hand

[Antoine Martin]

Note: AFAIK, you don't actually need the setup.cfg file, you can just do:

./setup.py build --compiler=mingw32
./setup.py install

(or -c msvc)

[migration script]

this ticket has been moved to: ​https://github.com/Xpra-org/xpra/issues/443