Xpra trac archive

Xpra: Ticket #443: adding lz4 encoding on win32

This ticket is just for reference, here's how one adds lz4 compression to the win32 builds (more info on packet encoding and compression here - lz4 is fast !) since simply doing an scripts\easy_install -Z lz4 sadly fails because of ggc-isms in the build files:

(see below for fixing the -mno-cygwin error)

If you get an "Unrecognized command line option '-mno-cygwin'" compilation error, simply remove all the references to '-mno-cygwin' from "cygwinccompiler.py (usually located in C:\Python27\Lib\distutils\cygwinccompiler.py) - as per this answer: Compiling with cython and mingw produces gcc: error: unrecognized command line option '-mno-cygwin'


Wed, 16 Oct 2013 05:50:32 GMT - Antoine Martin: attachment set

session info dialog showing lz4 enabled on win32 (XP 32 bit VirtualBox? VM)

Wed, 16 Oct 2013 05:53:57 GMT - Antoine Martin: owner, description changed

Please confirm that those steps work.

As of r4506 you should be able to see when the lz4 compression is enabled client side via session info: /raw-attachment/ticket/443/session-info-lz4-win32.png

And as of r4507 via "xpra info" too: 

$ xpra info | grep connection.compression=
client.connection.compression=lz4

Notes:

/raw-attachment/ticket/443/show-packet-encoders-and-compressors.png

Mon, 09 Dec 2013 05:52:50 GMT - Antoine Martin: attachment set

session-info features tab now shows packet encoders and compressors

Tue, 10 Dec 2013 00:02:03 GMT - Smo: status changed; resolution set

Tested instructions for building on win32. Built successfully and installed.

Will reopen if there are any issues.

Wed, 11 Dec 2013 14:44:06 GMT - Antoine Martin:

Note: r4925 adds a simple benchmark test to compare compression speed between lz4 and zlib on its fastest setting (level=1).

The results are very consistent (here on an AMD 945 CPU), and very very good:

(and that's a lower bound - most of the packets are of the type that compresses the fastest)

Wed, 11 Dec 2013 15:42:04 GMT - Antoine Martin:

The benchmark tool also revealed that both compressors were not saving much space, and for many types of smaller packets they were actually making the packet bigger! And also costing us at the other end by adding the decompression cost.

That's because rencode is already very efficient at packing things, and the zlib encoder was designed a long time ago, before we even had picture encodings!

So r4928 skips packet compression for packets smaller than 368 bytes - which is the vast majority of packets (except for picture and cursor packets). Which means that lz4 will only really get used for RGB areas with video encoders, or in RGB mode. See also ticket:471#comment:1

Sat, 27 Jun 2015 17:49:19 GMT - Antoine Martin: status changed; resolution deleted

@smo: for 0.16 onwards, you will need to update your win32 build host with this patch: https://www.xpra.org/trac/browser/xpra/trunk/rpmbuild/lz4_add_version.patch I built it for both Python 2.7.10 and Python 3.4.3 using: compiling using msvc Hopefully, a new python-lz4 will be released so we can just easy-install that instead.

Verify lz4 is still enabled with Network_info.exe or the session info dialog.

Thu, 13 Aug 2015 22:28:42 GMT - Smo:

I was attempting to update my osx build environment again today and noticed the patch from comment:5 is missing now. Did you move this somewhere else?

Fri, 14 Aug 2015 06:02:57 GMT - Antoine Martin:

The patch was removed in r9910 since the specfile is now for 0.8.0rc, you can find the last version of the patch here: browser/xpra/trunk/rpmbuild/lz4_add_version.patch. Feel free to re-add it in the osx modulesets tree.

Fri, 11 Sep 2015 03:18:14 GMT - Smo: owner, status changed

I did re-add this to the moduleset for osx.

When building lz4 0.7.0 with this patch and mingw I run into this issue however. 

src/python-lz4.c:205:51: error: too many decimal points in number
src/python-lz4.c:206:55: error: 'r119' undeclared (first use in this function)
src/python-lz4.c:206:55: note: each undeclared identifier is reported only once
for each function it appears in
error: command 'C:\\MinGW\\bin\\gcc.exe' failed with exit status 1

If we change these 2 lines in python-lz4.c to 

    PyModule_AddStringConstant(module, "VERSION", "0.7.0");
    PyModule_AddStringConstant(module, "LZ4_VERSION", "r119");

Then it compiles just fine.

Not sure what the best way to fix this is or I could make a patch with this fix?

Other than that this is working just fine. Any advice?

Fri, 11 Sep 2015 03:21:12 GMT - Antoine Martin: owner changed

A patch would be nice, but as long as things are documented this is not a huge problem, we shouldn't be rebuild this too often. If you make one, it's worth making one against 0.8.0-rc which is what we will use from now on. You can find the source here: http://xpra.org/src/python-lz4-0.8.0-rc1.tar.xz

Wed, 16 Sep 2015 02:54:27 GMT - Antoine Martin:

See attachment/ticket/878/python-lz4-win32.patch

Thu, 24 Sep 2015 21:28:45 GMT - Smo: attachment set

Thu, 24 Sep 2015 21:31:58 GMT - Smo:

for mingw32 which is what i'm using add new file setup.cfg after extracting which contains 

[build]
compiler = mingw32

Apply the pach python-lz4-8-mingw32.patch and python ./setup.py install afterwards extract .egg in site-packages by hand

Thu, 24 Sep 2015 21:32:14 GMT - Smo: status changed; resolution set

Sat, 26 Sep 2015 08:22:08 GMT - Antoine Martin:

Note: AFAIK, you don't actually need the setup.cfg file, you can just do: 

./setup.py build --compiler=mingw32
./setup.py install

(or -c msvc)

Sat, 23 Jan 2021 04:55:39 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/443