xpra icon
Bug tracker and wiki

Opened 6 years ago

Closed 4 years ago

Last modified 4 years ago

#443 closed task (fixed)

adding lz4 encoding on win32

Reported by: Antoine Martin Owned by: Smo
Priority: major Milestone: 0.11
Component: platforms Version:
Keywords: Cc:

Description (last modified by Antoine Martin)

This ticket is just for reference, here's how one adds lz4 compression to the win32 builds (more info on packet encoding and compression here - lz4 is fast !) since simply doing an scripts\easy_install -Z lz4 sadly fails because of ggc-isms in the build files:

  • download lz4
  • uncompress the archive
  • edit the setup.cfg file to force the build to use the mingw32/gcc compiler:
    [build]
    compiler = mingw32
    
  • from a mingw32 shell, run
    C:\Python\Python.exe setup.cfg install
    

(see below for fixing the -mno-cygwin error)

  • unzip the egg which has been installed (usually found in "C:\Python27\Lib\site-packages\lz4*.egg") so py2exe packaging will find it
  • build the installer with py2exe

If you get an "Unrecognized command line option '-mno-cygwin'" compilation error, simply remove all the references to '-mno-cygwin' from "cygwinccompiler.py (usually located in C:\Python27\Lib\distutils\cygwinccompiler.py) - as per this answer: Compiling with cython and mingw produces gcc: error: unrecognized command line option '-mno-cygwin'

Attachments (3)

session-info-lz4-win32.png (27.4 KB) - added by Antoine Martin 6 years ago.
session info dialog showing lz4 enabled on win32 (XP 32 bit VirtualBox? VM)
show-packet-encoders-and-compressors.png (57.8 KB) - added by Antoine Martin 6 years ago.
session-info features tab now shows packet encoders and compressors
python-lz4-8-mingw32.patch (574 bytes) - added by Smo 4 years ago.

Download all attachments as: .zip

Change History (16)

Changed 6 years ago by Antoine Martin

Attachment: session-info-lz4-win32.png added

session info dialog showing lz4 enabled on win32 (XP 32 bit VirtualBox? VM)

comment:1 Changed 6 years ago by Antoine Martin

Description: modified (diff)
Owner: changed from Antoine Martin to Smo

Please confirm that those steps work.

As of r4506 you should be able to see when the lz4 compression is enabled client side via session info:
/raw-attachment/ticket/443/session-info-lz4-win32.png

And as of r4507 via "xpra info" too:

$ xpra info | grep connection.compression=
client.connection.compression=lz4

Notes:

  • lz4 is enabled by default when compression level is set to 1 and both sides support it.
  • you can verify that the current client supports lz4 with:
    xpra info | grep client.features.lz4
    
  • with r4885 onwards, you can verify that the server supports it with:
    xpra info | grep network.lz4
    
  • with r4895 and the new xpra control feature (see #461), one can also change the encoder and compressor at runtime, ie:
    xpra control :10 compression lz4
    
    If lz4 is not present, this will return an error (beware this change takes effect whether the client supports lz4 or not..)
  • with r4889, you can also see which packet encoders and compressors are available for both client and server:

/raw-attachment/ticket/443/show-packet-encoders-and-compressors.png

Last edited 6 years ago by Antoine Martin (previous) (diff)

Changed 6 years ago by Antoine Martin

session-info features tab now shows packet encoders and compressors

comment:2 Changed 6 years ago by Smo

Resolution: fixed
Status: newclosed

Tested instructions for building on win32.
Built successfully and installed.

Will reopen if there are any issues.

comment:3 Changed 6 years ago by Antoine Martin

Note: r4925 adds a simple benchmark test to compare compression speed between lz4 and zlib on its fastest setting (level=1).

The results are very consistent (here on an AMD 945 CPU), and very very good:

  • for regular packets:
    average gain of lz4 over zlib: 9.9 times faster
    
  • for RGB pixels:
    average gain of lz4 over zlib: 18.8 times faster
    

(and that's a lower bound - most of the packets are of the type that compresses the fastest)

Last edited 6 years ago by Antoine Martin (previous) (diff)

comment:4 Changed 6 years ago by Antoine Martin

The benchmark tool also revealed that both compressors were not saving much space, and for many types of smaller packets they were actually making the packet bigger! And also costing us at the other end by adding the decompression cost.

That's because rencode is already very efficient at packing things, and the zlib encoder was designed a long time ago, before we even had picture encodings!

So r4928 skips packet compression for packets smaller than 368 bytes - which is the vast majority of packets (except for picture and cursor packets).
Which means that lz4 will only really get used for RGB areas with video encoders, or in RGB mode. See also ticket:471#comment:1

Last edited 6 years ago by Antoine Martin (previous) (diff)

comment:5 Changed 4 years ago by Antoine Martin

Resolution: fixed
Status: closedreopened

@smo: for 0.16 onwards, you will need to update your win32 build host with this patch: https://www.xpra.org/trac/browser/xpra/trunk/rpmbuild/lz4_add_version.patch
I built it for both Python 2.7.10 and Python 3.4.3 using: compiling using msvc
Hopefully, a new python-lz4 will be released so we can just easy-install that instead.

Verify lz4 is still enabled with Network_info.exe or the session info dialog.

Last edited 4 years ago by Antoine Martin (previous) (diff)

comment:6 Changed 4 years ago by Smo

I was attempting to update my osx build environment again today and noticed the patch from comment:5 is missing now. Did you move this somewhere else?

comment:7 Changed 4 years ago by Antoine Martin

The patch was removed in r9910 since the specfile is now for 0.8.0rc, you can find the last version of the patch here: browser/xpra/trunk/rpmbuild/lz4_add_version.patch. Feel free to re-add it in the osx modulesets tree.

comment:8 Changed 4 years ago by Smo

Owner: changed from Smo to Antoine Martin
Status: reopenednew

I did re-add this to the moduleset for osx.

When building lz4 0.7.0 with this patch and mingw I run into this issue however.

src/python-lz4.c:205:51: error: too many decimal points in number
src/python-lz4.c:206:55: error: 'r119' undeclared (first use in this function)
src/python-lz4.c:206:55: note: each undeclared identifier is reported only once
for each function it appears in
error: command 'C:\\MinGW\\bin\\gcc.exe' failed with exit status 1

If we change these 2 lines in python-lz4.c to

    PyModule_AddStringConstant(module, "VERSION", "0.7.0");
    PyModule_AddStringConstant(module, "LZ4_VERSION", "r119");

Then it compiles just fine.

Not sure what the best way to fix this is or I could make a patch with this fix?

Other than that this is working just fine. Any advice?

comment:9 Changed 4 years ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

A patch would be nice, but as long as things are documented this is not a huge problem, we shouldn't be rebuild this too often.
If you make one, it's worth making one against 0.8.0-rc which is what we will use from now on. You can find the source here: http://xpra.org/src/python-lz4-0.8.0-rc1.tar.xz

Changed 4 years ago by Smo

Attachment: python-lz4-8-mingw32.patch added

comment:11 Changed 4 years ago by Smo

for mingw32 which is what i'm using add new file setup.cfg after extracting which contains

[build]
compiler = mingw32

Apply the pach python-lz4-8-mingw32.patch and python ./setup.py install afterwards extract .egg in site-packages by hand

comment:12 Changed 4 years ago by Smo

Resolution: fixed
Status: newclosed

comment:13 Changed 4 years ago by Antoine Martin

Note: AFAIK, you don't actually need the setup.cfg file, you can just do:

./setup.py build --compiler=mingw32
./setup.py install

(or -c msvc)

Last edited 4 years ago by Antoine Martin (previous) (diff)
Note: See TracTickets for help on using tickets.