#443 closed task (fixed)
adding lz4 encoding on win32
Reported by: | Antoine Martin | Owned by: | Smo |
---|---|---|---|
Priority: | major | Milestone: | 0.11 |
Component: | platforms | Version: | |
Keywords: | Cc: |
Description (last modified by )
This ticket is just for reference, here's how one adds lz4
compression to the win32 builds (more info on packet encoding and compression here - lz4 is fast !) since simply doing an scripts\easy_install -Z lz4
sadly fails because of ggc-isms in the build files:
- download lz4
- uncompress the archive
- edit the
setup.cfg
file to force the build to use the mingw32/gcc compiler:[build] compiler = mingw32
- from a mingw32 shell, run
C:\Python\Python.exe setup.cfg install
(see below for fixing the -mno-cygwin
error)
- unzip the egg which has been installed (usually found in "
C:\Python27\Lib\site-packages\lz4*.egg
") so py2exe packaging will find it - build the installer with py2exe
If you get an "Unrecognized command line option '-mno-cygwin'
" compilation error, simply remove all the references to '-mno-cygwin'
from "cygwinccompiler.py
(usually located in C:\Python27\Lib\distutils\cygwinccompiler.py
) - as per this answer: Compiling with cython and mingw produces gcc: error: unrecognized command line option '-mno-cygwin'
Attachments (3)
Change History (17)
Changed 7 years ago by
Attachment: | session-info-lz4-win32.png added |
---|
comment:1 Changed 7 years ago by
Description: | modified (diff) |
---|---|
Owner: | changed from Antoine Martin to Smo |
Please confirm that those steps work.
As of r4506 you should be able to see when the lz4 compression is enabled client side via session info:
And as of r4507 via "xpra info" too:
$ xpra info | grep connection.compression= client.connection.compression=lz4
Notes:
- lz4 is enabled by default when compression level is set to
1
and both sides support it. - you can verify that the current client supports lz4 with:
xpra info | grep client.features.lz4
- with r4885 onwards, you can verify that the server supports it with:
xpra info | grep network.lz4
- with r4895 and the new
xpra control
feature (see #461), one can also change the encoder and compressor at runtime, ie:xpra control :10 compression lz4
If lz4 is not present, this will return an error (beware this change takes effect whether the client supports lz4 or not..) - with r4889, you can also see which packet encoders and compressors are available for both client and server:
Changed 7 years ago by
Attachment: | show-packet-encoders-and-compressors.png added |
---|
session-info features tab now shows packet encoders and compressors
comment:2 Changed 7 years ago by
Resolution: | → fixed |
---|---|
Status: | new → closed |
Tested instructions for building on win32.
Built successfully and installed.
Will reopen if there are any issues.
comment:3 Changed 7 years ago by
Note: r4925 adds a simple benchmark test to compare compression speed between lz4 and zlib on its fastest setting (level=1).
The results are very consistent (here on an AMD 945 CPU), and very very good:
- for regular packets:
average gain of lz4 over zlib: 9.9 times faster
- for RGB pixels:
average gain of lz4 over zlib: 18.8 times faster
(and that's a lower bound - most of the packets are of the type that compresses the fastest)
comment:4 Changed 7 years ago by
The benchmark tool also revealed that both compressors were not saving much space, and for many types of smaller packets they were actually making the packet bigger! And also costing us at the other end by adding the decompression cost.
That's because rencode is already very efficient at packing things, and the zlib encoder was designed a long time ago, before we even had picture encodings!
So r4928 skips packet compression for packets smaller than 368 bytes - which is the vast majority of packets (except for picture and cursor packets).
Which means that lz4 will only really get used for RGB areas with video encoders, or in RGB mode. See also ticket:471#comment:1
comment:5 Changed 6 years ago by
Resolution: | fixed |
---|---|
Status: | closed → reopened |
@smo: for 0.16 onwards, you will need to update your win32 build host with this patch: https://www.xpra.org/trac/browser/xpra/trunk/rpmbuild/lz4_add_version.patch
I built it for both Python 2.7.10 and Python 3.4.3 using: compiling using msvc
Hopefully, a new python-lz4 will be released so we can just easy-install that instead.
Verify lz4 is still enabled with Network_info.exe
or the session info dialog.
comment:6 Changed 6 years ago by
I was attempting to update my osx build environment again today and noticed the patch from comment:5 is missing now. Did you move this somewhere else?
comment:7 Changed 6 years ago by
The patch was removed in r9910 since the specfile is now for 0.8.0rc, you can find the last version of the patch here: browser/xpra/trunk/rpmbuild/lz4_add_version.patch. Feel free to re-add it in the osx modulesets tree.
comment:8 Changed 5 years ago by
Owner: | changed from Smo to Antoine Martin |
---|---|
Status: | reopened → new |
I did re-add this to the moduleset for osx.
When building lz4 0.7.0 with this patch and mingw I run into this issue however.
src/python-lz4.c:205:51: error: too many decimal points in number src/python-lz4.c:206:55: error: 'r119' undeclared (first use in this function) src/python-lz4.c:206:55: note: each undeclared identifier is reported only once for each function it appears in error: command 'C:\\MinGW\\bin\\gcc.exe' failed with exit status 1
If we change these 2 lines in python-lz4.c to
PyModule_AddStringConstant(module, "VERSION", "0.7.0"); PyModule_AddStringConstant(module, "LZ4_VERSION", "r119");
Then it compiles just fine.
Not sure what the best way to fix this is or I could make a patch with this fix?
Other than that this is working just fine. Any advice?
comment:9 Changed 5 years ago by
Owner: | changed from Antoine Martin to Smo |
---|
A patch would be nice, but as long as things are documented this is not a huge problem, we shouldn't be rebuild this too often.
If you make one, it's worth making one against 0.8.0-rc which is what we will use from now on. You can find the source here: http://xpra.org/src/python-lz4-0.8.0-rc1.tar.xz
Changed 5 years ago by
Attachment: | python-lz4-8-mingw32.patch added |
---|
comment:11 Changed 5 years ago by
for mingw32 which is what i'm using add new file setup.cfg
after extracting which contains
[build] compiler = mingw32
Apply the pach python-lz4-8-mingw32.patch and python ./setup.py install
afterwards extract .egg in site-packages by hand
comment:12 Changed 5 years ago by
Resolution: | → fixed |
---|---|
Status: | new → closed |
comment:13 Changed 5 years ago by
Note: AFAIK, you don't actually need the setup.cfg file, you can just do:
./setup.py build --compiler=mingw32 ./setup.py install
(or -c msvc
)
comment:14 Changed 5 weeks ago by
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/443
session info dialog showing lz4 enabled on win32 (XP 32 bit VirtualBox? VM)