Xpra: Ticket #876: support python-cryptography as well as pycrypto

pycrypto has completely stalled, this ones looks way better: https://cryptography.io/



Fri, 13 Nov 2015 05:36:10 GMT - Antoine Martin: priority, status changed

Raising as this will give us hardware acceleration, see #1029. See also #198


Wed, 06 Jan 2016 17:38:55 GMT - Antoine Martin:

r11607 adds a "python-cryptography" backend and the ability to switch between the backends using an env var, some unit tests, etc. It still defaults to "pycrypto" for now, but the performance improvements are noticeable (running from the unit tests):

pycrypto took 165ms
python-cryptography took 94ms

(that's encrypting + decrypting 64KB 20 times)

Still TODO:


Fri, 08 Jan 2016 10:24:23 GMT - Antoine Martin: owner, status changed

r11615 + r11616 makes it easier to compare different payload sizes and encryption vs decryption performance. Summary: python-cryptography always wins, sometimes by a huge margin: it isn't just faster overall, it has much lower latency too. This is such a big win that I will seriously consider backporting support to older branches.

I've measured the cipher initialization cost, and that's not the source of the problem: pycrypto just has a much higher call overhead.

Some results on a cheap and old i3 CPU, shown as pairs of elapsed time per iteration in milliseconds: pycrypto first, then python-cryptography.

Payload size: 16B 64B 64KB 1MB
Encryption 5.1 0.2 5.0 0.2 15.2 6.5 169 101
Decryption 4.6 0.2 4.7 0.3 92.5 82.4 1482 1391
Both 2.5 0.1 2.7 0.2 55.4 47.9 883 811

The ~5ms encryption overhead on a 16 Byte payload is very significant. With bigger payload sizes, the win narrows to about 10%. I'm not sure why they're both so slow at decryption: about 10 times slower than encryption with 1MB of data! (could be large output buffer re-allocation?) The wins could be even more significant on platforms with hardware acceleration (see #876).

@afarr: you may want to run this unit test on some Xeons to see how they fare. And maybe we'll stick this in the wiki somewhere as a justification for switching to python-cryptography.


Sat, 09 Jan 2016 12:07:02 GMT - Antoine Martin: priority changed

Ran the tests again on an Opteron system in order to verify that hardware acceleration is used (#1029) - it is. The gains are HUGE. On 64KB packets, more than 20 times faster!

Raising.


Sun, 10 Jan 2016 14:33:05 GMT - Antoine Martin:

For the build platform and packaging support:


Extra changesets needed for a potential backport: r11639, r11636, r11629 These changes are needed for client side support, but could be omitted if we do just a simple server-only backport: r11640, r11637,


Sun, 10 Jan 2016 15:08:22 GMT - Antoine Martin: attachment set

almost complete patch for building openssl on osx as part of the moduleset


Tue, 26 Jan 2016 01:27:34 GMT - Antoine Martin: owner changed

@smo: can you help me with this one? I believe that there will be some changes needed to packaging after this is done, on win32 we are missing lots of components (all of openssl..) giving us the stacktraces recorded in #1029


Tue, 26 Jan 2016 01:56:33 GMT - alas:

Not sure if this should be posted here or in #1029 - but when trying to test osx client 0.17.0 r11761 (for opus support) — I'm getting the following Error message client side (despite not using any encryption):

Schadenfreude:MacOS Schadenfreude$ ./xpra attach  --opengl=on --speaker-codec=opus
2016-01-25 17:45:51,574 Error: encryption library python-cryptography failed validation!
2016-01-25 17:45:51,574  sha1 is not supported for PBKDF2 by this backend.
2016-01-25 17:45:51,615 Xpra gtk2 client version 0.17.0-r11761

Tue, 26 Jan 2016 02:28:56 GMT - Antoine Martin:

https://github.com/pyca/cryptography/issues/2039 has some packaging workarounds. As per ticket:1029#comment:15, the warnings will remain until this is fixed.


Fri, 29 Jan 2016 04:44:34 GMT - Antoine Martin:

@smo: can you please take a look?


Fri, 05 Feb 2016 13:27:50 GMT - Smo:

Instructions for win32 worked with no issue. I got my precompiled binaries from this site https://slproweb.com/products/Win32OpenSSL.html not sure what it is that you used.

Still working on osx as this is my issue no matter what I try I always end up with -arch x86_64 and -arch i386 when invoking gcc so I will continue to look for a solution. Just for the record this is the type of error I see.

making all in crypto/objects...
ar  r ../../libcrypto.a o_names.o obj_dat.o obj_lib.o obj_err.o obj_xref.o
ar: ../../libcrypto.a is a fat file (use libtool(1) or lipo(1) and ar(1) on it)
ar: ../../libcrypto.a: Inappropriate file type or format
make[2]: *** [lib] Error 1
make[1]: *** [subdirs] Error 1
make: *** [build_crypto] Error 1

It's worth noting that i'm trying to do this on a 64 bit osx system running el capitan

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.1
BuildVersion:   15B42

Fri, 05 Feb 2016 13:35:29 GMT - Smo:

r11859 adds python-ipaddress to jhbuild moduleset


Fri, 05 Feb 2016 13:38:36 GMT - Smo:

r11860 adds python-idna to jhbuild moduleset


Sat, 06 Feb 2016 08:53:48 GMT - Antoine Martin: summary changed

Almost complete as of r11865:

the moduleset does that automatically for me with the changes I made - you may need to tweak it for 64-bit builds.. Two things still need fixing for openssl:

With all this in place, I now have osx builds with python-cryptography (bumped to 1.2.2 in r11866, updated my win32 VM with the same version and openssl 1.0.2f)

@afarr: this is ready for testing.

(I have changed the ticket title to better reflect what has been done: we may deprecate and eventually drop pycrypto, but not just yet)


Mon, 08 Feb 2016 20:55:49 GMT - alas:

Still waiting on an osx client package... but ran a quick test (reminding myself the syntax from 1029#comment:6) with 0.17.0 r11886 win32 client against 0.17.0 r11888 fedora 23 server.

With the acceleration, I got the following numbers (seem a little better than those mentioned in #1029):

[jimador@jimador net]$ python crypto_test.py
.Encryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.5ms:                3KB/s
python-cryptography              took   0.2ms:               82KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.5ms:              223KB/s
python-cryptography              took   0.2ms:             5055KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  13.9ms:            73698KB/s
python-cryptography              took   6.5ms:           158749KB/s
Decryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.0ms:                3KB/s
python-cryptography              took   0.2ms:               83KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.2ms:              240KB/s
python-cryptography              took   0.3ms:             3794KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  83.2ms:            12304KB/s
python-cryptography              took  76.8ms:            13328KB/s
Global Performance:
test_perf: size: 16 Bytes
pycrypto                         took   2.2ms:                6KB/s
python-cryptography              took   0.1ms:              138KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   2.3ms:              441KB/s
python-cryptography              took   0.1ms:             6939KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  49.4ms:            20739KB/s
python-cryptography              took  44.6ms:            22936KB/s
.
----------------------------------------------------------------------
Ran 2 tests in 4.461s
OK

With acceleration turned off:

[jimador@jimador net]$ OPENSSL_ia32cap="~0x200000200000000" python crypto_test.py
.Encryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.5ms:                3KB/s
python-cryptography              took   0.2ms:               80KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.4ms:              225KB/s
python-cryptography              took   0.2ms:             5149KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  13.6ms:            75115KB/s
python-cryptography              took   6.4ms:           159003KB/s
Decryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.1ms:                3KB/s
python-cryptography              took   0.2ms:               86KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.2ms:              239KB/s
python-cryptography              took   0.2ms:             4007KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  84.6ms:            12104KB/s
python-cryptography              took  76.1ms:            13464KB/s
Global Performance:
test_perf: size: 16 Bytes
pycrypto                         took   2.2ms:                7KB/s
python-cryptography              took   0.1ms:              150KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   2.3ms:              437KB/s
python-cryptography              took   0.1ms:             6924KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  49.6ms:            20648KB/s
python-cryptography              took  70.7ms:            14490KB/s
.
----------------------------------------------------------------------
Ran 2 tests in 4.985s
OK

Except for the last entry though, there doesn't seem to be much difference when using a VM as a server.

I guess I'll have to check into what the KVM is doing with the filtering after all.


Mon, 08 Feb 2016 21:02:01 GMT - Antoine Martin:

@afarr: your (identical) numbers show that you are still NOT using any hardware acceleration.


Fri, 12 Feb 2016 19:06:09 GMT - J. Max Mena:

Ran the tests on a hardware machine (mid range i5 4460) in Fedora 23 with trunk r11908:

[root@vorfuehreffekt-spikes-eng net]# python crypto_test.py
.Encryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   2.3ms:                6KB/s
python-cryptography              took   0.1ms:              123KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   2.3ms:              433KB/s
python-cryptography              took   0.1ms:             7936KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took   8.9ms:           115468KB/s
python-cryptography              took   2.3ms:           440013KB/s
Decryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   2.1ms:                7KB/s
python-cryptography              took   0.1ms:              127KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   2.2ms:              461KB/s
python-cryptography              took   0.2ms:             6235KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  57.7ms:            17757KB/s
python-cryptography              took  49.9ms:            20505KB/s
Global Performance:
test_perf: size: 16 Bytes
pycrypto                         took   1.2ms:               13KB/s
python-cryptography              took   0.1ms:              232KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   1.2ms:              833KB/s
python-cryptography              took   0.1ms:            10438KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  34.7ms:            29470KB/s
python-cryptography              took  28.3ms:            36156KB/s
.
----------------------------------------------------------------------
Ran 2 tests in 2.891s
OK

and without acceleration:

[root@vorfuehreffekt-spikes-eng net]# OPENSSL_ia32cap="~0x200000200000000" python crypto_test.py
.Encryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   2.3ms:                6KB/s
python-cryptography              took   0.1ms:              121KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   2.3ms:              429KB/s
python-cryptography              took   0.1ms:             7621KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took   8.8ms:           115707KB/s
python-cryptography              took   4.5ms:           225109KB/s
Decryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   2.1ms:                7KB/s
python-cryptography              took   0.1ms:              125KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   2.2ms:              459KB/s
python-cryptography              took   0.2ms:             5684KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  57.5ms:            17813KB/s
python-cryptography              took  52.2ms:            19616KB/s
Global Performance:
test_perf: size: 16 Bytes
pycrypto                         took   1.1ms:               13KB/s
python-cryptography              took   0.1ms:              197KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   1.2ms:              829KB/s
python-cryptography              took   0.1ms:            10266KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  34.5ms:            29716KB/s
python-cryptography              took  30.8ms:            33215KB/s
.
----------------------------------------------------------------------
Ran 2 tests in 2.981s
OK

I'm also getting similar results from another Fedora 23 machine with and without the acceleration. I have a feeling that either the acceleration isn't enabled, or we're not disabling it properly since I'm getting such similar numbers across multiple machines.

I also did some research and the CPU on both computers (identical CPUs) and that specific model supports AES encryption and decryption hardware acceleration.


Sat, 13 Feb 2016 03:04:46 GMT - Antoine Martin:

I have a feeling that either the acceleration isn't enabled, or we're not disabling it properly since I'm getting such similar numbers across multiple machines.


You need to bear in mind that:

Bar issues with KVM and cpu flags, I think we can close this ticket.


Tue, 16 Feb 2016 20:33:40 GMT - alas: status changed; resolution set

Well, everything looks good except our getting the KVM flags sorted out.

I'll close this and just open another if, once that gets sorted out, I see an issue.


Sat, 24 Sep 2016 05:53:25 GMT - Antoine Martin: attachment set

fix win32 build against openssl 1.1


Sat, 28 Jan 2017 09:41:39 GMT - Antoine Martin:

Note: in version 2.0, we are dropping support for pycrypto - that project looks dead: no releases in almost 3 years.


Sat, 23 Jan 2021 05:08:34 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/876