Using encryption can be costly (apparently causing a 20 to 30% framerate reduction on video playback), we should be using AES-NI to speed it up.
Great post on AES-NI: The Intel Advanced Encryption Standard (AES) Extensions, including some information on attack vectors..
The current version of pycrypto is very stale (June 2014 with no updates scheduled), but there is some code to support it in git: problems with this implementation though: at least a bug, and maybe some licensing issues too. I think we should move to Crypto++): #876.
Alternatives:
See also #198
#876 should give us hardware acceleration for free. It is a little bit difficult to verify that hardware acceleration is enabled as this is not exposed through the wrapper...
Note: for those desperate to get more performance, consider using newer openssl versions: switching from v1.0.1 to v1.0.2 can give you 10 to 50% gains: https://github.com/nodejs/node/wiki/Crypto-Performance-Notes-for-OpenSSL-1.0.2a-on-iojs-v1.8.0
Here's how you check that your openssl builds supports the CPU acceleration found in the CPU (if any!): How can I check if OpenSSL is support/use the Intel AES-NI?.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes without 158670.12k 171487.51k 179068.59k 181488.30k 181723.14k with-accel 417907.64k 515590.31k 590289.83k 611770.37k 617321.81k
This system shows both "avx" and "aes" in /proc/cpuinfo. (without-accel means running with OPENSSL_ia32cap="~0x200000200000000"
, with-accel means running with it unset - which is not the same as setting it to an empty string!)
My guess is that "avx" is probably always used if available.
So, sticking to the systems with aes acceleration for further tests, (both running against openssl 1.0.2e).
I can then compare with and without hardware acceleration from our python code by running our unit test (see #876) both with and without:
PYTHONPATH=. python ./tests/unit/net/crypto_test.py OPENSSL_ia32cap="~0x200000200000000" PYTHONPATH=. python ./tests/unit/net/crypto_test.py
And although the difference is not noticeable at all on small packets (1KB and lower) since we are mostly measuring the overheads (python and actual measurements), the results on larger packets (64KB and above) are impressive: the performance is more than doubled. (on top of the huge gains already made by switching to python-cryptography #876)
@afarr: this is mostly a FYI, the work to be done is to switch to python-cryptography as per #876, and unless you are using ancient CPUs or an ancient openssl library, you will get the benefits of hardware acceleration automatically. The only potential problem is if you are using virtual machines that strip some of the cpu flags. Some systems do this for supporting live migration and other exotic features - often a complete waste of good CPU extensions. So you may want to verify both on the host and in the virtual machine, you should get similar results. (the virtual machine overhead should be negligible)
Hmm... I'm running into an issue when I try to run --encryption=AES
on my 0.17.0 r11669 fedora 23 (VM) server.
Running python /home/jimador/xpra-trunk/src/tests/unit/net/crypto_test.py
I get output that looks like I have all the listed ciphers:
[jimador@jimador net]$ python crypto_test.py .Encryption Performance: test_perf: size: 16 Bytes pycrypto took 4.9ms: 3KB/s python-cryptography took 0.2ms: 64KB/s test_perf: size: 1024 Bytes pycrypto took 4.9ms: 203KB/s python-cryptography took 0.2ms: 4901KB/s test_perf: size: 1048576 Bytes pycrypto took 13.7ms: 74790KB/s python-cryptography took 6.6ms: 155833KB/s Decryption Performance: test_perf: size: 16 Bytes pycrypto took 4.0ms: 3KB/s python-cryptography took 0.2ms: 85KB/s test_perf: size: 1024 Bytes pycrypto took 4.1ms: 241KB/s python-cryptography took 0.3ms: 3914KB/s test_perf: size: 1048576 Bytes pycrypto took 86.0ms: 11902KB/s python-cryptography took 79.7ms: 12853KB/s Global Performance: test_perf: size: 16 Bytes pycrypto took 2.3ms: 6KB/s python-cryptography took 0.1ms: 113KB/s test_perf: size: 1024 Bytes pycrypto took 2.3ms: 440KB/s python-cryptography took 0.1ms: 6868KB/s test_perf: size: 1048576 Bytes pycrypto took 50.4ms: 20313KB/s python-cryptography took 45.9ms: 22331KB/s . ---------------------------------------------------------------------- Ran 2 tests in 4.858s OK
But, when I launch with --encryption=AES
, the server crashes with the following traceback:
2016-01-12 16:02:36,328 Error: cannot start the xpra server Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/xpra/scripts/server.py", line 1006, in run_server app.init(opts) File "/usr/lib64/python2.7/site-packages/xpra/x11/server.py", line 196, in init X11ServerBase.init(self, opts) File "/usr/lib64/python2.7/site-packages/xpra/x11/x11_server_base.py", line 81, in init GTKServerBase.init(self, opts) File "/usr/lib64/python2.7/site-packages/xpra/server/server_base.py", line 194, in init ServerCore.init(self, opts) File "/usr/lib64/python2.7/site-packages/xpra/server/server_core.py", line 178, in init validate_encryption(opts) File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 795, in validate_encryption do_validate_encryption(opts.encryption, opts.tcp_encryption, opts.password_file, opts.encryption_keyfile, opts.tcp_encryption_keyfile) File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 801, in do_validate_encryption raise InitException("cannot use encryption: no ciphers available (a crypto library must be installed)") InitException: cannot use encryption: no ciphers available (a crypto library must be installed) 2016-01-12 16:02:36,332 cannot use encryption: no ciphers available (a crypto library must be installed)
The unit test results definitely look promising though.
I get output that looks like I have all the listed ciphers:
You're not telling us what sort of CPU this is.
It would be good to know if hardware acceleration is being used (as per previous comment, using OPENSSL_ia32cap=..
)
But, when I launch with --encryption=AES, the server crashes with the following traceback:
Oops, sorry about that, incomplete work: should be OK with r11680, further improved with r11681 so we don't even load the crypto bits when they aren't being used, in the case of python-cryptography this will save having the openssl library loaded in memory if we aren't using it (ie: when running the sound subprocess, we don't need it or want it)
I would've assumed that there was no hardware acceleration being used on the vm I use as a server... but, as you mention above (comment:3), "without-accel means running with OPENSSL_ia32cap="~0x200000200000000"
, with-accel means running with it unset".
Comparison seems to suggest there's some hardware acceleration happening somewhere.
Running it unset (=with-accel):
[jimador@jimador net]$ python crypto_test.py .Encryption Performance: test_perf: size: 16 Bytes pycrypto took 4.5ms: 3KB/s python-cryptography took 0.2ms: 81KB/s test_perf: size: 1024 Bytes pycrypto took 4.5ms: 222KB/s python-cryptography took 0.2ms: 5027KB/s test_perf: size: 1048576 Bytes pycrypto took 13.8ms: 74343KB/s python-cryptography took 6.6ms: 154726KB/s Decryption Performance: test_perf: size: 16 Bytes pycrypto took 4.1ms: 3KB/s python-cryptography took 0.2ms: 85KB/s test_perf: size: 1024 Bytes pycrypto took 4.2ms: 239KB/s python-cryptography took 0.3ms: 3910KB/s test_perf: size: 1048576 Bytes pycrypto took 87.4ms: 11719KB/s python-cryptography took 80.2ms: 12768KB/s Global Performance: test_perf: size: 16 Bytes pycrypto took 2.4ms: 6KB/s python-cryptography took 0.1ms: 140KB/s test_perf: size: 1024 Bytes pycrypto took 2.4ms: 415KB/s python-cryptography took 0.2ms: 6478KB/s test_perf: size: 1048576 Bytes pycrypto took 56.6ms: 18085KB/s python-cryptography took 48.2ms: 21262KB/s . ---------------------------------------------------------------------- Ran 2 tests in 4.785s OK
Vs. "without accel":
[jimador@jimador net]$ OPENSSL_ia32cap="~0x200000200000000" python crypto_test.py .Encryption Performance: test_perf: size: 16 Bytes pycrypto took 4.9ms: 3KB/s python-cryptography took 0.3ms: 59KB/s test_perf: size: 1024 Bytes pycrypto took 4.9ms: 202KB/s python-cryptography took 0.2ms: 4992KB/s test_perf: size: 1048576 Bytes pycrypto took 14.1ms: 72510KB/s python-cryptography took 7.1ms: 145223KB/s Decryption Performance: test_perf: size: 16 Bytes pycrypto took 4.1ms: 3KB/s python-cryptography took 0.2ms: 82KB/s test_perf: size: 1024 Bytes pycrypto took 4.4ms: 228KB/s python-cryptography took 0.3ms: 3776KB/s test_perf: size: 1048576 Bytes pycrypto took 87.4ms: 11710KB/s python-cryptography took 78.8ms: 12989KB/s Global Performance: test_perf: size: 16 Bytes pycrypto took 2.2ms: 6KB/s python-cryptography took 0.1ms: 138KB/s test_perf: size: 1024 Bytes pycrypto took 2.3ms: 443KB/s python-cryptography took 0.1ms: 6835KB/s test_perf: size: 1048576 Bytes pycrypto took 50.8ms: 20150KB/s python-cryptography took 46.9ms: 21815KB/s . ---------------------------------------------------------------------- Ran 2 tests in 4.858s OK
Looks at least a few seconds better with unset, even with a vm.
That said, server now seems to launch happily (though I see it is now enforcing the use of an encryption-keyfile=
rather than allowing the old usage of --password-file=
to suffice if running --encryption=AES
).
Unfortunately, neither a windows nor an OSX client (0.17.0 r11687) will successfully connect.
./xpra attach --opengl=on --desktop-scaling=103%,101% --encryption-keyfile=a-password --encryption=AES ... 2016-01-13 15:57:45,237 receiving data using AES encryption 2016-01-13 15:57:45,237 error preparing connection: sha1 is not supported for PBKDF2 by this backend. Traceback (most recent call last): File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/client/client_base.py", line 293, in send_hello hello = self.make_hello_base() File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/client/client_base.py", line 366, in make_hello_base self._protocol.set_cipher_in(self.encryption, iv, key, key_salt, iterations, padding) File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/net/protocol.py", line 192, in set_cipher_in self.cipher_in, self.cipher_in_block_size = get_decryptor(ciphername, iv, password, key_salt, iterations) File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/net/crypto.py", line 140, in get_decryptor key = backend.get_key(password, key_salt, block_size, iterations) File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/net/pycryptography_backend.py", line 27, in get_key kdf = PBKDF2HMAC(algorithm=hashes.SHA1(), length=block_size, salt=strtobytes(key_salt), iterations=iterations, backend=default_backend()) File "cryptography/hazmat/primitives/kdf/pbkdf2.pyc", line 29, in __init__ UnsupportedAlgorithm: sha1 is not supported for PBKDF2 by this backend. 2016-01-13 15:57:45,251 Connection lost
Looks at least a few seconds better with unset, even with a vm.
The difference is very small, I would expect hardware acceleration to make a much more noticeable improvement, especially on larger block sizes.
Please check for "aes" and "avx" in your /proc/cpuinfo
flags. Both on the host and in the VM.
As for the UnsupportedAlgorithm: sha1 is not supported for PBKDF2 by this backend.
, this looks like https://github.com/pyca/cryptography/issues/1958, which seems to be a package installation issue, or an outdated package version.
r11705 will log more information when you run ./xpra/net/pycryptography_backend.py
.
You should have the "openssl" listed in the "backends".
I suspect that your package may be too old, ie: Fedora 21 is no longer supported. Please include the full package version and distro info. ie:
$ rpm -qa python-cryptography python-cryptography-1.2.1-1.fc23.x86_64
Hmmm -
/proc/cpuinfo
. Both are listed on the host though.
(Will look into passing more through from hosts and see if results are... better.)
Running pycryptography_backend.py I do inded see openssl listed in my backends...
backend : python-cryptography backends : ['openssl'] python-cryptography : True python-cryptography.version : 1.0.2
And
rpm -qa python-cryptography python-cryptography-1.0.2-2.fc23.x86_64
Looks like it is a bit old, but a dnf update doesn't seem to find a python-cryptography match for update, and says nothing to do for python-crypto - is there another package name I should try to update?
Both are listed on the host though.
Then they are being filtered out by your hypervisor (KVM).
See http://www.linux-kvm.org/page/Tuning_KVM.
Looks like it is a bit old, but a dnf update doesn't seem to find a python-cryptography match for update...
There won't be, Fedora 21 has been EOLed. Update to something still supported...
Well, I'm not hoping for much with the old fedora 21 VMs we still use... but the rpm -qa python-cryptography
above was run on a fedora 23 VM (python-cryptography-1.0.2-2.fc23.x86_64
does specify fc23).
Detail that catches my eye though, is that my version (1.0.2-2) seems older than the one you indicated from your system (1.2.1-1).
Ah, my bad, will re-spin some packages.
Nope, I don't need to spin any packages, we don't even have a specfile for python-cryptography
.
It is in the regular fedora update channel. A simple dnf update
should get you that version.
You can look it up here: http://rpm.pbone.net/, go to "Advanced Search", select Fedora 23 and type in the package name.
I may just blacklist older versions.
And... I've just checked and even v1.0-1 works fine here!
Interesting.
-d auth
) on launch. (dbus-launch xpra --no-daemon --bind-tcp=0.0.0.0:1203 --start-child=xterm --start-child=xterm --mdns=no start :13 --start-new-commands=yes --tcp-encryption-keyfile=password --tcp-encryption=AES -d auth
)
dbus-launch xpra attach tcp:[IP]:[port] --encryption-keyfile=password --encryption=AES -d auth
), I don't see any problems... and the -d auth
server side outputs the following:
2016-01-22 14:26:55,837 socktype=tcp, auth class=None, encryption=AES, keyfile=password 2016-01-22 14:26:56,039 get_encryption_key(None, password) 2016-01-22 14:26:56,040 loading encryption key from keyfile: password 2016-01-22 14:26:56,040 set output cipher using encryption key bob 2016-01-22 14:26:56,041 sending data using AES encryption 2016-01-22 14:26:56,044 receiving data using AES encryption 2016-01-22 14:26:56,046 server cipher={'cipher.key_salt': '64e74d1760084e43988326760ecae61558311e9be4ea49ad8cadd98cadc274c8', 'cipher.padding': 'PKCS#7', 'cipher.padding.options': ['PKCS#7', 'legacy'], 'cipher.key_stretch_iterations': 1000, 'cipher.iv': '03a140c86c88449b', 'cipher': 'AES'}
However:
C:\Program Files (x86)\Xpra>xpra_cmd.exe attach tcp:[IP]:[port] --opengl=on --desktop-scaling=1.34,157% --speaker-codec=opus --encryption-keyfile=key.txt --encryption=AES -d auth ... 2016-01-22 14:22:56,359 error preparing connection: sha1 is not supported for PBKDF2 by this backend. Traceback (most recent call last): File "xpra\client\client_base.pyc", line 293, in send_hello File "xpra\client\client_base.pyc", line 366, in make_hello_base File "xpra\net\protocol.pyc", line 192, in set_cipher_in File "xpra\net\crypto.pyc", line 140, in get_decryptor File "xpra\net\pycryptography_backend.pyc", line 37, in get_key File "cryptography\hazmat\primitives\kdf\pbkdf2.pyc", line 29, in __init__ UnsupportedAlgorithm: sha1 is not supported for PBKDF2 by this backend. 2016-01-22 14:22:56,391 Connection lost
The server, with -d auth
outputs the following when the client tries (& fails) to connect.
2016-01-22 14:22:14,521 New tcp connection received from [win32 client IP] 2016-01-22 14:22:14,523 socktype=tcp, auth class=None, encryption=AES, keyfile=password 2016-01-22 14:22:14,664 Warning: client does not provide encryption tokens 2016-01-22 14:22:14,664 Warning: authentication failed: missing encryption 2016-01-22 14:22:15,666 Disconnecting client '[win32 client IP]': 2016-01-22 14:22:15,666 missing encryption 2016-01-22 14:22:15,667 Connection lost
I don't have a fix for this yet, until then r11760 will validate the backends before choosing one, so win32 and osx should fallback to pycrypto until I figure it out.
Ok, testing with 0.17.0 r11765 windows client against a 0.17.0 r11767 fedora 23 server, I'm still getting the Error: encryption library python-cryptography failed validation!
message, but the client must be falling back successfully, because the session is initializing (launching with xpra_cmd.exe attach --password-file=key.txt --opengl=on --encryption-keyfile=encryption.txt --encryption=AES
... note, however, that launching with --tcp-encryption-keyfile=encryption.txt
give a encryption AES cannot be used without a keyfile (see --encryption-keyfile option)
error message and fails to connect).
--tcp-encryption-keyfile=
) with osx 0.17.0 r11761 client.
... so, fallback seems to be working, specifying tcp-encryption (or password) on the client side (just to match server side expectations/syntax) doesn't seem to be supported.
Will obviously leave this open until the fallback isn't a necessary evil.
Tested again, connection seems to be working with windows clients at this point (0.17.0 r11886 client against 0.17.0 r11888 fedora 23 server).
Looks like we're still waiting on osx packaging, so won't close the ticket quite yet.
It looks like the stats for with/without acceleration should be put into #876, so I'll put them there... but there is one syntax detail I'll mention here.
When forgetting to specify --auth=
or --tcp-auth=
upon server launch, I get a warning, but without any hints about what the syntax should be exactly. Making the bad guess of --tcp-auth=password-file
I got an error message, rather than a gentle reminder that I'm an idiot.
2016-02-08 11:33:37,855 Error: cannot start the xpra server Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/xpra/scripts/server.py", line 1091, in run_server app.init(opts) File "/usr/lib64/python2.7/site-packages/xpra/x11/server.py", line 196, in init X11ServerBase.init(self, opts) File "/usr/lib64/python2.7/site-packages/xpra/x11/x11_server_base.py", line 81, in init GTKServerBase.init(self, opts) File "/usr/lib64/python2.7/site-packages/xpra/server/server_base.py", line 199, in init ServerCore.init(self, opts) File "/usr/lib64/python2.7/site-packages/xpra/server/server_core.py", line 197, in init self.init_auth(opts) File "/usr/lib64/python2.7/site-packages/xpra/server/server_core.py", line 201, in init_auth self.tcp_auth_class = self.get_auth_module("tcp-socket", opts.tcp_auth or opts.auth, opts) File "/usr/lib64/python2.7/site-packages/xpra/server/server_core.py", line 237, in get_auth_module raise Exception("cannot find authentication module '%s' (supported: %s)", auth, AUTH_MODULES.keys()) Exception: ("cannot find authentication module '%s' (supported: %s)", 'password-file', ['none', 'allow', 'file', 'reject', 'fail', 'pam']) 2016-02-08 11:33:37,858 ("cannot find authentication module '%s' (supported: %s)", 'password-file', ['none', 'allow', 'file ', 'reject', 'fail', 'pam'])
There seem to be all the options mentioned in the error... the inclusion of the error in the message may seem clumsy.
Looks like we're still waiting on osx packaging, so won't close the ticket quite yet.
The latest OSX images should have it included already. Do they not?
I get a warning, but without any hints about what the syntax should be exactly.
This is detailed in the manual.
Making the bad ... I got an error message, rather than a gentle reminder that I'm an idiot.
r11890 gets rid of the ugly stacktrace, r11891 also ensures we don't leave the Xvfb process orphaned. (both could be backported - the second one is a bit more risky)
Ok - found the beta directory osx 0.17.0 r11886 client to test.
Looks like the python-cryptography package is in place and working.
I'll close this and we can continue to follow up with the numbers (once I manage to sort out the KVM issues) in #876.
Note: updating to openssl 1.1.0 requires updating to python-cryptography 1.5 which breaks on win32...
The list of libraries built by openssl have changed, to fix the build against 1.1.0 _get_openssl_libraries
should now return:
return ["libssl", "libcrypto", "crypt32", "gdi32", "user32", "ws2_32"]
And then, we need to patch our build to find the dlss: r13485 + r13486, and just place the openssl build in %XPRA_WIN32_BUILD_LIB_PREFIX%\OpenSSL
.
For the html5 client, see #2615.
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1029