Xpra: Ticket #1029: hardware accelerated AES

Using encryption can be costly (apparently causing a 20 to 30% framerate reduction on video playback), we should be using AES-NI to speed it up.

Great post on AES-NI: The Intel Advanced Encryption Standard (AES) Extensions, including some information on attack vectors..

The current version of pycrypto is very stale (June 2014 with no updates scheduled), but there is some code to support it in git: problems with this implementation though: at least a bug, and maybe some licensing issues too. I think we should move to Crypto++): #876.

Alternatives:



Fri, 13 Nov 2015 07:10:53 GMT - Antoine Martin: status, milestone changed

See also #198


Fri, 08 Jan 2016 10:23:21 GMT - Antoine Martin:

#876 should give us hardware acceleration for free. It is a little bit difficult to verify that hardware acceleration is enabled as this is not exposed through the wrapper...

Note: for those desperate to get more performance, consider using newer openssl versions: switching from v1.0.1 to v1.0.2 can give you 10 to 50% gains: https://github.com/nodejs/node/wiki/Crypto-Performance-Notes-for-OpenSSL-1.0.2a-on-iojs-v1.8.0


Sat, 09 Jan 2016 12:13:11 GMT - Antoine Martin: owner, status changed

Here's how you check that your openssl builds supports the CPU acceleration found in the CPU (if any!): How can I check if OpenSSL is support/use the Intel AES-NI?.

This system shows both "avx" and "aes" in /proc/cpuinfo. (without-accel means running with OPENSSL_ia32cap="~0x200000200000000", with-accel means running with it unset - which is not the same as setting it to an empty string!) My guess is that "avx" is probably always used if available.


So, sticking to the systems with aes acceleration for further tests, (both running against openssl 1.0.2e).

I can then compare with and without hardware acceleration from our python code by running our unit test (see #876) both with and without:

PYTHONPATH=. python ./tests/unit/net/crypto_test.py
OPENSSL_ia32cap="~0x200000200000000" PYTHONPATH=. python ./tests/unit/net/crypto_test.py

And although the difference is not noticeable at all on small packets (1KB and lower) since we are mostly measuring the overheads (python and actual measurements), the results on larger packets (64KB and above) are impressive: the performance is more than doubled. (on top of the huge gains already made by switching to python-cryptography #876)


@afarr: this is mostly a FYI, the work to be done is to switch to python-cryptography as per #876, and unless you are using ancient CPUs or an ancient openssl library, you will get the benefits of hardware acceleration automatically. The only potential problem is if you are using virtual machines that strip some of the cpu flags. Some systems do this for supporting live migration and other exotic features - often a complete waste of good CPU extensions. So you may want to verify both on the host and in the virtual machine, you should get similar results. (the virtual machine overhead should be negligible)


Wed, 13 Jan 2016 00:44:09 GMT - alas:

Hmm... I'm running into an issue when I try to run --encryption=AES on my 0.17.0 r11669 fedora 23 (VM) server.

Running python /home/jimador/xpra-trunk/src/tests/unit/net/crypto_test.py I get output that looks like I have all the listed ciphers:

[jimador@jimador net]$ python crypto_test.py
.Encryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.9ms:                3KB/s
python-cryptography              took   0.2ms:               64KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.9ms:              203KB/s
python-cryptography              took   0.2ms:             4901KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  13.7ms:            74790KB/s
python-cryptography              took   6.6ms:           155833KB/s
Decryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.0ms:                3KB/s
python-cryptography              took   0.2ms:               85KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.1ms:              241KB/s
python-cryptography              took   0.3ms:             3914KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  86.0ms:            11902KB/s
python-cryptography              took  79.7ms:            12853KB/s
Global Performance:
test_perf: size: 16 Bytes
pycrypto                         took   2.3ms:                6KB/s
python-cryptography              took   0.1ms:              113KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   2.3ms:              440KB/s
python-cryptography              took   0.1ms:             6868KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  50.4ms:            20313KB/s
python-cryptography              took  45.9ms:            22331KB/s
.
----------------------------------------------------------------------
Ran 2 tests in 4.858s
OK

But, when I launch with --encryption=AES, the server crashes with the following traceback:

2016-01-12 16:02:36,328 Error: cannot start the xpra server
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xpra/scripts/server.py", line 1006, in run_server
    app.init(opts)
  File "/usr/lib64/python2.7/site-packages/xpra/x11/server.py", line 196, in init
    X11ServerBase.init(self, opts)
  File "/usr/lib64/python2.7/site-packages/xpra/x11/x11_server_base.py", line 81, in init
    GTKServerBase.init(self, opts)
  File "/usr/lib64/python2.7/site-packages/xpra/server/server_base.py", line 194, in init
    ServerCore.init(self, opts)
  File "/usr/lib64/python2.7/site-packages/xpra/server/server_core.py", line 178, in init
    validate_encryption(opts)
  File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 795, in validate_encryption
    do_validate_encryption(opts.encryption, opts.tcp_encryption, opts.password_file, opts.encryption_keyfile, opts.tcp_encryption_keyfile)
  File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 801, in do_validate_encryption
    raise InitException("cannot use encryption: no ciphers available (a crypto library must be installed)")
InitException: cannot use encryption: no ciphers available (a crypto library must be installed)
2016-01-12 16:02:36,332 cannot use encryption: no ciphers available (a crypto library must be installed)

The unit test results definitely look promising though.


Wed, 13 Jan 2016 09:51:58 GMT - Antoine Martin:

I get output that looks like I have all the listed ciphers:


You're not telling us what sort of CPU this is. It would be good to know if hardware acceleration is being used (as per previous comment, using OPENSSL_ia32cap=..)


But, when I launch with --encryption=AES, the server crashes with the following traceback:


Oops, sorry about that, incomplete work: should be OK with r11680, further improved with r11681 so we don't even load the crypto bits when they aren't being used, in the case of python-cryptography this will save having the openssl library loaded in memory if we aren't using it (ie: when running the sound subprocess, we don't need it or want it)


Thu, 14 Jan 2016 00:07:37 GMT - alas:

I would've assumed that there was no hardware acceleration being used on the vm I use as a server... but, as you mention above (comment:3), "without-accel means running with OPENSSL_ia32cap="~0x200000200000000", with-accel means running with it unset".

Comparison seems to suggest there's some hardware acceleration happening somewhere.

Running it unset (=with-accel):

[jimador@jimador net]$ python crypto_test.py
.Encryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.5ms:                3KB/s
python-cryptography              took   0.2ms:               81KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.5ms:              222KB/s
python-cryptography              took   0.2ms:             5027KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  13.8ms:            74343KB/s
python-cryptography              took   6.6ms:           154726KB/s
Decryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.1ms:                3KB/s
python-cryptography              took   0.2ms:               85KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.2ms:              239KB/s
python-cryptography              took   0.3ms:             3910KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  87.4ms:            11719KB/s
python-cryptography              took  80.2ms:            12768KB/s
Global Performance:
test_perf: size: 16 Bytes
pycrypto                         took   2.4ms:                6KB/s
python-cryptography              took   0.1ms:              140KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   2.4ms:              415KB/s
python-cryptography              took   0.2ms:             6478KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  56.6ms:            18085KB/s
python-cryptography              took  48.2ms:            21262KB/s
.
----------------------------------------------------------------------
Ran 2 tests in 4.785s
OK

Vs. "without accel":

[jimador@jimador net]$ OPENSSL_ia32cap="~0x200000200000000" python crypto_test.py
.Encryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.9ms:                3KB/s
python-cryptography              took   0.3ms:               59KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.9ms:              202KB/s
python-cryptography              took   0.2ms:             4992KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  14.1ms:            72510KB/s
python-cryptography              took   7.1ms:           145223KB/s
Decryption Performance:
test_perf: size: 16 Bytes
pycrypto                         took   4.1ms:                3KB/s
python-cryptography              took   0.2ms:               82KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   4.4ms:              228KB/s
python-cryptography              took   0.3ms:             3776KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  87.4ms:            11710KB/s
python-cryptography              took  78.8ms:            12989KB/s
Global Performance:
test_perf: size: 16 Bytes
pycrypto                         took   2.2ms:                6KB/s
python-cryptography              took   0.1ms:              138KB/s
test_perf: size: 1024 Bytes
pycrypto                         took   2.3ms:              443KB/s
python-cryptography              took   0.1ms:             6835KB/s
test_perf: size: 1048576 Bytes
pycrypto                         took  50.8ms:            20150KB/s
python-cryptography              took  46.9ms:            21815KB/s
.
----------------------------------------------------------------------
Ran 2 tests in 4.858s
OK


Looks at least a few seconds better with unset, even with a vm.



That said, server now seems to launch happily (though I see it is now enforcing the use of an encryption-keyfile= rather than allowing the old usage of --password-file= to suffice if running --encryption=AES).

Unfortunately, neither a windows nor an OSX client (0.17.0 r11687) will successfully connect.

./xpra attach  --opengl=on --desktop-scaling=103%,101% --encryption-keyfile=a-password --encryption=AES
...
2016-01-13 15:57:45,237 receiving data using AES encryption
2016-01-13 15:57:45,237 error preparing connection: sha1 is not supported for PBKDF2 by this backend.
Traceback (most recent call last):
  File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/client/client_base.py", line 293, in send_hello
    hello = self.make_hello_base()
  File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/client/client_base.py", line 366, in make_hello_base
    self._protocol.set_cipher_in(self.encryption, iv, key, key_salt, iterations, padding)
  File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/net/protocol.py", line 192, in set_cipher_in
    self.cipher_in, self.cipher_in_block_size = get_decryptor(ciphername, iv, password, key_salt, iterations)
  File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/net/crypto.py", line 140, in get_decryptor
    key = backend.get_key(password, key_salt, block_size, iterations)
  File "/Users/Schadenfreude/Desktop/xpra-catalog/xpra-ant-17-11687/Xpra.app/Contents/Resources/lib/python/xpra/net/pycryptography_backend.py", line 27, in get_key
    kdf = PBKDF2HMAC(algorithm=hashes.SHA1(), length=block_size, salt=strtobytes(key_salt), iterations=iterations, backend=default_backend())
  File "cryptography/hazmat/primitives/kdf/pbkdf2.pyc", line 29, in __init__
UnsupportedAlgorithm: sha1 is not supported for PBKDF2 by this backend.
2016-01-13 15:57:45,251 Connection lost

Thu, 21 Jan 2016 06:46:03 GMT - Antoine Martin:

Looks at least a few seconds better with unset, even with a vm.


The difference is very small, I would expect hardware acceleration to make a much more noticeable improvement, especially on larger block sizes. Please check for "aes" and "avx" in your /proc/cpuinfo flags. Both on the host and in the VM.


As for the UnsupportedAlgorithm: sha1 is not supported for PBKDF2 by this backend., this looks like https://github.com/pyca/cryptography/issues/1958, which seems to be a package installation issue, or an outdated package version.

r11705 will log more information when you run ./xpra/net/pycryptography_backend.py. You should have the "openssl" listed in the "backends".

I suspect that your package may be too old, ie: Fedora 21 is no longer supported. Please include the full package version and distro info. ie:

$ rpm -qa python-cryptography
python-cryptography-1.2.1-1.fc23.x86_64

Thu, 21 Jan 2016 23:37:40 GMT - alas:

Hmmm -

(Will look into passing more through from hosts and see if results are... better.)


Running pycryptography_backend.py I do inded see openssl listed in my backends...

backend                        : python-cryptography
backends                       : ['openssl']
python-cryptography            : True
python-cryptography.version    : 1.0.2

And

rpm -qa python-cryptography
python-cryptography-1.0.2-2.fc23.x86_64

Looks like it is a bit old, but a dnf update doesn't seem to find a python-cryptography match for update, and says nothing to do for python-crypto - is there another package name I should try to update?


Thu, 21 Jan 2016 23:58:12 GMT - Antoine Martin:

Both are listed on the host though.


Then they are being filtered out by your hypervisor (KVM). See http://www.linux-kvm.org/page/Tuning_KVM.


Looks like it is a bit old, but a dnf update doesn't seem to find a python-cryptography match for update...


There won't be, Fedora 21 has been EOLed. Update to something still supported...


Fri, 22 Jan 2016 18:21:38 GMT - alas:

Well, I'm not hoping for much with the old fedora 21 VMs we still use... but the rpm -qa python-cryptography above was run on a fedora 23 VM (python-cryptography-1.0.2-2.fc23.x86_64 does specify fc23).

Detail that catches my eye though, is that my version (1.0.2-2) seems older than the one you indicated from your system (1.2.1-1).


Fri, 22 Jan 2016 18:24:46 GMT - Antoine Martin: owner, status changed

Ah, my bad, will re-spin some packages.


Fri, 22 Jan 2016 19:28:42 GMT - Antoine Martin: owner, status changed

Nope, I don't need to spin any packages, we don't even have a specfile for python-cryptography. It is in the regular fedora update channel. A simple dnf update should get you that version. You can look it up here: http://rpm.pbone.net/, go to "Advanced Search", select Fedora 23 and type in the package name.

I may just blacklist older versions.


Fri, 22 Jan 2016 19:35:19 GMT - Antoine Martin:

And... I've just checked and even v1.0-1 works fine here!


Sat, 23 Jan 2016 00:30:48 GMT - alas:

Interesting.

2016-01-22 14:26:55,837 socktype=tcp, auth class=None, encryption=AES, keyfile=password
2016-01-22 14:26:56,039 get_encryption_key(None, password)
2016-01-22 14:26:56,040 loading encryption key from keyfile: password
2016-01-22 14:26:56,040 set output cipher using encryption key bob
2016-01-22 14:26:56,041 sending data using AES encryption
2016-01-22 14:26:56,044 receiving data using AES encryption
2016-01-22 14:26:56,046 server cipher={'cipher.key_salt': '64e74d1760084e43988326760ecae61558311e9be4ea49ad8cadd98cadc274c8', 'cipher.padding': 'PKCS#7', 'cipher.padding.options': ['PKCS#7', 'legacy'], 'cipher.key_stretch_iterations': 1000, 'cipher.iv': '03a140c86c88449b', 'cipher': 'AES'}

However:

C:\Program Files (x86)\Xpra>xpra_cmd.exe attach tcp:[IP]:[port] --opengl=on --desktop-scaling=1.34,157% --speaker-codec=opus --encryption-keyfile=key.txt --encryption=AES -d auth
...
2016-01-22 14:22:56,359 error preparing connection: sha1 is not supported for PBKDF2 by this backend.
Traceback (most recent call last):
  File "xpra\client\client_base.pyc", line 293, in send_hello
  File "xpra\client\client_base.pyc", line 366, in make_hello_base
  File "xpra\net\protocol.pyc", line 192, in set_cipher_in
  File "xpra\net\crypto.pyc", line 140, in get_decryptor
  File "xpra\net\pycryptography_backend.pyc", line 37, in get_key
  File "cryptography\hazmat\primitives\kdf\pbkdf2.pyc", line 29, in __init__
UnsupportedAlgorithm: sha1 is not supported for PBKDF2 by this backend.
2016-01-22 14:22:56,391 Connection lost

The server, with -d auth outputs the following when the client tries (& fails) to connect.

2016-01-22 14:22:14,521 New tcp connection received from [win32 client IP]
2016-01-22 14:22:14,523 socktype=tcp, auth class=None, encryption=AES, keyfile=password
2016-01-22 14:22:14,664 Warning: client does not provide encryption tokens
2016-01-22 14:22:14,664 Warning: authentication failed: missing encryption
2016-01-22 14:22:15,666 Disconnecting client '[win32 client IP]':
2016-01-22 14:22:15,666  missing encryption
2016-01-22 14:22:15,667 Connection lost

Tue, 26 Jan 2016 01:05:24 GMT - Antoine Martin: owner, status changed

I don't have a fix for this yet, until then r11760 will validate the backends before choosing one, so win32 and osx should fallback to pycrypto until I figure it out.


Thu, 28 Jan 2016 03:12:58 GMT - alas:

Ok, testing with 0.17.0 r11765 windows client against a 0.17.0 r11767 fedora 23 server, I'm still getting the Error: encryption library python-cryptography failed validation! message, but the client must be falling back successfully, because the session is initializing (launching with xpra_cmd.exe attach --password-file=key.txt --opengl=on --encryption-keyfile=encryption.txt --encryption=AES... note, however, that launching with --tcp-encryption-keyfile=encryption.txt give a encryption AES cannot be used without a keyfile (see --encryption-keyfile option) error message and fails to connect).

... so, fallback seems to be working, specifying tcp-encryption (or password) on the client side (just to match server side expectations/syntax) doesn't seem to be supported.

Will obviously leave this open until the fallback isn't a necessary evil.


Mon, 08 Feb 2016 20:37:00 GMT - alas:

Tested again, connection seems to be working with windows clients at this point (0.17.0 r11886 client against 0.17.0 r11888 fedora 23 server).

Looks like we're still waiting on osx packaging, so won't close the ticket quite yet.

It looks like the stats for with/without acceleration should be put into #876, so I'll put them there... but there is one syntax detail I'll mention here.

When forgetting to specify --auth= or --tcp-auth= upon server launch, I get a warning, but without any hints about what the syntax should be exactly. Making the bad guess of --tcp-auth=password-file I got an error message, rather than a gentle reminder that I'm an idiot.

2016-02-08 11:33:37,855 Error: cannot start the xpra server
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xpra/scripts/server.py", line 1091, in run_server
    app.init(opts)
  File "/usr/lib64/python2.7/site-packages/xpra/x11/server.py", line 196, in init
    X11ServerBase.init(self, opts)
  File "/usr/lib64/python2.7/site-packages/xpra/x11/x11_server_base.py", line 81, in init
    GTKServerBase.init(self, opts)
  File "/usr/lib64/python2.7/site-packages/xpra/server/server_base.py", line 199, in init
    ServerCore.init(self, opts)
  File "/usr/lib64/python2.7/site-packages/xpra/server/server_core.py", line 197, in init
    self.init_auth(opts)
  File "/usr/lib64/python2.7/site-packages/xpra/server/server_core.py", line 201, in init_auth
    self.tcp_auth_class = self.get_auth_module("tcp-socket", opts.tcp_auth or opts.auth, opts)
  File "/usr/lib64/python2.7/site-packages/xpra/server/server_core.py", line 237, in get_auth_module
    raise Exception("cannot find authentication module '%s' (supported: %s)", auth, AUTH_MODULES.keys())
Exception: ("cannot find authentication module '%s' (supported: %s)", 'password-file', ['none', 'allow', 'file', 'reject',  'fail', 'pam'])
2016-02-08 11:33:37,858 ("cannot find authentication module '%s' (supported: %s)", 'password-file', ['none', 'allow', 'file ', 'reject', 'fail', 'pam'])

There seem to be all the options mentioned in the error... the inclusion of the error in the message may seem clumsy.


Mon, 08 Feb 2016 20:58:40 GMT - Antoine Martin: owner, status changed

Looks like we're still waiting on osx packaging, so won't close the ticket quite yet.


The latest OSX images should have it included already. Do they not?


I get a warning, but without any hints about what the syntax should be exactly.


This is detailed in the manual.


Making the bad ... I got an error message, rather than a gentle reminder that I'm an idiot.


r11890 gets rid of the ugly stacktrace, r11891 also ensures we don't leave the Xvfb process orphaned. (both could be backported - the second one is a bit more risky)


Mon, 08 Feb 2016 23:00:16 GMT - alas: status changed; resolution set

Ok - found the beta directory osx 0.17.0 r11886 client to test.

Looks like the python-cryptography package is in place and working.

I'll close this and we can continue to follow up with the numbers (once I manage to sort out the KVM issues) in #876.


Sun, 28 Aug 2016 16:03:49 GMT - Antoine Martin:

Note: updating to openssl 1.1.0 requires updating to python-cryptography 1.5 which breaks on win32... The list of libraries built by openssl have changed, to fix the build against 1.1.0 _get_openssl_libraries should now return:

return ["libssl", "libcrypto",
        "crypt32", "gdi32", "user32", "ws2_32"]

And then, we need to patch our build to find the dlss: r13485 + r13486, and just place the openssl build in %XPRA_WIN32_BUILD_LIB_PREFIX%\OpenSSL.


Fri, 03 Jul 2020 10:56:10 GMT - Antoine Martin:

For the html5 client, see #2615.


Sat, 23 Jan 2021 05:12:57 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1029