Xpra: Ticket #1501: 2.1 moduleset updates

Tracker ticket for the moduleset changes and those that may need to be backported. (see #1404 for 2.0)

Probably safe to backport to 1.x and 2.0.x:

Not needed:



Sat, 15 Apr 2017 08:06:01 GMT - Antoine Martin: attachment set

patch to allow us to build 1.0.x versions on a newer build system (libraries renamed or just not installed)


Sat, 15 Apr 2017 08:41:26 GMT - Antoine Martin: owner changed

All built OK except for ffmpeg which tried to link with "-framework CoreMedia", removing all those from the configure script allowed it to build just fine.

beta 1.0.6 packages posted: http://xpra.org/beta/osx.

@smo: how do we patch the configure script? (ideally using sed rather than a patch which may not apply cleanly with updates)


Sun, 30 Apr 2017 06:32:59 GMT - Antoine Martin:

Important update from upstream: Removal of support for obsolete MacOS X Versions and global module update: I've completed the removal of support for versions of MacOS X earlier than 10.8 from gtk-osx. If you still need to build for 10.4u or 10.5, or on 10.4-10.7, I've tagged the last commit that supports that with "Last-Lion" to make it easy to have a jhbuildrc and modulesets to support your needs.


Mon, 15 May 2017 06:12:09 GMT - Antoine Martin:

@smo: how to I update nasm? nasm 2.13 is out.


Mon, 15 May 2017 08:51:28 GMT - Antoine Martin: owner, status changed


I had to fix the gstreamer 1.12.0 osxaudio compilation in gst-plugins-good to avoid this error:

$ gst-launch-1.0 audiotestsrc ! osxaudiosink
(gst-plugin-scanner:86177): GStreamer-WARNING **: Failed to load plugin \
    '/Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so': dlopen(/Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so, 2): \
    Symbol not found: _AudioComponentFindNext
  Referenced from: /Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so (which was built for Mac OS X 10.11)
  Expected in: /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
 in /Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so
(gst-plugin-scanner:86177): GStreamer-WARNING **: Failed to load plugin \
    '/Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so': dlopen(/Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so, 2): \
    Symbol not found: _AudioComponentFindNext
  Referenced from: /Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so (which was built for Mac OS X 10.11)
  Expected in: /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
 in /Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so
WARNING: erroneous pipeline: no element "osxaudiosink"

I fixed this by adding -framework AudioUnit -framework CoreAudio -framework AudioToolbox to the LDFLAGS in the sys/osxaudio/Makefile. (inspired by OSX 10.6 gnuradio: Audio Library Error)

Not sure why this has come up now. And then... things get worse: most of the muxers no longer work: ogg and mka formats all fail with the dreaded error:

sound output pipeline error: gst-stream-error-quark: Internal data stream error. (1)
sound output  appsrc do-timestamp=1 name=src emit-signals=0 block=0 is-live=0 stream-type=0 format=4 ! \
    matroskademux ! opusdec ! audioconvert ! audioresample ! volume name=volume volume=0 ! \
    queue name=queue min-threshold-time=0 max-size-buffers=0 max-size-bytes=0 max-size-time=450000000 leaky=2 ! \
    osxaudiosink async=True qos=True sync=False
sound output  gstbasesrc.c(2939)
sound output  void gst_base_src_loop(GstPad *) ()
sound output  /GstPipeline:pipeline0/GstAppSrc:src:
sound output  streaming stopped, reason not-negotiated (-4)

Which is why this was reverted.


Mon, 15 May 2017 17:26:23 GMT - Antoine Martin:

Ended up starting from scratch again because gstreamer failed again at runtime.


Tue, 16 May 2017 08:06:16 GMT - Antoine Martin: owner, status changed

Panic is over, things build fine again, provided that:

(some of the modules should be picked up automatically but for some unknown reason aren't: "python-setupttols-scm" should get built before "python-lz4" automatically for example)

Confirmed that the gstreamer problems occur when switching to 1.12.0, though it could still be a problem with the toolchain / leftovers from the previous version.


r15865 adds a module for "brotli", but the Makefile doesn't have an "install" command, so this fails. I can install it by appending this to the Makefile by hand:

install :
	install -d $(BINDIR) $(JHBUILD_PREFIX)/bin
	install -m 0755 $(BINDIR)/bro $(JHBUILD_PREFIX)/bin/

But this then fails when running it from jhbuild with:

W: Ignoring uninstalled package: brotli
Traceback (most recent call last):
  File "/Users/osx/.local/bin/jhbuild", line 32, in <module>
    jhbuild.main.main(sys.argv[1:])
  File "/Users/osx/Source/jhbuild/jhbuild/main.py", line 155, in main
    rc = jhbuild.commands.run(command, config, args, help=lambda: print_help(parser))
  File "/Users/osx/Source/jhbuild/jhbuild/commands/__init__.py", line 183, in run
    return cmd.execute(config, args, help)
  File "/Users/osx/Source/jhbuild/jhbuild/commands/__init__.py", line 52, in execute
    return self.run(config, options, args, help)
  File "/Users/osx/Source/jhbuild/jhbuild/commands/base.py", line 268, in run
    return build.build()
  File "/Users/osx/Source/jhbuild/jhbuild/frontends/buildscript.py", line 163, in build
    error, altphases = module.run_phase(self, phase)
  File "/Users/osx/Source/jhbuild/jhbuild/modtypes/__init__.py", line 420, in run_phase
    method(buildscript)
  File "/Users/osx/Source/jhbuild/jhbuild/modtypes/autotools.py", line 314, in do_install
    self.process_install(buildscript, self.get_revision())
  File "/Users/osx/Source/jhbuild/jhbuild/modtypes/__init__.py", line 316, in process_install
    new_contents = fileutils.accumulate_dirtree_contents(destdir_prefix)
  File "/Users/osx/Source/jhbuild/jhbuild/utils/fileutils.py", line 45, in accumulate_dirtree_contents
    _accumulate_dirtree_contents_recurse(path, contents)
  File "/Users/osx/Source/jhbuild/jhbuild/utils/fileutils.py", line 27, in _accumulate_dirtree_contents_recurse
    names = os.listdir(path)
OSError: [Errno 2] No such file or directory: '/Users/osx/gtk/inst/_jhbuild/root-brotli/Users/osx/gtk/inst'

@smo: how do I make it play nice?


Thu, 18 May 2017 04:28:02 GMT - Smo:

*** Installing brotli *** [1/1]
make DESTDIR=/Users/spikesinc/gtk/inst/_jhbuild/root-brotli install

Have to install to DESTDIR instead and that should work


Thu, 18 May 2017 04:30:29 GMT - Smo:

Easier way is to just use cmake it would seem


Thu, 18 May 2017 04:31:57 GMT - Smo:

change brotli to cmake in r15880


Thu, 18 May 2017 04:36:56 GMT - Smo:

I had a few issues myself when building all of this from the start again. First off I change my .jhbuildrc-custom to be like yours and downloaded the cacert.pem. This worked nicely.

*** Checking out glib *** [20/134]
curl --continue-at - -L http://ftp.gnome.org/pub/GNOME/sources/glib/2.48/glib-2.48.0.tar.xz -o /Users/spikesinc/gtk/source/pkgs/glib-2.48.0.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   377  100   377    0     0    577      0 --:--:-- --:--:-- --:--:--   576
100 7312k  100 7312k    0     0  1727k      0  0:00:04  0:00:04 --:--:-- 2685k
xzcat -d "/Users/spikesinc/gtk/source/pkgs/glib-2.48.0.tar.xz" | tar xf -
*** Error during phase checkout of glib: could not download patch (error: 404) *** [20/134]

Ran into one of the patches missing so I will have to look into what that was. I was able to just start the build again and continue.

I also had issues where some of the compiling processes would get killed? Restarting the build usually got me by these issues.

python-setuptools-scm tried to install before setuptools was installed which caused it to produce this stacktrace

Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    import setuptools
ImportError: No module named setuptools

Also came across this error trying to install python-zeroconf not sure what to do about this one.

*** Building python-zeroconf *** [87/134]
python setup.py build
Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    [l for l in open(join(PROJECT_ROOT, 'zeroconf.py')) if '__version__' in l][0]
LookupError: unknown encoding:
*** Error during phase build of python-zeroconf: ########## Error running python setup.py build *** [87/134]

Thu, 18 May 2017 04:40:00 GMT - Smo:

I guess that was too easy. Still have this issue with brotli

[JH][~/xpra-trunk/osx/jhbuild]$ bro
dyld: Library not loaded: libbrotlienc.0.6.0.dylib
  Referenced from: /Users/build/gtk/inst/bin/bro
  Reason: image not found
Trace/BPT trap: 5

Thu, 18 May 2017 07:35:12 GMT - Antoine Martin:

I don't have my macos VM with me so I cannot test today, but maybe the cmake build is different (I don't think the pure "make" build uses shared libraries), and maybe the shared library isn't installed, or not properly installed?


Fri, 19 May 2017 14:16:49 GMT - Antoine Martin:

I also had issues where some of the compiling processes would get killed? Restarting the build usually got me by these issues.

OOM killer maybe? I'm giving this build VM lots of cpus and tons of ram..

python-setuptools-scm tried to install before setuptools was installed which caused it to produce this stacktrace

Adding "python-setuptools" before "python-setuptools-scm" in jhbuildrc-custom should fix that.

Also came across this error trying to install python-zeroconf not sure what to do about this one.

Weird one. I can't reproduce. Could be something in your environment which is different.

I guess that was too easy. Still have this issue with brotli

Are you trying to run it from within a jhbuild shell? (I got the same error when I tried from outside the jhbuild shell env)


Fri, 02 Jun 2017 17:55:04 GMT - Antoine Martin:

Updates (all safe to backport and recommended):

TODO:


Sun, 11 Jun 2017 16:57:32 GMT - Antoine Martin:


Sun, 11 Jun 2017 20:16:11 GMT - Antoine Martin: owner, status changed

FFS: I fixed most of the modules, see comment:15 but now we have a new packaging problem with python-cryptography (detected by the unit tests):

ERROR: unit tests failed, see /Users/osx/Xpra-svn/trunk/src/unittests/unittest.log:
Traceback (most recent call last):
  File "/Users/osx/Xpra-svn/trunk/src/unittests/unit/net/crypto_test.py", line 30, in setUp
    pycryptography_backend.init()
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/xpra/net/pycryptography_backend.py", line 52, in init
    backend = default_backend()
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 16, in <module>
    from cryptography import utils, x509
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/x509/__init__.py", line 8, in <module>
    from cryptography.x509 import certificate_transparency
ImportError: cannot import name certificate_transparency

Sun, 11 Jun 2017 21:08:38 GMT - Antoine Martin: priority changed

Figuring out the new dependency:

$ python -c "from cryptography.hazmat.backends.openssl.backend import backend"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 16, in <module>
    from cryptography import utils, x509
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/x509/__init__.py", line 9, in <module>
    from cryptography.x509.base import (
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/x509/base.py", line 16, in <module>
    from cryptography.x509.extensions import Extension, ExtensionType
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/x509/extensions.py", line 13, in <module>
    from asn1crypto.keys import PublicKeyInfo
ImportError: No module named asn1crypto.keys

So r16045 adds asn1crypto to the moduleset.

But then we hit a new problem:

ERROR: test_backends (__main__.TestCrypto)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/osx/Xpra-svn/trunk/src/unittests/unit/net/crypto_test.py", line 83, in test_backends
    self.do_test_backend()
  File "/Users/osx/Xpra-svn/trunk/src/unittests/unit/net/crypto_test.py", line 63, in do_test_backend
    v = enc.encrypt(message)
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/primitives/ciphers/base.py", line 149, in update
    return self._ctx.update(data)
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py", line 116, in update
    n = self.update_into(data, buf)
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py", line 127, in update_into
    "unsigned char *", self._backend._ffi.from_buffer(buf)
TypeError: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object

Looks like we can't feed it a string or bytearray now. Discussion here: Creating a cdata from a buffer This is a CFFI issue... and we're stuck on version 1.5.2 (see ticket:1404#comment:18 for details). Downgrade python-cryptography? (we can't do that forever..) Major PITA.

Trying to use ctypes to wrap the string buffer does not help:

from ctypes import
bmsg = create_string_buffer(
v = enc.encrypt(bmsg)

Fails with the same error message, also tried feeding it a numpy array, memoryview. Same error. WTF? Also tried upgrading CFFI to the latest version.


r16044 contains many backports to the v1.0.x branch.


Mon, 12 Jun 2017 16:01:41 GMT - Antoine Martin:

Reverted back to python-cryptography 1.8.1 in r16047. We're stuck with this version for now, moving this to a new ticket: #1544


Wed, 21 Jun 2017 07:44:03 GMT - Antoine Martin:

Both should be backported.


Sat, 01 Jul 2017 08:54:11 GMT - Antoine Martin:

Updates:

Backports: r16148 (+r16154 + r16155), r16149, r16150 (+r16151 + r16152)


Sat, 01 Jul 2017 09:50:25 GMT - Antoine Martin: owner, status changed

AFAICT, all looks good - except for the CoreMedia linkage failure from comment:1 when building the 1.0.x branch on 10.5.x We just need a trivial sed -i -e s/-framework CoreMedia//g configure somewhere before running configure.

(and ignoring the lz4 build issue which seems to be this bug: setuptools_scm issue 164)

@smo: any ideas?


Wed, 05 Jul 2017 08:09:07 GMT - Antoine Martin:

Found those two packages by looking at all the .dylib and .so with otool, identifying those that link to libx264 and then finding the packages that contain those files.

But I had to do a full rebuild instead:

jhbuild build -f

Got a build error on cryptography:

building '_openssl' extension
/usr/bin/gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch x86_64 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.10 -I/Users/osx/gtk/inst/include -arch x86_64 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I/Users/osx/gtk/inst/include/python2.7 -c build/temp.macosx-10.10-x86_64-2.7/_openssl.c -o build/temp.macosx-10.10-x86_64-2.7/build/temp.macosx-10.10-x86_64-2.7/_openssl.o
build/temp.macosx-10.10-x86_64-2.7/_openssl.c:745:23: error: conflicting types for 'ASN1_TIME_to_generalizedtime'
ASN1_GENERALIZEDTIME *ASN1_TIME_to_generalizedtime(ASN1_TIME *,
                      ^
/Users/osx/gtk/inst/include/openssl/asn1.h:628:23: note: previous declaration is here
ASN1_GENERALIZEDTIME *ASN1_TIME_to_generalizedtime(const ASN1_TIME *t,
(..)
build/temp.macosx-10.10-x86_64-2.7/_openssl.c:64175:29: warning: passing 'X509_ALGOR **' (aka 'struct X509_algor_st **') to parameter of type
      'const X509_ALGOR **' (aka 'const struct X509_algor_st **') discards qualifiers in nested pointer types
      [-Wincompatible-pointer-types-discards-qualifiers]
  { X509_get0_signature(x0, x1, x2); }
                            ^~
/Users/osx/gtk/inst/include/openssl/x509.h:553:45: note: passing argument to parameter 'palg' here
                         const X509_ALGOR **palg, const X509 *x);
                                            ^
50 warnings and 1 error generated.

Which I will ignore for now...


Thu, 06 Jul 2017 08:55:42 GMT - Antoine Martin:

The openssl issue is tracked in this ticket: various backports for 1.8.2.

Built fine now, though I had to download the updated xpra.modules file by hand as jhbuild insisted on using the old one.. no idea why.


Sat, 08 Jul 2017 06:18:36 GMT - Antoine Martin:


Thu, 20 Jul 2017 07:16:49 GMT - Antoine Martin: status changed; resolution set

crickets - moving to 2.2: #1587


Sat, 23 Jan 2021 05:26:01 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1501