xpra icon
Bug tracker and wiki

Opened 3 months ago

Closed 7 days ago

#1501 closed task (fixed)

2.1 moduleset updates

Reported by: Antoine Martin Owned by: Smo
Priority: critical Milestone: 2.1
Component: platforms Version: trunk
Keywords: osx Cc:

Description

Tracker ticket for the moduleset changes and those that may need to be backported. (see #1404 for 2.0)

Probably safe to backport to 1.x and 2.0.x:

Not needed:

Attachments (1)

build-1.0-with-newer-system.patch (1.2 KB) - added by Antoine Martin 3 months ago.
patch to allow us to build 1.0.x versions on a newer build system (libraries renamed or just not installed)

Download all attachments as: .zip

Change History (26)

Changed 3 months ago by Antoine Martin

patch to allow us to build 1.0.x versions on a newer build system (libraries renamed or just not installed)

comment:1 Changed 3 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

All built OK except for ffmpeg which tried to link with "-framework CoreMedia", removing all those from the configure script allowed it to build just fine.

beta 1.0.6 packages posted: http://xpra.org/beta/osx.

@smo: how do we patch the configure script? (ideally using sed rather than a patch which may not apply cleanly with updates)

comment:2 Changed 3 months ago by Antoine Martin

Important update from upstream: Removal of support for obsolete MacOS X Versions and global module update: I've completed the removal of support for versions of MacOS X earlier than 10.8 from gtk-osx. If you still need to build for 10.4u or 10.5, or on 10.4-10.7, I've tagged the last commit that supports that with
"Last-Lion" to make it easy to have a jhbuildrc and modulesets to support your needs.

comment:3 Changed 2 months ago by Antoine Martin

@smo: how to I update nasm? nasm 2.13 is out.

Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:4 Changed 2 months ago by Antoine Martin

Owner: changed from Smo to Antoine Martin
Status: newassigned
  • r15857: libepoxy 1.4.2
  • r15856: try to switch to gstreamer 1.12.0, reverted in r15858 (see below) then bumped to 1.10.4 in r15859.

I had to fix the gstreamer 1.12.0 osxaudio compilation in gst-plugins-good to avoid this error:

$ gst-launch-1.0 audiotestsrc ! osxaudiosink

(gst-plugin-scanner:86177): GStreamer-WARNING **: Failed to load plugin \
    '/Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so': dlopen(/Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so, 2): \
    Symbol not found: _AudioComponentFindNext
  Referenced from: /Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so (which was built for Mac OS X 10.11)
  Expected in: /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
 in /Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so

(gst-plugin-scanner:86177): GStreamer-WARNING **: Failed to load plugin \
    '/Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so': dlopen(/Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so, 2): \
    Symbol not found: _AudioComponentFindNext
  Referenced from: /Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so (which was built for Mac OS X 10.11)
  Expected in: /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
 in /Users/osx/gtk/inst/lib/gstreamer-1.0/libgstosxaudio.so
WARNING: erroneous pipeline: no element "osxaudiosink"

I fixed this by adding -framework AudioUnit -framework CoreAudio -framework AudioToolbox to the LDFLAGS in the sys/osxaudio/Makefile. (inspired by OSX 10.6 gnuradio: Audio Library Error)

Not sure why this has come up now.
And then... things get worse: most of the muxers no longer work: ogg and mka formats all fail with the dreaded error:

sound output pipeline error: gst-stream-error-quark: Internal data stream error. (1)
sound output  appsrc do-timestamp=1 name=src emit-signals=0 block=0 is-live=0 stream-type=0 format=4 ! \
    matroskademux ! opusdec ! audioconvert ! audioresample ! volume name=volume volume=0 ! \
    queue name=queue min-threshold-time=0 max-size-buffers=0 max-size-bytes=0 max-size-time=450000000 leaky=2 ! \
    osxaudiosink async=True qos=True sync=False
sound output  gstbasesrc.c(2939)
sound output  void gst_base_src_loop(GstPad *) ()
sound output  /GstPipeline:pipeline0/GstAppSrc:src:
sound output  streaming stopped, reason not-negotiated (-4)

Which is why this was reverted.

comment:5 Changed 2 months ago by Antoine Martin

Ended up starting from scratch again because gstreamer failed again at runtime.

Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:6 Changed 2 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo
Status: assignednew

Panic is over, things build fine again, provided that:

  • don't run jhbuild from its own shell - nesting doesn't work anymore (should file ticket upstream)
  • make sure that the .jhbuildrc-custom customization looks more or less like this one:
    _gtk_osx_use_jhbuild_python = True
    
    setup_sdk(target="10.10", sdk_version="10.10", architectures=["x86_64"])
    os.environ["CC"] = "/usr/bin/gcc"
    os.environ["DYLD_LIBRARY_PATH"] = ""
    build_policy = "updated-deps"
    
    modules = [ "openssl", "python", "yasm", "nasm", "libxml2", "itstool", "gtk-doc",
    "meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
    "libpng", "python-setuptools-scm",
    "libcroco", "librsvg", "meta-gtk-osx-python", "meta-gtk-osx-themes",
    "gtk-quartz-engine", "gtk-mac-integration-python", "meta-osx-xpra-deps",
    "meta-subversion-xpra", "meta-osx-xpra-pkgtools"]
      
    moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
    
    os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"
    

(some of the modules should be picked up automatically but for some unknown reason aren't: "python-setupttols-scm" should get built before "python-lz4" automatically for example)

Confirmed that the gstreamer problems occur when switching to 1.12.0, though it could still be a problem with the toolchain / leftovers from the previous version.


r15865 adds a module for "brotli", but the Makefile doesn't have an "install" command, so this fails.
I can install it by appending this to the Makefile by hand:

install :
	install -d $(BINDIR) $(JHBUILD_PREFIX)/bin
	install -m 0755 $(BINDIR)/bro $(JHBUILD_PREFIX)/bin/

But this then fails when running it from jhbuild with:

W: Ignoring uninstalled package: brotli
Traceback (most recent call last):
  File "/Users/osx/.local/bin/jhbuild", line 32, in <module>
    jhbuild.main.main(sys.argv[1:])
  File "/Users/osx/Source/jhbuild/jhbuild/main.py", line 155, in main
    rc = jhbuild.commands.run(command, config, args, help=lambda: print_help(parser))
  File "/Users/osx/Source/jhbuild/jhbuild/commands/__init__.py", line 183, in run
    return cmd.execute(config, args, help)
  File "/Users/osx/Source/jhbuild/jhbuild/commands/__init__.py", line 52, in execute
    return self.run(config, options, args, help)
  File "/Users/osx/Source/jhbuild/jhbuild/commands/base.py", line 268, in run
    return build.build()
  File "/Users/osx/Source/jhbuild/jhbuild/frontends/buildscript.py", line 163, in build
    error, altphases = module.run_phase(self, phase)
  File "/Users/osx/Source/jhbuild/jhbuild/modtypes/__init__.py", line 420, in run_phase
    method(buildscript)
  File "/Users/osx/Source/jhbuild/jhbuild/modtypes/autotools.py", line 314, in do_install
    self.process_install(buildscript, self.get_revision())
  File "/Users/osx/Source/jhbuild/jhbuild/modtypes/__init__.py", line 316, in process_install
    new_contents = fileutils.accumulate_dirtree_contents(destdir_prefix)
  File "/Users/osx/Source/jhbuild/jhbuild/utils/fileutils.py", line 45, in accumulate_dirtree_contents
    _accumulate_dirtree_contents_recurse(path, contents)
  File "/Users/osx/Source/jhbuild/jhbuild/utils/fileutils.py", line 27, in _accumulate_dirtree_contents_recurse
    names = os.listdir(path)
OSError: [Errno 2] No such file or directory: '/Users/osx/gtk/inst/_jhbuild/root-brotli/Users/osx/gtk/inst'

@smo: how do I make it play nice?

Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:7 Changed 2 months ago by Smo

*** Installing brotli *** [1/1]
make DESTDIR=/Users/spikesinc/gtk/inst/_jhbuild/root-brotli install

Have to install to DESTDIR instead and that should work

comment:8 Changed 2 months ago by Smo

Easier way is to just use cmake it would seem

comment:9 Changed 2 months ago by Smo

change brotli to cmake in r15880

comment:10 Changed 2 months ago by Smo

I had a few issues myself when building all of this from the start again. First off I change my .jhbuildrc-custom to be like yours and downloaded the cacert.pem. This worked nicely.

*** Checking out glib *** [20/134]
curl --continue-at - -L http://ftp.gnome.org/pub/GNOME/sources/glib/2.48/glib-2.48.0.tar.xz -o /Users/spikesinc/gtk/source/pkgs/glib-2.48.0.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   377  100   377    0     0    577      0 --:--:-- --:--:-- --:--:--   576
100 7312k  100 7312k    0     0  1727k      0  0:00:04  0:00:04 --:--:-- 2685k
xzcat -d "/Users/spikesinc/gtk/source/pkgs/glib-2.48.0.tar.xz" | tar xf -
*** Error during phase checkout of glib: could not download patch (error: 404) *** [20/134]

Ran into one of the patches missing so I will have to look into what that was. I was able to just start the build again and continue.

I also had issues where some of the compiling processes would get killed? Restarting the build usually got me by these issues.

python-setuptools-scm tried to install before setuptools was installed which caused it to produce this stacktrace

Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    import setuptools
ImportError: No module named setuptools

Also came across this error trying to install python-zeroconf not sure what to do about this one.

*** Building python-zeroconf *** [87/134]
python setup.py build
Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    [l for l in open(join(PROJECT_ROOT, 'zeroconf.py')) if '__version__' in l][0]
LookupError: unknown encoding: 
*** Error during phase build of python-zeroconf: ########## Error running python setup.py build *** [87/134]
Last edited 2 months ago by Smo (previous) (diff)

comment:11 Changed 2 months ago by Smo

I guess that was too easy. Still have this issue with brotli

[JH][~/xpra-trunk/osx/jhbuild]$ bro
dyld: Library not loaded: libbrotlienc.0.6.0.dylib
  Referenced from: /Users/build/gtk/inst/bin/bro
  Reason: image not found
Trace/BPT trap: 5
Last edited 2 months ago by Smo (previous) (diff)

comment:12 Changed 2 months ago by Antoine Martin

I don't have my macos VM with me so I cannot test today, but maybe the cmake build is different (I don't think the pure "make" build uses shared libraries), and maybe the shared library isn't installed, or not properly installed?

comment:13 Changed 2 months ago by Antoine Martin

I also had issues where some of the compiling processes would get killed? Restarting the build usually got me by these issues.

OOM killer maybe? I'm giving this build VM lots of cpus and tons of ram..

python-setuptools-scm tried to install before setuptools was installed which caused it to produce this stacktrace

Adding "python-setuptools" before "python-setuptools-scm" in jhbuildrc-custom should fix that.

Also came across this error trying to install python-zeroconf not sure what to do about this one.

Weird one. I can't reproduce. Could be something in your environment which is different.

I guess that was too easy. Still have this issue with brotli

Are you trying to run it from within a jhbuild shell?
(I got the same error when I tried from outside the jhbuild shell env)

comment:14 Changed 8 weeks ago by Antoine Martin

Updates (all safe to backport and recommended):

TODO:

  • lz4 is broken:
    *** Building python-lz4 *** [99/122]
    python setup.py build
    Traceback (most recent call last):
      File "setup.py", line 161, in <module>
        'Programming Language :: Python :: 3.6',
      File "/Users/osx/gtk/inst/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "/Users/osx/gtk/inst/lib/python2.7/site-packages/setuptools/dist.py", line 271, in __init__
        _Distribution.__init__(self,attrs)
      File "/Users/osx/gtk/inst/lib/python2.7/distutils/dist.py", line 287, in __init__
        self.finalize_options()
      File "/Users/osx/gtk/inst/lib/python2.7/site-packages/setuptools/dist.py", line 326, in finalize_options
        ep.load()(self, ep.name, value)
      File "/Users/osx/gtk/inst/lib/python2.7/site-packages/setuptools_scm/integration.py", line 19, in version_keyword
        dist.metadata.version = get_version(**value)
      File "/Users/osx/gtk/inst/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 117, in get_version
        parsed_version = _do_parse(root, parse)
      File "/Users/osx/gtk/inst/lib/python2.7/site-packages/setuptools_scm/__init__.py", line 95, in _do_parse
        "use git+https://github.com/user/proj.git#egg=proj" % root)
    LookupError: setuptools-scm was unable to detect version for '/Users/osx/gtk/source/python-lz4-0.9.4'.
    
  • tiff is broken:
    *** Configuring libtiff *** [16/122]
    ./configure --prefix /Users/osx/gtk/inst --without-x  
    (...)
    config.status: error: cannot find input file: `html/Makefile.in'
    *** Error during phase configure of libtiff: ########## Error running ./configure --prefix /Users/osx/gtk/inst --without-x   *** [16/122]
    
Last edited 7 weeks ago by Antoine Martin (previous) (diff)

comment:15 Changed 6 weeks ago by Antoine Martin

Last edited 6 weeks ago by Antoine Martin (previous) (diff)

comment:16 Changed 6 weeks ago by Antoine Martin

Owner: changed from Smo to Antoine Martin
Status: newassigned

FFS: I fixed most of the modules, see comment:15 but now we have a new packaging problem with python-cryptography (detected by the unit tests):

ERROR: unit tests failed, see /Users/osx/Xpra-svn/trunk/src/unittests/unittest.log:
Traceback (most recent call last):
  File "/Users/osx/Xpra-svn/trunk/src/unittests/unit/net/crypto_test.py", line 30, in setUp
    pycryptography_backend.init()
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/xpra/net/pycryptography_backend.py", line 52, in init
    backend = default_backend()
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 16, in <module>
    from cryptography import utils, x509
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/x509/__init__.py", line 8, in <module>
    from cryptography.x509 import certificate_transparency
ImportError: cannot import name certificate_transparency

comment:17 Changed 6 weeks ago by Antoine Martin

Priority: majorcritical

Figuring out the new dependency:

$ python -c "from cryptography.hazmat.backends.openssl.backend import backend"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 16, in <module>
    from cryptography import utils, x509
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/x509/__init__.py", line 9, in <module>
    from cryptography.x509.base import (
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/x509/base.py", line 16, in <module>
    from cryptography.x509.extensions import Extension, ExtensionType
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/x509/extensions.py", line 13, in <module>
    from asn1crypto.keys import PublicKeyInfo
ImportError: No module named asn1crypto.keys

So r16045 adds asn1crypto to the moduleset.

But then we hit a new problem:

ERROR: test_backends (__main__.TestCrypto)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/osx/Xpra-svn/trunk/src/unittests/unit/net/crypto_test.py", line 83, in test_backends
    self.do_test_backend()
  File "/Users/osx/Xpra-svn/trunk/src/unittests/unit/net/crypto_test.py", line 63, in do_test_backend
    v = enc.encrypt(message)
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/primitives/ciphers/base.py", line 149, in update
    return self._ctx.update(data)
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py", line 116, in update
    n = self.update_into(data, buf)
  File "/Users/osx/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/ciphers.py", line 127, in update_into
    "unsigned char *", self._backend._ffi.from_buffer(buf)
TypeError: from_buffer() cannot return the address of the raw string within a str or unicode or bytearray object

Looks like we can't feed it a string or bytearray now.
Discussion here: Creating a cdata from a buffer
This is a CFFI issue... and we're stuck on version 1.5.2 (see ticket:1404#comment:18 for details).
Downgrade python-cryptography? (we can't do that forever..) Major PITA.

Trying to use ctypes to wrap the string buffer does not help:

from ctypes import 
bmsg = create_string_buffer(
v = enc.encrypt(bmsg)

Fails with the same error message, also tried feeding it a numpy array, memoryview. Same error. WTF?
Also tried upgrading CFFI to the latest version.


r16044 contains many backports to the v1.0.x branch.

Last edited 6 weeks ago by Antoine Martin (previous) (diff)

comment:18 Changed 6 weeks ago by Antoine Martin

Reverted back to python-cryptography 1.8.1 in r16047.
We're stuck with this version for now, moving this to a new ticket: #1544

comment:19 Changed 5 weeks ago by Antoine Martin

Both should be backported.

Last edited 4 weeks ago by Antoine Martin (previous) (diff)

comment:20 Changed 4 weeks ago by Antoine Martin

Updates:

Backports: r16148 (+r16154 + r16155), r16149, r16150 (+r16151 + r16152)

Last edited 4 weeks ago by Antoine Martin (previous) (diff)

comment:21 Changed 4 weeks ago by Antoine Martin

Owner: changed from Antoine Martin to Smo
Status: assignednew

AFAICT, all looks good - except for the CoreMedia linkage failure from comment:1 when building the 1.0.x branch on 10.5.x
We just need a trivial sed -i -e s/-framework CoreMedia//g configure somewhere before running configure.

(and ignoring the lz4 build issue which seems to be this bug: setuptools_scm issue 164)

@smo: any ideas?

Last edited 3 weeks ago by Antoine Martin (previous) (diff)

comment:22 Changed 3 weeks ago by Antoine Martin

  • r16187 removes python-pam (follow up from r16184) - not to be backported
  • r16189 Pillow 4.2.0
  • r16188 new x264 snapshot, I thought this rebuild would be enough to fix some bundler errors on libx264:
    jhbuild build -f ffmpeg
    jhbuild build -f gst-plugins-ugly-1.0
    

Found those two packages by looking at all the .dylib and .so with otool, identifying those that link to libx264 and then finding the packages that contain those files.

But I had to do a full rebuild instead:

jhbuild build -f

Got a build error on cryptography:

building '_openssl' extension
/usr/bin/gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch x86_64 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.10 -I/Users/osx/gtk/inst/include -arch x86_64 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I/Users/osx/gtk/inst/include/python2.7 -c build/temp.macosx-10.10-x86_64-2.7/_openssl.c -o build/temp.macosx-10.10-x86_64-2.7/build/temp.macosx-10.10-x86_64-2.7/_openssl.o
build/temp.macosx-10.10-x86_64-2.7/_openssl.c:745:23: error: conflicting types for 'ASN1_TIME_to_generalizedtime'
ASN1_GENERALIZEDTIME *ASN1_TIME_to_generalizedtime(ASN1_TIME *,
                      ^
/Users/osx/gtk/inst/include/openssl/asn1.h:628:23: note: previous declaration is here
ASN1_GENERALIZEDTIME *ASN1_TIME_to_generalizedtime(const ASN1_TIME *t,
(..)
build/temp.macosx-10.10-x86_64-2.7/_openssl.c:64175:29: warning: passing 'X509_ALGOR **' (aka 'struct X509_algor_st **') to parameter of type
      'const X509_ALGOR **' (aka 'const struct X509_algor_st **') discards qualifiers in nested pointer types
      [-Wincompatible-pointer-types-discards-qualifiers]
  { X509_get0_signature(x0, x1, x2); }
                            ^~
/Users/osx/gtk/inst/include/openssl/x509.h:553:45: note: passing argument to parameter 'palg' here
                         const X509_ALGOR **palg, const X509 *x);
                                            ^
50 warnings and 1 error generated.

Which I will ignore for now...

comment:23 Changed 3 weeks ago by Antoine Martin

The openssl issue is tracked in this ticket: fix compilation on 1.1.0f and the fix is included in 1.8.2: various backports for 1.8.2.

Built fine now, though I had to download the updated xpra.modules file by hand as jhbuild insisted on using the old one.. no idea why.

comment:24 Changed 3 weeks ago by Antoine Martin

  • r16220: libjpeg-turbo 1.5.2 (safe to backport)

comment:25 Changed 7 days ago by Antoine Martin

Resolution: fixed
Status: newclosed

crickets - moving to 2.2: #1587

Note: See TracTickets for help on using tickets.