xpra icon
Bug tracker and wiki

Opened 4 months ago

Last modified 2 weeks ago

#1404 new task

2.0 moduleset updates

Reported by: Antoine Martin Owned by: Smo
Priority: major Milestone: 2.0
Component: platforms Version: trunk
Keywords: osx Cc:

Description

Now that we've dropped support for older build systems, we can update the modulesets with upstream changes again:

  • r14737: gettext, autoconf (safe for 1.x?)
  • r14738: websocket_client (will apply to 1.x)
  • r14740: libxml2 (safe for 1.x?), gettext (safe for 1.x), gtk-mac-integration (not safe), glib (not safe for 1.x), libffi (safe?), gobject-introspection (not safe), pango (safe?), gdk-pixbuf (not safe?), gtk+-3.0 (not used), pixman (safe?)

Change History (30)

comment:1 Changed 4 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

I encountered some build problems on "yelp-xsl", "pango" and "gdk-pixbuf":

dyld: Library not loaded: /Users/osx/gtk/inst/lib/libintl.8.dylib
  Referenced from: /Users/osx/gtk/inst/bin/xgettext
  Reason: image not found

Which clearly we're not the first ones to hit: dyld: Library not loaded: /usr/local/libodep/lib/libintl.8.dylib (and many more hits)

Fixed by rebuilding libcroco. Hit a similar thing later on "gtk+", solved by rebuilding "atk".

Then it gets wedged with gobject-introspection:

*** Configuring gobject-introspection *** [27/111]
(..)
checking for flex... flex
checking lex output file root... configure: error: cannot find output from flex; giving up

Fixed by re-installing flex. ("make install" from the source dir - odd that "jhbuild buildone flex" didn't work here, oh well)

Maybe I should have just ran everything from the top ("jhbuild bootstrap").

@smo: any changes to the list of safe/unsafe candidates for applying to the 1.x branch?

comment:2 Changed 4 months ago by Antoine Martin

Building from scratch ("jhbuild bootstrap" then "jhbuild build") works fine, just takes longer.

But trying to build the xpra package then fails with:

calling 'gtk-mac-bundler Xpra.bundle' in /Users/osx/Xpra-svn/trunk/osx
Traceback (most recent call last):
  File "/Users/osx/.local/bin/gtk-mac-bundler", line 6, in <module>
    bundler.main.main(sys.argv[1:])
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/main.py", line 18, in main
    bundler.run()
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/bundler.py", line 611, in run
    self.copy_binaries(self.project.get_binaries())
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/bundler.py", line 207, in copy_binaries
    dest = self.copy_path(path)
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/bundler.py", line 224, in copy_path
    source = self.project.evaluate_path(Path.source)
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/project.py", line 288, in evaluate_path
    path = utils.evaluate_pkgconfig_variables(path)
  File "/Users/osx/gtk/source/gtk-mac-bundler-0.7.3/bundler/utils.py", line 27, in evaluate_pkgconfig_variables
    raise Exception("pkg-config variable '%s %s' is undefined" % (key, module))
Exception: pkg-config variable 'pango_module_version pango' is undefined
ERROR: gtk-mac-bundler failed

Following the advice from Make pango modules related code optional to fix build with pango 1.38, r14746 removes pango from the bundle file.
We still need the patch to be applied to gtk-mac-bundler (it applies cleanly - skipping the examples), this needs to be added to the moduleset.

comment:3 Changed 4 months ago by Smo

r14752 adds patch for gtk-mac-bundler without examples
r14753 upgrades to newer version of gtk-mac-bundler and applies new patch also old patch is not necessary anymore

comment:4 Changed 4 months ago by Antoine Martin

One of those updates, likely r14753, broke our packaging scripts, r14760 fixes that.

comment:5 Changed 4 months ago by Smo

r14782 add patch for pygtkglext
r14785 renames patch
r14786 uses the patch in the moduleset
r14787 enabled dependency for pygobject3 which should build gtk3 properly now.

Last edited 4 months ago by Smo (previous) (diff)

comment:6 Changed 4 months ago by Smo

Owner: changed from Smo to Antoine Martin

comment:7 Changed 4 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

Works for me too, closed ticket:840#comment:37.

Can you comment on which libraries are safe to apply to 1.0? (see ticket description).
My guess: gettext and autoconf only? gdk-pixbuf?

comment:8 Changed 3 months ago by Smo

Sorry for the delay.

This is what I think

shoud be safe
gettext, autoconf, libxml2, gettext

maybe
pango, pixman, gdk-pixbuf

not safe
gtk-mac-integration, libffi

comment:9 Changed 3 months ago by Antoine Martin

Resolution: fixed
Status: newclosed
  • applied the "should be safe" in r14861
  • tested then applied the "maybe" list in r14862

Notes:

  • jhbuild build is not enough - you may get errors like in comment:1, best to start again from the top: jhbuild bootstrap -f;jhbuild build -f
  • rsvg was moaning:
    Making all in .
      CCLD     rsvg-view-3
    ld: library not found for -lgtk-3
    collect2: ld returned 1 exit status
    make[2]: *** [rsvg-view-3] Error 1
    make[1]: *** [all-recursive] Error 1
    make: *** [all] Error 2
    

Fixed by: jhbuild buildone -f gtk+-3.0

  • libtheora got stuck on SDL detection (odd - don't care)

comment:10 Changed 3 months ago by Antoine Martin

Was wrong about the pixman update, reverted in r14904.

comment:11 Changed 3 months ago by Antoine Martin

r14910 also updates the bundler and properly fixes the issues with the new pango version

comment:12 Changed 3 months ago by Antoine Martin

For the record: my VM got hosed by a system update.. (wouldn't boot no matter what bootloader or boot CD I tried to use..) so I reverted back to an earlier snapshot.. and jhbuild just fell over itself too many times trying to update + ugprade (libintl.8.dylib and more) so I decided to start again from scratch - a bad idea (tm).

That didn't work until I completely wiped the gtk directory. And even then it failed to build jpeg because nasm was the default system one (too old).
Got a few SSL certificate problems along the way too: cairographics.org, gstreamer.freedesktop.org, etc.. PITA
python-cryptography needs openssl, but somehow that wasn't built by then.. and then I found out that the current openssl version in the moduleset had been removed from the download location we use... so I bumped openssl in r14944. (we need permanent urls for everything instead)

I'll then forced a full rebuild (jhbuild build -f) at the end to ensure that everything is linked against the library version we ship. (ie: our newer openssl for python, etc)

comment:13 Changed 2 months ago by Antoine Martin

More in r15055:

  • ffmpeg 3.2.4
  • cryptography 1.7.2
  • python-xxhash 1.0.0

comment:14 Changed 2 months ago by Antoine Martin

Lots more updates in r15138 (copying changelog message):

  • sshpass 1.06
  • opus 1.1.4
  • setuptools 34.2.0
  • pycparser 2.17
  • cffi 1.9.1
  • curl 7.53.0
  • pyasn1 0.2.2
  • py2app 0.12
  • numpy 1.12
  • macholib 1.8
  • modulegraph 0.14
  • altgraph 0.13
  • ipaddress 1.0.18
  • idna 2.2
  • enum34 1.1.6
  • scons 2.5.1
  • subversion 1.9.5

Some of those should be backported sooner rather than later: sshpass, opus - and maybe also numpy.

Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:15 Changed 2 months ago by Antoine Martin

Resolution: fixed
Status: closedreopened

Two of those update are problematic:

I'd rather fix this rather than downgrade again, but can't spend too much time on this either.

comment:16 Changed 2 months ago by Antoine Martin

Owner: changed from Smo to Antoine Martin
Status: reopenednew

comment:17 Changed 2 months ago by Smo

r15143 changes setup tools to 33.1.1. There seems to be an issue with 34 which will take some more work to fix.

comment:18 Changed 2 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

Then it also seems to get confused about which platform it is running on and blows up some other win32 modules we don't care about. (removing them from the source tree helps, temporarily at least)
Downgrading to 0.11 still shows problems: it fails during its qt5 import recipes (missing 'sip' dependency)... but we don't even use qt, what are you doing there py2app?
So r15165 downgrades py2app from version 0.12 to 0.10, also adds a warning message in the moduleset file.

  • cffi: tried a few versions and only versions up to 1.5.2 allow us to package python-cryptography properly, so r15168 downgrades it and adds a note.

@smo: just a FYI, things should be packaged OK again. If you had rebuilt python-cryptography with a newer cffi, you may need to run

jhbuild buildone -f python-cryptography
Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:19 Changed 2 months ago by Smo

Everything builds fine. Even forced rebuild of cffi and python-cryptography but i'm seeing this unit test fail now.

======================================================================
ERROR: test_perf (__main__.TestCrypto)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/build/xpra-trunk/src/unittests/unit/net/crypto_test.py", line 31, in setUp
    pycryptography_backend.init()
  File "/Users/build/gtk/inst/lib/python2.7/site-packages/xpra/net/pycryptography_backend.py", line 52, in init
    backend = default_backend()
  File "/Users/build/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 71, in default_backend
    _default_backend = MultiBackend(_available_backends())
  File "/Users/build/gtk/inst/lib/python2.7/site-packages/cryptography/hazmat/backends/multibackend.py", line 36, in __init__
    "Multibackend cannot be initialized with no backends. If you "
ValueError: Multibackend cannot be initialized with no backends. If you are seeing this error when trying to use default_backend() please try uninstalling and reinstalling cryptography.

----------------------------------------------------------------------
Ran 2 tests in 5.384s

Maybe I need to do something more to get this to work properly again?

comment:20 Changed 2 months ago by Antoine Martin

The error in comment:19 is definitely the error that made me downgrade cffi in r15168.
You may need to ensure you don't have an old version laying around (remove all and re-install it). Then you may need to zap the python-cryptography source directory too.
If that still doesn't help, try zapping all the other downgrades (setuptools, etc).

comment:21 Changed 2 months ago by Smo

After deleting all directories having to do with setuptools, cffi and python-cryptography I was able to rebuild them then the unittests pass.

Must have been some issue with the downgrades.

I should note I also deleted stuff from

~/gtk/inst/lib/python2.7/site-packages

cffi*, cryptography* and setuptools*

comment:22 Changed 2 months ago by Smo

Even on newer versions of osx we still need the package tools to make the pkg.

Making Xpra-x86_64-2.0-r15188.pkg
229163 blocks
4 blocks
mkbom: invalid option -- u
Usage: mkbom [-h] [-s] directory bom
       mkbom [-s] -i filelist bom
PKG Signing skipped (no keyname)
33      ./image/Xpra-x86_64-2.0-r15188.pkg
Done PKG

I built them with this

jhbuild build meta-osx-xpra-pkgtools
Last edited 2 months ago by Smo (previous) (diff)

comment:23 Changed 2 months ago by Smo

Ran into this issue and it appears it is a problem with openssl 1.1. This might take some work to fix. Not sure at the moment. Going to check if we really need this on newer versions of osx.

checking for openssl/evp.h... yes
checking for OpenSSL_add_all_ciphers in -lcrypto... no
configure: error: Cannot build without libcrypto (OpenSSL)
*** Error during phase configure of xar: ########## Error running ./configure --prefix /Users/build/gtk/inst    *** [1/4]


*** module meta-osx-xpra-pkgtools not built due to non buildable xar *** [4/4]

comment:24 Changed 2 months ago by Smo

Making Xpra-x86_64-2.0-r15188.pkg
229148 blocks
4 blocks
PKG Signing skipped (no keyname)
34      ./image/Xpra-x86_64-2.0-r15188.pkg
Done PKG

r15189 comments out xar and adds a note

comment:25 Changed 2 months ago by Antoine Martin

The pygobject flags patch had gone MIA: I have removed it in the r15149 cleanup because it was not referenced anywhere since r13713, but that was a mistake: we need it to avoid the warnings (see ticket:1444#comment:2).
Re-added in r15191. I wasn't just seeing things then.

comment:26 Changed 7 weeks ago by Antoine Martin

We should do something about the git checkouts: either use a specific git id, or take a snapshot (version it ourselves). ie: libepoxy and others.
And since we also want to be able to apply patches.. maybe snapshots is the way to go?

comment:27 Changed 7 weeks ago by Smo

r15254 adds grep and pcre which is needed for grep

Added these because I needed them to rule out issues with cmake and configuring libyuv. The system grep doesn't support the '-P' option so it errors when running cmake for libyuv.

This didn't fix the issue with libyuv but I figure I should leave them in there in case we need them later.

As for the stuff from git goes if we need to apply any patches with the jhbuild system that doesn't work for us. The only way to do this is with tarball snapshots.

comment:28 Changed 5 weeks ago by Smo

I was updating my builds today and for some reason the dependency for setuptools_scm is being ignored for some reason I see that it was added for lz4 which is why I needed it.

It looks right in xpra.modules but it isn't doing what we want.

comment:29 Changed 5 weeks ago by Smo

Added libyuv now so we can build it within the moduleset and added it to the meta module so it should automatically get built.

details in #1280

comment:30 Changed 2 weeks ago by Antoine Martin

Follow up for 2.1: #1404

Please close unless something still needs doing?

Note: See TracTickets for help on using tickets.