xpra icon
Bug tracker and wiki

Opened 16 months ago

Closed 5 months ago

#1575 closed task (fixed)

python3 packaging for macos

Reported by: Antoine Martin Owned by: Smo
Priority: blocker Milestone: 2.3
Component: packaging Version: trunk
Keywords: Cc:

Description

python3 tracker ticket: #1568.

TODO:

  • update moduleset to latest python (ie: 3.6.x, currently building 3.4!)
  • add all dependencies to moduleset
  • add a new meta module for python3 xpra
  • py2app packaging

etc..

Change History (27)

comment:1 Changed 16 months ago by Antoine Martin

Status: newassigned

This may fix issues like #761 and #404, worth a try.

Does the GTK3 macos port support opengl though? (#1569)

Last edited 12 months ago by Antoine Martin (previous) (diff)

comment:2 Changed 12 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo
Status: assignednew

@smo: any updates?

comment:3 Changed 11 months ago by Antoine Martin

Priority: majorcritical

Needed for window transparency: #1570

comment:4 Changed 11 months ago by Antoine Martin

Milestone: 3.02.3

comment:5 Changed 11 months ago by Antoine Martin

See also #1708: 2.3 moduleset updates

comment:6 Changed 10 months ago by Antoine Martin

Priority: criticalblocker

@smo: please put anything you have already done in this ticket, then I can take a look.

comment:7 Changed 10 months ago by Smo

I tried to build the up to date moduleset but had a problem with pygobject. Here is my contact to the mailing list which didn't really go anywhere.

https://mail.gnome.org/archives/gtk-osx-devel-list/2017-September/msg00000.html

comment:8 Changed 10 months ago by Antoine Martin

Owner: changed from Smo to Antoine Martin
Status: newassigned

Using a clean user account and following the build instructions from https://wiki.gnome.org/Projects/GTK+/OSX/Building:

wget --no-check-certificate \
    "https://git.gnome.org/browse/gtk-osx/plain/gtk-osx-build-setup.sh"
sh ./gtk-osx-build-setup.sh
PATH=/Users/gtk3/.local/bin:$PATH
jhbuild bootstrap

Got a few certificate errors with: xz, pkg-config, flex, expat, etc (download without checking certs and continue).
Then it fails to build python3 with another cert error:

jhbuild buildone python3
jhbuild buildone: could not download http://git.gnome.org/browse/gtk-osx/plain/modulesets-stable/gtk-osx.modules: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

This time fixed by downloading the cacert.pem from here:

https://curl.haxx.se/docs/caextract.html and then telling jhbuild to use it:

CA_CERT_FILE=$HOME/cacert.pem jhbuild build python3

Make python3 the default (as per docs - though the upstream instructions look wrong for the second symlink):

jhbuild shell
cd $JHBUILD_PREFIX/bin
ln -s python3 python
ln -s python3-config python-config
jhbuild build openssl

(upstream notes apply: interrupt the build and run it by hand - we should upstream our openssl 1.1 build commands)

jhbuid build meta-gtk-osx-bootstrap meta-gtk-osx-gtk3

Got a minor failure building glib:

xzcat -d "/Users/gtk3/gtk/source/pkgs/glib-2.52.2.tar.xz" | tar xf -
*** Applying patch https://git.gnome.org/browse/gtk-osx/plain/patches/glib-gint64-long-long.patch *** [14/39]
patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-gint64-long-long.patch"
patching file configure.ac
Hunk #1 succeeded at 3002 (offset 48 lines).
Hunk #2 succeeded at 3114 (offset 48 lines).
Hunk #3 succeeded at 3266 (offset 49 lines).
*** Applying patch https://git.gnome.org/browse/gtk-osx/plain/patches/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch *** [14/39]
patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch"
patching file gio/gosxappinfo.c
patching file gio/gosxcontenttype.c
Hunk #1 succeeded at 52 (offset -6 lines).
patching file gio/tests/contenttype.c
Hunk #1 succeeded at 323 with fuzz 1 (offset -38 lines).
Hunk #2 FAILED at 359.
1 out of 2 hunks FAILED -- saving rejects to file gio/tests/contenttype.c.rej
*** Error during phase checkout of glib: ########## Error running patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch" *** [14/39]

Applied the one line from the reject file by hand and continued.

Everything built correctly after that.

@smo: whatever you did (which was not recorded anywhere AFAICT) must have been different?


Continuing from there:

  • r17840 makes the moduleset more reusable
  • add our moduleset to jhbuild:
    cat > $HOME/.jhbuildrc-custom << EOF
    moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
    EOF
    
  • and build some things we will need:
    jhbuild buildone sshpass -f
    jhbuild build meta-subversion-xpra
    jhbuild build meta-osx-xpra-pkgtools
    jhbuild build meta-osx-xpra-codec-deps
    

Got one failure we should fix on lame:

gzip -dc "/Users/gtk3/gtk/source/pkgs/lame-3.99.5.tar.gz" | tar xf -
*** Applying patch http://xpra.org/svn/Xpra/trunk/osx/jhbuild/patches/lame-channels.patch *** [8/23]
patch -p0 < "/Users/gtk3/.cache/jhbuild/lame-channels.patch"
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/libmp3lame/set_get.c
|+++ b/libmp3lame/set_get.c
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
patch: **** malformed patch at line 10:

Next we have to build python and switch back to python3 afterwards so we can build the python deps against python3:

jhbuild buildone python
jhbuild shell
cd $JHBUILD_PREFIX/bin
rm python python-config
ln -s python3 python
ln -s python3-config python-config

Build the rest:

jhbuild build meta-osx-xpra-python-deps

lz4 had a problem (will fix).
Then python3 had a problem finding ssl, it just needed to be rebuilt now that we brought in a newer openssl lib version:

jhbuild buildone python3 -f

At that point, everything is built OK, AFAICT.

Still TODO:

  • merge stable moduleset updates (oh joy..)
  • add python3 / gtk3 macos handling to our setup.py

comment:9 Changed 10 months ago by Antoine Martin

@smo: why are we so far behind https://git.gnome.org/browse/gtk-osx/tree/?
Even gtk2 is outdated here. (only our gstreamer and network modulesets are slightly ahead)
Is there anything there that we should not merge and why?

I've merged most of the changes:

  • r17846: mostly things we don't even use - and maybe we shouldn't copy them then, just use an https module reference and be done with it - and we can still override some packages with newer ones, only when we need to keep an older version do we really need to copy the moduleset, right?
  • r17847: bootstrap, network, python
  • r17848: main gtk-osx moduleset
  • r17849: glib for gobject-introspection (will check the python2 re-build later)
  • r17850 + r17851: moduleset updates for packaging tools
  • r17855: gtkosx_application import for GTK3 (gi bindings)
  • r17856: python-cryptography new requirement?
  • r17857: tiny py3k string fix
  • r17858 + r17859: force py2app to not include the desktop server (raising ImportError does it)
  • r17860: a really annoying issue is that although the headers look the same, py3cairo and pycairo install in different places, but our source needs to reference just one header file.. so this a manual step is still needed:
    ln -sf py3cairo.h /Users/gtk3/gtk/inst/include/pycairo/pycairo.h
    

Also added latest gdb by hand:

wget https://ftp.gnu.org/gnu/gdb/gdb-8.0.1.tar.gz
tar -zxvf gdb-8.0.1.tar.gz 
cd gdb-8.0.1
./configure --prefix=${JHBUILD_PREFIX}
make && make install
sudo chgrp procmod ${JHBUILD_PREFIX}/bin/gdb
sudo chmod g+s ${JHBUILD_PREFIX}/bin/gdb

But... still needs signing to be used: codesign gdb.

comment:10 Changed 10 months ago by Antoine Martin

More work on this:

  • r17861: gtk-mac-bundler requires python2 to run
  • r17864: yet another new python-lz4 dependency (sigh), r17868 + r17872: macos packaging for it
  • r17870: support new versions of python-lz4 (applies to all arches)
  • r17866: gtk-quartz-engine isn't in the moduleset any more? it is gtk2 and contains build errors... remove it
  • r17865: fixup r17855 gtkosx application import
  • r17867 + r17881: packaging tweaks: ship more gi bindings, skip (py)gtk2 stuff, etc
  • r17876: clipboard support for gtk3
  • r17877 + r17878: tray, clipboard and gui porting

Still TODO:

  • gdk bindings need porting (input devices stuff, see ticket:1157#comment:24) unless GTK3 gives us the devices and the correct values out of the box?
  • opengl: need to implement the gl backend glue, unfortunately there is no easy way to get the "nsview" from a gdk window in GTK3... so we may have to use a Cython class
  • fix the GtkMenuBar invalid cast warnings, somehow

Apart from that, it looks usable. Beta builds posted here with the "Python3" filename: https://xpra.org/beta/osx/.

comment:11 Changed 10 months ago by Antoine Martin

Notes after rebuilding again from scratch to verify:

  • pygobject3 needed this hack to avoid errors on "stack_size":
    find . -name "Makefile" -exec sed -i '' 's+-Wl,-stack_size,1000000 ++g' {} \;
    
  • r17883 + r17884: re-instate openssl 1.1
  • r17885 + r17887: better openssl config command, fix "lame" patch context stripping
  • r17886: dependency tweaks
  • r17888: don't duplicate modulesets we don't need or modify, just reference them
  • r17889: follow upstream and remove "1.0" suffix from all gstreamer modules names
  • r17891: avoid runtime warning with GTK3
  • r17892: remove python2 ssl file path (may need to be re-added via the shell script)
  • r17893: fix syntax error in moduleset
  • r17894: move all python modules to a new moduleset file we include
  • r17895 + r17896 + r17897 + r17898: python3 version of the moduleset
  • r17899: cosmetic
  • we should fix xar: Fails to build against OpenSSL version 1.1.0g
  • libxml2 can be built against python3 with this simple hackish patch (found here: Bug 1406920 - libxml2 should no longer use PyVerify_fd):
    +--- a/libxml2-2.9.4/python/types.c	2016-02-09 03:17:33.000000000 -0700
    ++++ b/libxml2-2.9.4/python/types.c	2016-12-21 12:34:06.755650986 -0700
    +@@ -31,8 +31,6 @@
    +     const char *mode;
    + 
    +     fd = PyObject_AsFileDescriptor(f);
    +-    if (!_PyVerify_fd(fd))
    +-        return(NULL);
    +     /*
    +      * Get the flags on the fd to understand how it was opened
    +      */
    

Saves having to temporarily switch back to python2 to build it.

The python2-only builds started hitting the issue from comment:7, and now the mixed python2-python3 builds do too. Sigh.

One more weird issue started happening: the "py2app" step does something that loads gdk and tries to access the display server and crashes when it fails, means that the builds now have to be executed from a GUI shell. PITA.


So at this point, the python3 jhbuild env is almost straightforward (bar python3 vs python issues) with this file:

cat > .jhbuildrc-custom  <<EOF
_gtk_osx_use_jhbuild_python = True
setup_sdk(target="10.10", sdk_version="10.10", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "openssl", "python3", "yasm", "nasm", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core", "meta-gtk-osx-gtk3",
"libpng",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-mac-integration-python", "meta-osx-python3-xpra-deps"]
moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"
EOF
Last edited 10 months ago by Antoine Martin (previous) (diff)

comment:12 Changed 10 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo
Status: assignednew

Lots more updates (in large part because testing jhbuild is hard, so I commit then fix the numerous mistakes one at a time).
I've also sent some questions and pull requests upstream to try to reduce our diff:

Changesets:

With all these changes:

  • the modulesets are cleaner, and closer to upstream
  • the differences between the python2 and python3 / gtk3 modulesets are easier to see and maintain
  • building everything from scratch is easier:
    cat > .jhbuildrc-custom <<EOF
    _gtk_osx_use_jhbuild_python = True
    setup_sdk(target="10.10", sdk_version="10.10", architectures=["x86_64"])
    os.environ["CC"] = "/usr/bin/gcc"
    os.environ["DYLD_LIBRARY_PATH"] = ""
    build_policy = "updated-deps"
    modules = ["meta-osx-xpra-deps"]
    moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtk3.modules"
    os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"
    EOF
    

Then just: jhbuild build.
The only difference for the "GTK2" moduleset is the moduleset URL:

moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtk2.modules"

Then when building the packages, we have to do:

PYTHON=python3 ./make-all.sh

(or PYTHON=python2 for GTK2 builds)

@smo: does that work for you?

comment:13 Changed 10 months ago by Antoine Martin

Note: the gi bingings were missing from the python2 builds (meaning no sound), to re-instate them I had to apply this patch: pygobject: remove references to deprecated GI_INFO_TYPE_ERROR_DOMAIN. (basically, just remove all references to GI_INFO_TYPE_ERROR_DOMAIN then rebuild with introspection enabled)
There might be a better way of doing this.

comment:14 Changed 9 months ago by Smo

gtk-engines fails when building for python3/gtk3 is this needed?

checking for GTK... no
configure: error: GTK+-2.12 is required to compile gtk-engines
*** Error during phase configure of gtk-engines: ########## Error running ./configure --prefix /Users/xpragtk3/gtk/inst    *** [47/129]

comment:15 Changed 9 months ago by Antoine Martin

gtk-engines fails when building for python3/gtk3 is this needed?

It isn't referenced anywhere in our modulesets.
Can you get some info on it? jhbuild info gtk-engines

comment:16 Changed 9 months ago by Smo

Name: gtk-engines
Module Set: gtk-osx-themes
Type: autogen
Install version: not installed
Install date: not installed
URL: http://ftp.gnome.org/pub/GNOME/sources/gtk-engines/2.20/gtk-engines-2.20.2.tar.bz2
Version: 2.20.2
Tree-ID: 2.20.2-d41d8cd98f00b204e9800998ecf8427e
Sourcedir: /Users/xpragtk3/gtk/source/gtk-engines-2.20.2
Requires: automake, libtool, make
Required by: meta-gtk-osx-themes
After: meta-gtk-osx-core

comment:17 Changed 9 months ago by Antoine Martin

meta-gtk-osx-themes is found here: https://github.com/GNOME/gtk-osx/blob/master/modulesets-stable/gtk-osx-themes.modules, but isn't referenced anywhere else in the modulesets.
There is a meta-gtk-osx-gtk3-core-themes, so I've switched to that in r18020.
@smo: does that fix things? Do we need to add icon-naming-utils?

comment:18 Changed 5 months ago by Smo

Started from scratch again today ran into some issues with libxml2. It would always just build the python2 variant and then would also fail with

ld: -stack_size option can only be used when linking a main executable

We can see the configure line here

./configure --prefix /Users/xpragtk3/gtk/inst --libdir="$JHBUILD_LIBDIR" --with-python  --with-python-install-dir=/Users/xpragtk3/gtk/inst/lib/python2.7/site-packages

I'm not sure where the --with-python-install-dir comes from it doesn't appear in the moduleset?

If you run configure again and change it to the correct python install dir it seems to install minus having to stop the build to use the same line as comment:11

python -c "import libxml2; print(libxml2)"
<module 'libxml2' from '/Users/xpragtk3/gtk/inst/lib/python3.6/site-packages/libxml2.py'>

seems to load properly now

Last edited 5 months ago by Smo (previous) (diff)

comment:19 Changed 5 months ago by Smo

itstool 2.0.2 doesn't work with python3
itstool 2.0.4 works with python3 but segfaults when used building gtk-doc
itstool from git seems to work just fine had to install this manually

comment:20 Changed 5 months ago by Smo

seems to be some problems with building lame still as well

** Checking out lame *** [105/135]
curl --continue-at - -L http://winswitch.org/src/lame-3.100.tar.gz -o /Users/xpragtk3/gtk/source/pkgs/lame-3.100.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3983  100  3983    0     0  11362      0 --:--:-- --:--:-- --:--:-- 11380
*** Error during phase checkout of lame: file hash is incorrect (expected ddfe36cab873794038ae2c1210557ad34857a4b6bdc515785d1da9e175b1da1e, got 30641cf26a0d4dc26ca6b97e719c52b4c72d49014a09cd28309a9d989a0408a2) *** [105/135]

Hash is wrong but the url is actually invalid for the download. Should this be switched or was there a good reason for having it on winswitch.org?

*** Checking out lame *** [105/135]
gzip -dc "/Users/xpragtk3/gtk/source/pkgs/lame-3.100.tar.gz" | tar xf -
*** Applying patch http://xpra.org/svn/Xpra/trunk/osx/jhbuild/patches/lame-channels.patch *** [105/135]
patch -p1 < "/Users/xpragtk3/.cache/jhbuild/lame-channels.patch"
patching file libmp3lame/set_get.c
Hunk #1 FAILED at 91.
1 out of 1 hunk FAILED -- saving rejects to file libmp3lame/set_get.c.rej
*** Error during phase checkout of lame: ########## Error running patch -p1 < "/Users/xpragtk3/.cache/jhbuild/lame-channels.patch" *** [105/135]

After getting the correct tarball this patch still fails.

If I ignore the patch failing and try to build I get build errors like these

Undefined symbols for architecture x86_64:
  "_lame_init_old", referenced from:
     -exported_symbol[s_list] command line option
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [libmp3lame.la] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of lame: ########## Error running make -j 3  *** [105/135]

comment:21 Changed 5 months ago by Smo

gst-plugins-base has issue building the examples

libtool: link: unable to infer tagged configuration
libtool:   error: specify a tag with '--tag'

Don't build the examples with --disable-examples configure flag and everything builds fine.

comment:22 Changed 5 months ago by Smo

scons doesn't build under python3

*** Building scons *** [131/135]
python setup.py build
scons: *** SCons does not run under Python version 3.6.4.
Python 3 and above are not yet supported.
*** Error during phase build of scons: ########## Error running python setup.py build *** [131/135]

need this to build serf and subversion

comment:23 Changed 5 months ago by Smo

Bunch of warnings I noticed as well when you start the build

W: meta-osx-xpra-python3-deps has a dependency on unknown "gst-python" module
W: meta-osx-xpra-python3-deps has a dependency on unknown "python3-pyasn1" module
W: meta-osx-xpra-python3-deps has a dependency on unknown "python3-pyasn1" module
W: python3-pyasn1-modules has a dependency on unknown "python3-pyasn1" module
W: python3-pyasn1-modules has a dependency on unknown "python3-pyasn1" module
W: python3-ldap has a dependency on unknown "python3-pyasn1" module
W: python3-ldap has a dependency on unknown "python3-pyasn1" module

Last edited 5 months ago by Smo (previous) (diff)

comment:24 Changed 5 months ago by Antoine Martin

Started from scratch again today ran into some issues with libxml2. It would always just build the python2 variant and then would also fail with

Weird, I didn't hit this one.

itstool from git seems to work just fine had to install this manually

Again, I didn't hit this one. (I'm building on 10.10)
We'll just wait for the next release and update itstool in the moduleset then.

seems to be some problems with building lame still as well

Yep. It's an upstream bug:

gst-plugins-base has issue building the examples

Fixed in r19356.

I hit a more fundamental issue rebuilding gstreamer:

  GEN      Gst-1.0.gir
(..)
ImportError: dlopen(/Users/gtk3/gtk/inst/lib/gobject-introspection/giscanner/_giscanner.so, 2): Symbol not found: _PyInt_FromLong
  Referenced from: /Users/gtk3/gtk/inst/lib/gobject-introspection/giscanner/_giscanner.so
  Expected in: flat namespace
 in /Users/gtk3/gtk/inst/lib/gobject-introspection/giscanner/_giscanner.so

No idea when the broke, or why.
Had to rebuild gobject-introspection which caused dozens of other modules to get rebuilt.

Then something else was not quite right - looks like some headers aren't installed by gst-plugings-base, gst-libav fails with:

/Users/gtk3/gtk/inst/include/gstreamer-1.0/gst/audio/audio.h:36:10: fatal error: 'gst/audio/gstaudioaggregator.h' file not found
#include <gst/audio/gstaudioaggregator.h>
         ^
1 error generated.

Fixed by:

cp ${JHBUILD_SOURCE}/gst-plugins-base-1.14.1/gst-libs/gst/audio/gstaudioaggregator.h ${JHBUILD_PREFIX}/include/gstreamer-1.0/gst/audio/

Then after that:

In file included from /Users/gtk3/gtk/inst/include/gstreamer-1.0/gst/audio/audio.h:36:
/Users/gtk3/gtk/inst/include/gstreamer-1.0/gst/audio/gstaudioaggregator.h:27:10: fatal error: 'gst/base/gstaggregator.h' file not found
#include <gst/base/gstaggregator.h>
         ^
1 error generated.

Fixed by:

cp ${JHBUILD_SOURCE}/gstreamer-1.14.1/libs/gst/base/gstaggregator.h ${JHBUILD_PREFIX}/include/gstreamer-1.0/gst/base/

Bunch of warnings I noticed as well when you start the build
pyasn1 ...

r19355 fixes that.

Last edited 5 months ago by Antoine Martin (previous) (diff)

comment:25 Changed 5 months ago by Smo

They seem to have a fix for the lame issue over at homebrew site

https://github.com/Homebrew/homebrew-core/blob/master/Formula/lame.rb

    # Fix undefined symbol error _lame_init_old
    # https://sourceforge.net/p/lame/mailman/message/36081038/
    inreplace "include/libmp3lame.sym", "lame_init_old\n", ""

Just remove the offending line from include/libmp3lame.sym and all is well. I'll add a patch.

comment:26 Changed 5 months ago by Smo

r19418 adds the patch to the repo
r19419 adds it to the moduleset

lame now builds without error

comment:27 Changed 5 months ago by Antoine Martin

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.