xpra icon
Bug tracker and wiki

Opened 7 months ago

Last modified 5 days ago

#1575 new task

python3 packaging for macos

Reported by: Antoine Martin Owned by: Smo
Priority: blocker Milestone: 2.3
Component: packaging Version: trunk
Keywords: Cc:

Description

python3 tracker ticket: #1568.

TODO:

  • update moduleset to latest python (ie: 3.6.x, currently building 3.4!)
  • add all dependencies to moduleset
  • add a new meta module for python3 xpra
  • py2app packaging

etc..

Change History (17)

comment:1 Changed 6 months ago by Antoine Martin

Status: newassigned

This may fix issues like #761 and #404, worth a try.

Does the GTK3 macos port support opengl though? (#1569)

Last edited 3 months ago by Antoine Martin (previous) (diff)

comment:2 Changed 3 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo
Status: assignednew

@smo: any updates?

comment:3 Changed 2 months ago by Antoine Martin

Priority: majorcritical

Needed for window transparency: #1570

comment:4 Changed 8 weeks ago by Antoine Martin

Milestone: 3.02.3

comment:5 Changed 8 weeks ago by Antoine Martin

See also #1708: 2.3 moduleset updates

comment:6 Changed 3 weeks ago by Antoine Martin

Priority: criticalblocker

@smo: please put anything you have already done in this ticket, then I can take a look.

comment:7 Changed 3 weeks ago by Smo

I tried to build the up to date moduleset but had a problem with pygobject. Here is my contact to the mailing list which didn't really go anywhere.

https://mail.gnome.org/archives/gtk-osx-devel-list/2017-September/msg00000.html

comment:8 Changed 3 weeks ago by Antoine Martin

Owner: changed from Smo to Antoine Martin
Status: newassigned

Using a clean user account and following the build instructions from https://wiki.gnome.org/Projects/GTK+/OSX/Building:

wget --no-check-certificate \
    "https://git.gnome.org/browse/gtk-osx/plain/gtk-osx-build-setup.sh"
sh ./gtk-osx-build-setup.sh
PATH=/Users/gtk3/.local/bin:$PATH
jhbuild bootstrap

Got a few certificate errors with: xz, pkg-config, flex, expat, etc (download without checking certs and continue).
Then it fails to build python3 with another cert error:

jhbuild buildone python3
jhbuild buildone: could not download http://git.gnome.org/browse/gtk-osx/plain/modulesets-stable/gtk-osx.modules: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

This time fixed by downloading the cacert.pem from here:

https://curl.haxx.se/docs/caextract.html and then telling jhbuild to use it:

CA_CERT_FILE=$HOME/cacert.pem jhbuild build python3

Make python3 the default (as per docs - though the upstream instructions look wrong for the second symlink):

jhbuild shell
cd $JHBUILD_PREFIX/bin
ln -s python3 python
ln -s python3-config python-config
jhbuild build openssl

(upstream notes apply: interrupt the build and run it by hand - we should upstream our openssl 1.1 build commands)

jhbuid build meta-gtk-osx-bootstrap meta-gtk-osx-gtk3

Got a minor failure building glib:

xzcat -d "/Users/gtk3/gtk/source/pkgs/glib-2.52.2.tar.xz" | tar xf -
*** Applying patch https://git.gnome.org/browse/gtk-osx/plain/patches/glib-gint64-long-long.patch *** [14/39]
patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-gint64-long-long.patch"
patching file configure.ac
Hunk #1 succeeded at 3002 (offset 48 lines).
Hunk #2 succeeded at 3114 (offset 48 lines).
Hunk #3 succeeded at 3266 (offset 49 lines).
*** Applying patch https://git.gnome.org/browse/gtk-osx/plain/patches/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch *** [14/39]
patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch"
patching file gio/gosxappinfo.c
patching file gio/gosxcontenttype.c
Hunk #1 succeeded at 52 (offset -6 lines).
patching file gio/tests/contenttype.c
Hunk #1 succeeded at 323 with fuzz 1 (offset -38 lines).
Hunk #2 FAILED at 359.
1 out of 2 hunks FAILED -- saving rejects to file gio/tests/contenttype.c.rej
*** Error during phase checkout of glib: ########## Error running patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch" *** [14/39]

Applied the one line from the reject file by hand and continued.

Everything built correctly after that.

@smo: whatever you did (which was not recorded anywhere AFAICT) must have been different?


Continuing from there:

  • r17840 makes the moduleset more reusable
  • add our moduleset to jhbuild:
    cat > $HOME/.jhbuildrc-custom << EOF
    moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
    EOF
    
  • and build some things we will need:
    jhbuild buildone sshpass -f
    jhbuild build meta-subversion-xpra
    jhbuild build meta-osx-xpra-pkgtools
    jhbuild build meta-osx-xpra-codec-deps
    

Got one failure we should fix on lame:

gzip -dc "/Users/gtk3/gtk/source/pkgs/lame-3.99.5.tar.gz" | tar xf -
*** Applying patch http://xpra.org/svn/Xpra/trunk/osx/jhbuild/patches/lame-channels.patch *** [8/23]
patch -p0 < "/Users/gtk3/.cache/jhbuild/lame-channels.patch"
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/libmp3lame/set_get.c
|+++ b/libmp3lame/set_get.c
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
patch: **** malformed patch at line 10:

Next we have to build python and switch back to python3 afterwards so we can build the python deps against python3:

jhbuild buildone python
jhbuild shell
cd $JHBUILD_PREFIX/bin
rm python python-config
ln -s python3 python
ln -s python3-config python-config

Build the rest:

jhbuild build meta-osx-xpra-python-deps

lz4 had a problem (will fix).
Then python3 had a problem finding ssl, it just needed to be rebuilt now that we brought in a newer openssl lib version:

jhbuild buildone python3 -f

At that point, everything is built OK, AFAICT.

Still TODO:

  • merge stable moduleset updates (oh joy..)
  • add python3 / gtk3 macos handling to our setup.py

comment:9 Changed 3 weeks ago by Antoine Martin

@smo: why are we so far behind https://git.gnome.org/browse/gtk-osx/tree/?
Even gtk2 is outdated here. (only our gstreamer and network modulesets are slightly ahead)
Is there anything there that we should not merge and why?

I've merged most of the changes:

  • r17846: mostly things we don't even use - and maybe we shouldn't copy them then, just use an https module reference and be done with it - and we can still override some packages with newer ones, only when we need to keep an older version do we really need to copy the moduleset, right?
  • r17847: bootstrap, network, python
  • r17848: main gtk-osx moduleset
  • r17849: glib for gobject-introspection (will check the python2 re-build later)
  • r17850 + r17851: moduleset updates for packaging tools
  • r17855: gtkosx_application import for GTK3 (gi bindings)
  • r17856: python-cryptography new requirement?
  • r17857: tiny py3k string fix
  • r17858 + r17859: force py2app to not include the desktop server (raising ImportError does it)
  • r17860: a really annoying issue is that although the headers look the same, py3cairo and pycairo install in different places, but our source needs to reference just one header file.. so this a manual step is still needed:
    ln -sf py3cairo.h /Users/gtk3/gtk/inst/include/pycairo/pycairo.h
    

Also added latest gdb by hand:

wget https://ftp.gnu.org/gnu/gdb/gdb-8.0.1.tar.gz
tar -zxvf gdb-8.0.1.tar.gz 
cd gdb-8.0.1
./configure --prefix=${JHBUILD_PREFIX}
make && make install
sudo chgrp procmod ${JHBUILD_PREFIX}/bin/gdb
sudo chmod g+s ${JHBUILD_PREFIX}/bin/gdb

But... still needs signing to be used: codesign gdb.

comment:10 Changed 3 weeks ago by Antoine Martin

More work on this:

  • r17861: gtk-mac-bundler requires python2 to run
  • r17864: yet another new python-lz4 dependency (sigh), r17868 + r17872: macos packaging for it
  • r17870: support new versions of python-lz4 (applies to all arches)
  • r17866: gtk-quartz-engine isn't in the moduleset any more? it is gtk2 and contains build errors... remove it
  • r17865: fixup r17855 gtkosx application import
  • r17867 + r17881: packaging tweaks: ship more gi bindings, skip (py)gtk2 stuff, etc
  • r17876: clipboard support for gtk3
  • r17877 + r17878: tray, clipboard and gui porting

Still TODO:

  • gdk bindings need porting (input devices stuff, see ticket:1157#comment:24) unless GTK3 gives us the devices and the correct values out of the box?
  • opengl: need to implement the gl backend glue, unfortunately there is no easy way to get the "nsview" from a gdk window in GTK3... so we may have to use a Cython class
  • fix the GtkMenuBar invalid cast warnings, somehow

Apart from that, it looks usable. Beta builds posted here with the "Python3" filename: https://xpra.org/beta/osx/.

comment:11 Changed 2 weeks ago by Antoine Martin

Notes after rebuilding again from scratch to verify:

  • pygobject3 needed this hack to avoid errors on "stack_size":
    find . -name "Makefile" -exec sed -i '' 's+-Wl,-stack_size,1000000 ++g' {} \;
    
  • r17883 + r17884: re-instate openssl 1.1
  • r17885 + r17887: better openssl config command, fix "lame" patch context stripping
  • r17886: dependency tweaks
  • r17888: don't duplicate modulesets we don't need or modify, just reference them
  • r17889: follow upstream and remove "1.0" suffix from all gstreamer modules names
  • r17891: avoid runtime warning with GTK3
  • r17892: remove python2 ssl file path (may need to be re-added via the shell script)
  • r17893: fix syntax error in moduleset
  • r17894: move all python modules to a new moduleset file we include
  • r17895 + r17896 + r17897 + r17898: python3 version of the moduleset
  • r17899: cosmetic
  • we should fix xar: Fails to build against OpenSSL version 1.1.0g
  • libxml2 can be built against python3 with this simple hackish patch (found here: Bug 1406920 - libxml2 should no longer use PyVerify_fd):
    +--- a/libxml2-2.9.4/python/types.c	2016-02-09 03:17:33.000000000 -0700
    ++++ b/libxml2-2.9.4/python/types.c	2016-12-21 12:34:06.755650986 -0700
    +@@ -31,8 +31,6 @@
    +     const char *mode;
    + 
    +     fd = PyObject_AsFileDescriptor(f);
    +-    if (!_PyVerify_fd(fd))
    +-        return(NULL);
    +     /*
    +      * Get the flags on the fd to understand how it was opened
    +      */
    

Saves having to temporarily switch back to python2 to build it.

The python2-only builds started hitting the issue from comment:7, and now the mixed python2-python3 builds do too. Sigh.

One more weird issue started happening: the "py2app" step does something that loads gdk and tries to access the display server and crashes when it fails, means that the builds now have to be executed from a GUI shell. PITA.


So at this point, the python3 jhbuild env is almost straightforward (bar python3 vs python issues) with this file:

cat > .jhbuildrc-custom  <<EOF
_gtk_osx_use_jhbuild_python = True
setup_sdk(target="10.10", sdk_version="10.10", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "openssl", "python3", "yasm", "nasm", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core", "meta-gtk-osx-gtk3",
"libpng",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-mac-integration-python", "meta-osx-python3-xpra-deps"]
moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"
EOF
Last edited 2 weeks ago by Antoine Martin (previous) (diff)

comment:12 Changed 2 weeks ago by Antoine Martin

Owner: changed from Antoine Martin to Smo
Status: assignednew

Lots more updates (in large part because testing jhbuild is hard, so I commit then fix the numerous mistakes one at a time).
I've also sent some questions and pull requests upstream to try to reduce our diff:

Changesets:

With all these changes:

  • the modulesets are cleaner, and closer to upstream
  • the differences between the python2 and python3 / gtk3 modulesets are easier to see and maintain
  • building everything from scratch is easier:
    cat > .jhbuildrc-custom <<EOF
    _gtk_osx_use_jhbuild_python = True
    setup_sdk(target="10.10", sdk_version="10.10", architectures=["x86_64"])
    os.environ["CC"] = "/usr/bin/gcc"
    os.environ["DYLD_LIBRARY_PATH"] = ""
    build_policy = "updated-deps"
    modules = ["meta-osx-xpra-deps"]
    moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtk3.modules"
    os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"
    EOF
    

Then just: jhbuild build.
The only difference for the "GTK2" moduleset is the moduleset URL:

moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtk2.modules"

Then when building the packages, we have to do:

PYTHON=python3 ./make-all.sh

(or PYTHON=python2 for GTK2 builds)

@smo: does that work for you?

comment:13 Changed 2 weeks ago by Antoine Martin

Note: the gi bingings were missing from the python2 builds (meaning no sound), to re-instate them I had to apply this patch: pygobject: remove references to deprecated GI_INFO_TYPE_ERROR_DOMAIN. (basically, just remove all references to GI_INFO_TYPE_ERROR_DOMAIN then rebuild with introspection enabled)
There might be a better way of doing this.

comment:14 Changed 11 days ago by Smo

gtk-engines fails when building for python3/gtk3 is this needed?

checking for GTK... no
configure: error: GTK+-2.12 is required to compile gtk-engines
*** Error during phase configure of gtk-engines: ########## Error running ./configure --prefix /Users/xpragtk3/gtk/inst    *** [47/129]

comment:15 Changed 11 days ago by Antoine Martin

gtk-engines fails when building for python3/gtk3 is this needed?

It isn't referenced anywhere in our modulesets.
Can you get some info on it? jhbuild info gtk-engines

comment:16 Changed 5 days ago by Smo

Name: gtk-engines
Module Set: gtk-osx-themes
Type: autogen
Install version: not installed
Install date: not installed
URL: http://ftp.gnome.org/pub/GNOME/sources/gtk-engines/2.20/gtk-engines-2.20.2.tar.bz2
Version: 2.20.2
Tree-ID: 2.20.2-d41d8cd98f00b204e9800998ecf8427e
Sourcedir: /Users/xpragtk3/gtk/source/gtk-engines-2.20.2
Requires: automake, libtool, make
Required by: meta-gtk-osx-themes
After: meta-gtk-osx-core

comment:17 Changed 5 days ago by Antoine Martin

meta-gtk-osx-themes is found here: https://github.com/GNOME/gtk-osx/blob/master/modulesets-stable/gtk-osx-themes.modules, but isn't referenced anywhere else in the modulesets.
There is a meta-gtk-osx-gtk3-core-themes, so I've switched to that in r18020.
@smo: does that fix things? Do we need to add icon-naming-utils?

Note: See TracTickets for help on using tickets.