xpra icon
Bug tracker and wiki

Opened 3 years ago

Closed 9 months ago

Last modified 9 months ago

#840 closed task (fixed)

64-bit builds for OSX using moduleset

Reported by: Antoine Martin Owned by: Smo
Priority: blocker Milestone: 2.0
Component: platforms Version: trunk
Keywords: osx Cc:

Description (last modified by Antoine Martin)

Split from #533. See ticket:533#comment:77

Minor fixes + tweaks already done: r9089, r9090.

Things that need fixing:

  • the moduleset needs updating, adding cups, etc as per the comments in #533.
  • fix the pyobjc build failure
  • detect when trying to run on a 32-bit system and provide a helpful error message
  • add new download links for it
  • nice-to-have: libvpx 1.4 (see failure in ticket/832#comment:1, still fails on 64-bit..)

That's it!

Change History (37)

comment:1 Changed 3 years ago by Antoine Martin

Description: modified (diff)

FWIW: I've uploaded a beta 0.15.0 build which works quite well, see http://xpra.org/beta/osx/x86_64.

@afarr: it would be worth playing with it a bit to see if there are any other issues - stability issues (or even improvements), or if there are any differences in performance, etc..

comment:2 Changed 3 years ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

See also:

@smo: you may want to post the errors to the pyobjc mailing list, that's the blocker. The rest is relatively easy or optional.

Last edited 10 months ago by Antoine Martin (previous) (diff)

comment:3 Changed 3 years ago by Antoine Martin

Lots of builds updates, see ticket:533#comment:79 and later comments.
In particular: r9123 which removes lots of i386

comment:4 Changed 2 years ago by Antoine Martin

See also #984

comment:5 Changed 2 years ago by Smo

Updated in r10674 from upstream. We may still need to make further updates to this for our needs.

comment:6 Changed 2 years ago by Smo

This is still on hold until we can get gstreamer up to date.

comment:7 Changed 2 years ago by Smo

Owner: changed from Smo to Antoine Martin

Can we reschedule this for a further release. Unless you think 64bit builds are not worth the bother and we can just close this.

comment:8 Changed 2 years ago by Antoine Martin

Milestone: 0.161.0
Owner: changed from Antoine Martin to Smo

Re-scheduling. Worth doing, maybe after the next LTS so we have both 32-bit and 64-bit.

comment:9 Changed 2 years ago by Smo

Tried this myself today on osx 10.9.5 didn't get as far as I hoped some weird issue here.

(cd /Users/user/gtk/inst/_jhbuild/root-tango-icon-theme/Users/user/gtk/inst/share/icons/Tango/16x16 && /Users/user/gtk/inst/libexec/icon-name-mapping -c actions)
Bizarre copy of UNKNOWN in scalar assignment at /Users/user/gtk/inst/lib/perl5/5.16.2/darwin-thread-multi-2level/XML/Parser/Expat.pm line 125.
make[3]: *** [install-data-local] Error 25
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install-recursive] Error 1
*** Error during phase install of tango-icon-theme: ########## Error running make install DESTDIR=/Users/user/gtk/inst/_jhbuild/root-tango-icon-theme *** [41/97]

If I run this utility I also get the same issue.

/Users/user/gtk/inst/libexec/icon-name-mapping -c actions
Bizarre copy of UNKNOWN in scalar assignment at /Users/user/gtk/inst/lib/perl5/5.16.2/darwin-thread-multi-2level/XML/Parser/Expat.pm line 125.

Doesn't appear to be a 64bit issue really just some sort of broken perl module?

comment:10 Changed 23 months ago by Smo

Building on 10.11.1 with xcode 7.1 and these were the ones that failed

*** the following modules were not built *** [97/97]
libvpx sdl gst-plugins-bad-xpra pygtkglext meta-osx-xpra-deps

sdl was missing headers for X
libvpx needs updating to 1.5.0 which builds fine need to update moduleset

gst-plugins-bad-xpra failed to build with an error about ssl header files we might be able to disable this

In file included from gstapexraop.c:33:
./gstapexraop.h:33:10: fatal error: 'openssl/rand.h' file not found
#include <openssl/rand.h>
         ^
In file included from gstapexplugin.c:29:
In file included from ./gstapexsink.h:32:
./gstapexraop.h:33:10: fatal error: 'openssl/rand.h' file not found
#include <openssl/rand.h>

pygtkglext failed with this error

gdkglext.override:36:10: fatal error: 'GL/gl.h' file not found
#include <GL/gl.h>

comment:11 Changed 23 months ago by Antoine Martin

comment:12 Changed 23 months ago by Smo

gtk/gdkgl/gdkglext.c change to #include <OpenGL/gl.h> afterwards this builds no problem.

I'm not sure about having multiple modulesets at the moment I include it by url in customrc file but one of the files that gets included also has a url we include from svn trunk

comment:13 Changed 20 months ago by Antoine Martin

This should allow us to get libyuv on osx: ticket:973#comment:3

comment:14 Changed 19 months ago by Antoine Martin

Milestone: 1.00.18

Should be a priority for the next milestone.

comment:15 Changed 17 months ago by Smo

Still having issues trying to do this now I think its an issue with jhbuild during the configure phase of x264 it produces this stacktrace

*** Configuring x264 *** [72/116]
Traceback (most recent call last):
  File "/Users/user/.local/bin/jhbuild", line 32, in <module>
    jhbuild.main.main(sys.argv[1:])
  File "/Users/user/Source/jhbuild/jhbuild/main.py", line 155, in main
    rc = jhbuild.commands.run(command, config, args, help=lambda: print_help(parser))
  File "/Users/user/Source/jhbuild/jhbuild/commands/__init__.py", line 183, in run
    return cmd.execute(config, args, help)
  File "/Users/user/Source/jhbuild/jhbuild/commands/__init__.py", line 52, in execute
    return self.run(config, options, args, help)
  File "/Users/user/Source/jhbuild/jhbuild/commands/base.py", line 268, in run
    return build.build()
  File "/Users/user/Source/jhbuild/jhbuild/frontends/buildscript.py", line 163, in build
    error, altphases = module.run_phase(self, phase)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/__init__.py", line 420, in run_phase
    method(buildscript)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/autotools.py", line 217, in do_configure
    cmd = self._get_configure_cmd(buildscript)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/autotools.py", line 130, in _get_configure_cmd
    cmd = self.static_analyzer_pre_cmd(buildscript) + template % vars
KeyError: u'libdir'

comment:16 Changed 16 months ago by Antoine Martin

Any progress? We can skip x264 and do it by hand if need be.

Blocker for #1231.

Last edited 16 months ago by Antoine Martin (previous) (diff)

comment:17 Changed 16 months ago by Smo

The problem is with libdir replacement in those strings maybe this isn't needed anymore since if I remove it I can compile this package through jhbuild.

I will commit fix for this shortly to trunk. Looks like a similar issue with ffmpeg as well.

comment:18 Changed 16 months ago by Smo

my current .jhbuildrc-custom for el capitan machine

setup_sdk(target="10.9", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "python", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine","gtk-mac-integration-python",
"gstreamer", "gst-plugins-base", "gst-plugins-good", "meta-gstreamer-1.0",
"meta-osx-xpra-deps"]

#skip iconv and add some switches to turn off stuff that was breaking
skip.append("libiconv")
#skip.append("expat")
append_autogenargs('gstreamer', '--enable-introspection=no')
append_autogenargs('gst-plugins-base', '--enable-introspection=no')
append_autogenargs('gst-plugins-good', '--enable-introspection=no --disable-deinterlace')
append_autogenargs('gst-plugins-bad-1.0', '--enable-opengl=no --disable-apple_media')

#change moduleset
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"

comment:19 Changed 16 months ago by Antoine Martin

This does not look ready to me, or I need some more precise instructions.

Following, more or less: https://winswitch.org/dev/macosx.html but with the new customrc on a 10.10 system. Found some useful commands along the way:

$ sw_vers -productVersion
10.10.5
$ xcrun --show-sdk-version
10.11
$ xcrun --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk

And now for the (long) list of problems:

  • during boostrap:
    ERROR: no certificate subject alternative name matches
    	requested host name `cmake.org'.
    
  • more problematic during the build step:
    *** Configuring itstool *** [5/116]
    (..)
    checking for python module libxml2... ./configure: line 2422: 16059 Done                    echo "import $py_module"
         16060 Abort trap: 6           | python - >&/dev/null
    not found
    configure: error: Python module libxml2 is needed to run this package
    

Probably the same cause as this other error:

Making all in doc
/Users/osx/gtk/inst/etc/xml/catalog:1: parser error : Document is empty
I/O error : Attempt to load network entity http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd
(..)

And another related one (as I found out much later..):

GISCAN   GLib-2.0.gir
Traceback (most recent call last):
  File "./g-ir-scanner", line 53, in <module>
    from giscanner.scannermain import scanner_main
  File "./giscanner/scannermain.py", line 29, in <module>
    import tempfile
  File "/Users/osx/gtk/inst/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/Users/osx/gtk/inst/lib/python2.7/io.py", line 51, in <module>
    import _io
ImportError: No module named _io
make[2]: *** [GLib-2.0.gir] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of gobject-introspection: ########## Error running make -j 2  *** [27/116]

Same for gst-python and most python modules, pango (gtk-doc), gtk-mac-integration, librsvg, gtk-engines, gtk-quartz-engine ..
It tries to build pycairo-python2.6 - that looks wrong.
Looks like we want to force python to build earlier. I found that out later, after much time wasted. Most python modules then build OK.
Maybe just a _gtk_osx_use_jhbuild_python = True line?
Or maybe I was meant to build meta-osx-xpra-deps and not just a plain build?

Last edited 16 months ago by Antoine Martin (previous) (diff)

comment:20 Changed 16 months ago by Antoine Martin

At some point we will have to switch the existing build environment to a locked version, and move forward with the trunk / 64-bit / newer build env.
Until then, r12959 fixes building ffmpeg but this may also get in the way of supporting webcam capture in the future (we can't have both..).

comment:21 Changed 16 months ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

comment:22 Changed 14 months ago by Antoine Martin

FYI: bumped to openssl 1.1.0 in r13481, which requires a newer perl version than the one available on my 10.5.x VM, so then I had to figure out how to build the latest stable perl there:

tar -zxvf perl-5.24.0.tar.gz 
cd perl-5.24.0
./Configure -des -Dprefix={JHBUILD_PREFIX} \
    -Accflags="-arch i686 -I{JHBUILD_PREFIX}/include/ -L{JHBUILD_PREFIX}/lib/" \
    -Aldflags="-arch i686 -I{JHBUILD_PREFIX}/include/ -L{JHBUILD_PREFIX}/lib/"
make && make install

Bonus: this version seems to install cleanly without needing a chmod.
Downside: python-cryptography needs updating to 1.5 to support openssl 1.1 : https://cryptography.io/en/latest/changelog/. (done in r13482)

@smo: if that works for you, please add perl to the moduleset.

comment:23 Changed 14 months ago by Smo

Okay I ran into the perl issue as well.

I will add this to the moduleset but should we require it or leave it as an optional thing on older systems that need it?

comment:24 Changed 14 months ago by Antoine Martin

Hopefully we will be moving to a newer 64-bit build system soon, so this should be optional.

comment:25 Changed 14 months ago by Smo

r13692:13695 adds perl

Was some issues with the configure line since it is non standard. Not sure if I need to add the cc and ld flags so I left them out for now.

If the new openssl seems to install properly i'll see if I can remove some of the hacks from the moduleset for installing it in the first place.

comment:26 Changed 13 months ago by Smo

Tried with the latest changes to the moduleset on 64 bit. Seems to get further now. I got this error today

build/temp.macosx-10.9-x86_64-2.7/_openssl.c:415:10: fatal error: 'openssl/opensslv.h' file not found
#include <openssl/opensslv.h>
         ^
1 error generated.
error: command '/usr/bin/gcc' failed with exit status 1
*** Error during phase build of python-cryptography: ########## Error running python setup.py build *** [106/119]

Confirmed that openssl is installed.

comment:27 Changed 13 months ago by Antoine Martin

Milestone: 1.02.0
Priority: majorblocker

Using 10.10+ as a build platform will also allow us to fix #404 and #1305

comment:28 Changed 13 months ago by Antoine Martin

Tried again as part of #1340:

  • got an error with librsvg not finding gdk-pixbuf gir
  • ffmpeg complained about missing libvpx
  • occasionally had to switch things around with code-select: jhbuild requires the developer path but compiling things require the command line tools.. PITA - how do get get both to work at the same time?

comment:29 Changed 12 months ago by Antoine Martin

Also a blocker for #1340.

comment:31 Changed 10 months ago by Antoine Martin

After moving to a newer build host, we can start using a number of new APIs and do:

  • #776
  • webcam support

etc

comment:32 Changed 10 months ago by Antoine Martin

I have pointed my 10.5.x build VM to the stable 1.0.x branch after merging the latest set of safe updates (r14622).
Now we can start "breaking" things again to make 64-bit builds work properly, I am building on 10.10.x with xcode 7.2.1:

After that:

  • jhbuild buildone -f openssl to make sure it is rebuilt before python (this should probably be in the moduleset instead)
  • jhbuild build -f - and wait a while (see minor fixes below)
  • make a signed build in one swoop (based on the instructions in ticket:1340#comment:19):
    security unlock-keychain -p THEACCOUNTPASSWORD
    export CODESIGN_KEYNAME="Antoine Martin"
    ./make-all.sh
    
  • the resulting DMG / PKG are not much smaller than before: http://xpra.org/beta/osx

Things left TODO:

  • openssl is still a pain to build + install, still requires
    chmod +x ${JHBUILD_PREFIX}/lib/libssl.* ${JHBUILD_PREFIX}/lib/libcrypto.*
    
  • the gi bindings are missing and we really need those for gstreamer.. but [ https://github.com/anholt/libepoxy libepoxy] won't build (could be related to Adds support for OS X without X11), and it is required by GTK3, which is needed by the gobject-introspection module... I still managed to build it with jhbuild buildone -f pygobject3
  • gstreamer 1.10.x (version bumped in r14627, fix for gst-plugins-bad in r14631): downloads fail because they redirect to https and we get a certificate issue, again (as found in comment:19)
  • speex codec is missing, needs adding to moduleset?
  • pyobjc-core moaned:
    error: SDK root '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk' does not exist
    

Not sure why it has the same path 3 times as the command it runs to get that path only returns it once, but running like this allows me to move on:

export SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
python ./setup.py build
python ./setup.py install
  • IIRC, some versions of glib / gi were held back for compatibility reasons (with what?) - can we move forward again?

Note: there is a weird setup issue on my 10.10.x system which forces me to switch to command line tools to ensure the build finds gcc / clang but switch back to "developer" mode to launch jhbuild:

  • sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer
  • sudo xcode-select -switch /Library/Developer/CommandLineTools
Last edited 10 months ago by Antoine Martin (previous) (diff)

comment:33 Changed 10 months ago by Antoine Martin

Updates:

  • r14648 removes libdir from gtkglext
  • r14647 fixes mpfr download link, r14649 uses a reliable link (one that will not expire when the next release comes out), r14650 (undoes an unwanted change)
  • r14646 opus update to 1.1.3
  • r14645 fixes openssl install (no longer installed every time or needing chmod)
  • r14651 + r14652 + r14653: use "1.0" suffix for gstreamer modules, enable gobject-introspection
Last edited 10 months ago by Antoine Martin (previous) (diff)

comment:34 Changed 10 months ago by Antoine Martin

Some of these updates have been applied to the v1.0.x branch: r14657 (libpng), r14658 (mpfr), r14659 (opus)


TODO:

  • pygobject3 dependency? (build gtk3, surely that can be fixed?)
  • gi / glib updates?

Will ignore: pyobjc and other build env weirdness for now.

Last edited 10 months ago by Antoine Martin (previous) (diff)

comment:35 Changed 9 months ago by Smo

Everything builds for me on this machine

sw_vers && xcodebuild -version
ProductName:	Mac OS X
ProductVersion:	10.12.1
BuildVersion:	16B2555
Xcode 8.2.1
Build version 8C1002
Last edited 9 months ago by Smo (previous) (diff)

comment:36 Changed 9 months ago by Smo

Also works on this system

sw_vers && xcodebuild -version
ProductName:    Mac OS X
ProductVersion: 10.11.1
BuildVersion:   15B42
Xcode 7.1
Build version 7B91b

I put nasm in the modules part of the config because libjpeg-turbo will need it.
using this config
~/.jhbuildrc-custom

_gtk_osx_use_jhbuild_python = True
setup_sdk(target="10.9", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"

modules = [ "openssl", "python", "yasm", "nasm", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine","gtk-mac-integration-python", "meta-osx-xpra-deps"]

#change moduleset  
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"

comment:37 Changed 9 months ago by Antoine Martin

Resolution: fixed
Status: newclosed

Works for me too (doing another full rebuild from scratch to check), minus the weirdness of having to run xcodeselect on this build system. (PITA but not a showstopper)

We can follow up in #1404.
The next release will be 64-bit only for OSX! (and targetting 10.9+ only)

Last edited 9 months ago by Antoine Martin (previous) (diff)
Note: See TracTickets for help on using tickets.