xpra icon
Bug tracker and wiki

Opened 2 years ago

Closed 3 months ago

Last modified 3 months ago

#840 closed task (fixed)

64-bit builds for OSX using moduleset

Reported by: Antoine Martin Owned by: Smo
Priority: blocker Milestone: 2.0
Component: platforms Version: trunk
Keywords: osx Cc:

Description (last modified by Antoine Martin)

Split from #533. See ticket:533#comment:77

Minor fixes + tweaks already done: r9089, r9090.

Things that need fixing:

  • the moduleset needs updating, adding cups, etc as per the comments in #533.
  • fix the pyobjc build failure
  • detect when trying to run on a 32-bit system and provide a helpful error message
  • add new download links for it
  • nice-to-have: libvpx 1.4 (see failure in ticket/832#comment:1, still fails on 64-bit..)

That's it!

Change History (37)

comment:1 Changed 2 years ago by Antoine Martin

Description: modified (diff)

FWIW: I've uploaded a beta 0.15.0 build which works quite well, see http://xpra.org/beta/osx/x86_64.

@afarr: it would be worth playing with it a bit to see if there are any other issues - stability issues (or even improvements), or if there are any differences in performance, etc..

comment:2 Changed 2 years ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

See also:

@smo: you may want to post the errors to the pyobjc mailing list, that's the blocker. The rest is relatively easy or optional.

Last edited 4 months ago by Antoine Martin (previous) (diff)

comment:3 Changed 2 years ago by Antoine Martin

Lots of builds updates, see ticket:533#comment:79 and later comments.
In particular: r9123 which removes lots of i386

comment:4 Changed 20 months ago by Antoine Martin

See also #984

comment:5 Changed 19 months ago by Smo

Updated in r10674 from upstream. We may still need to make further updates to this for our needs.

comment:6 Changed 18 months ago by Smo

This is still on hold until we can get gstreamer up to date.

comment:7 Changed 18 months ago by Smo

Owner: changed from Smo to Antoine Martin

Can we reschedule this for a further release. Unless you think 64bit builds are not worth the bother and we can just close this.

comment:8 Changed 18 months ago by Antoine Martin

Milestone: 0.161.0
Owner: changed from Antoine Martin to Smo

Re-scheduling. Worth doing, maybe after the next LTS so we have both 32-bit and 64-bit.

comment:9 Changed 18 months ago by Smo

Tried this myself today on osx 10.9.5 didn't get as far as I hoped some weird issue here.

(cd /Users/user/gtk/inst/_jhbuild/root-tango-icon-theme/Users/user/gtk/inst/share/icons/Tango/16x16 && /Users/user/gtk/inst/libexec/icon-name-mapping -c actions)
Bizarre copy of UNKNOWN in scalar assignment at /Users/user/gtk/inst/lib/perl5/5.16.2/darwin-thread-multi-2level/XML/Parser/Expat.pm line 125.
make[3]: *** [install-data-local] Error 25
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install-recursive] Error 1
*** Error during phase install of tango-icon-theme: ########## Error running make install DESTDIR=/Users/user/gtk/inst/_jhbuild/root-tango-icon-theme *** [41/97]

If I run this utility I also get the same issue.

/Users/user/gtk/inst/libexec/icon-name-mapping -c actions
Bizarre copy of UNKNOWN in scalar assignment at /Users/user/gtk/inst/lib/perl5/5.16.2/darwin-thread-multi-2level/XML/Parser/Expat.pm line 125.

Doesn't appear to be a 64bit issue really just some sort of broken perl module?

comment:10 Changed 17 months ago by Smo

Building on 10.11.1 with xcode 7.1 and these were the ones that failed

*** the following modules were not built *** [97/97]
libvpx sdl gst-plugins-bad-xpra pygtkglext meta-osx-xpra-deps

sdl was missing headers for X
libvpx needs updating to 1.5.0 which builds fine need to update moduleset

gst-plugins-bad-xpra failed to build with an error about ssl header files we might be able to disable this

In file included from gstapexraop.c:33:
./gstapexraop.h:33:10: fatal error: 'openssl/rand.h' file not found
#include <openssl/rand.h>
         ^
In file included from gstapexplugin.c:29:
In file included from ./gstapexsink.h:32:
./gstapexraop.h:33:10: fatal error: 'openssl/rand.h' file not found
#include <openssl/rand.h>

pygtkglext failed with this error

gdkglext.override:36:10: fatal error: 'GL/gl.h' file not found
#include <GL/gl.h>

comment:11 Changed 17 months ago by Antoine Martin

comment:12 Changed 17 months ago by Smo

gtk/gdkgl/gdkglext.c change to #include <OpenGL/gl.h> afterwards this builds no problem.

I'm not sure about having multiple modulesets at the moment I include it by url in customrc file but one of the files that gets included also has a url we include from svn trunk

comment:13 Changed 14 months ago by Antoine Martin

This should allow us to get libyuv on osx: ticket:973#comment:3

comment:14 Changed 13 months ago by Antoine Martin

Milestone: 1.00.18

Should be a priority for the next milestone.

comment:15 Changed 11 months ago by Smo

Still having issues trying to do this now I think its an issue with jhbuild during the configure phase of x264 it produces this stacktrace

*** Configuring x264 *** [72/116]
Traceback (most recent call last):
  File "/Users/user/.local/bin/jhbuild", line 32, in <module>
    jhbuild.main.main(sys.argv[1:])
  File "/Users/user/Source/jhbuild/jhbuild/main.py", line 155, in main
    rc = jhbuild.commands.run(command, config, args, help=lambda: print_help(parser))
  File "/Users/user/Source/jhbuild/jhbuild/commands/__init__.py", line 183, in run
    return cmd.execute(config, args, help)
  File "/Users/user/Source/jhbuild/jhbuild/commands/__init__.py", line 52, in execute
    return self.run(config, options, args, help)
  File "/Users/user/Source/jhbuild/jhbuild/commands/base.py", line 268, in run
    return build.build()
  File "/Users/user/Source/jhbuild/jhbuild/frontends/buildscript.py", line 163, in build
    error, altphases = module.run_phase(self, phase)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/__init__.py", line 420, in run_phase
    method(buildscript)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/autotools.py", line 217, in do_configure
    cmd = self._get_configure_cmd(buildscript)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/autotools.py", line 130, in _get_configure_cmd
    cmd = self.static_analyzer_pre_cmd(buildscript) + template % vars
KeyError: u'libdir'

comment:16 Changed 10 months ago by Antoine Martin

Any progress? We can skip x264 and do it by hand if need be.

Blocker for #1231.

Last edited 10 months ago by Antoine Martin (previous) (diff)

comment:17 Changed 10 months ago by Smo

The problem is with libdir replacement in those strings maybe this isn't needed anymore since if I remove it I can compile this package through jhbuild.

I will commit fix for this shortly to trunk. Looks like a similar issue with ffmpeg as well.

comment:18 Changed 10 months ago by Smo

my current .jhbuildrc-custom for el capitan machine

setup_sdk(target="10.9", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "python", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine","gtk-mac-integration-python",
"gstreamer", "gst-plugins-base", "gst-plugins-good", "meta-gstreamer-1.0",
"meta-osx-xpra-deps"]

#skip iconv and add some switches to turn off stuff that was breaking
skip.append("libiconv")
#skip.append("expat")
append_autogenargs('gstreamer', '--enable-introspection=no')
append_autogenargs('gst-plugins-base', '--enable-introspection=no')
append_autogenargs('gst-plugins-good', '--enable-introspection=no --disable-deinterlace')
append_autogenargs('gst-plugins-bad-1.0', '--enable-opengl=no --disable-apple_media')

#change moduleset
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"

comment:19 Changed 10 months ago by Antoine Martin

This does not look ready to me, or I need some more precise instructions.

Following, more or less: https://winswitch.org/dev/macosx.html but with the new customrc on a 10.10 system. Found some useful commands along the way:

$ sw_vers -productVersion
10.10.5
$ xcrun --show-sdk-version
10.11
$ xcrun --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk

And now for the (long) list of problems:

  • during boostrap:
    ERROR: no certificate subject alternative name matches
    	requested host name `cmake.org'.
    
  • more problematic during the build step:
    *** Configuring itstool *** [5/116]
    (..)
    checking for python module libxml2... ./configure: line 2422: 16059 Done                    echo "import $py_module"
         16060 Abort trap: 6           | python - >&/dev/null
    not found
    configure: error: Python module libxml2 is needed to run this package
    

Probably the same cause as this other error:

Making all in doc
/Users/osx/gtk/inst/etc/xml/catalog:1: parser error : Document is empty
I/O error : Attempt to load network entity http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd
(..)

And another related one (as I found out much later..):

GISCAN   GLib-2.0.gir
Traceback (most recent call last):
  File "./g-ir-scanner", line 53, in <module>
    from giscanner.scannermain import scanner_main
  File "./giscanner/scannermain.py", line 29, in <module>
    import tempfile
  File "/Users/osx/gtk/inst/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/Users/osx/gtk/inst/lib/python2.7/io.py", line 51, in <module>
    import _io
ImportError: No module named _io
make[2]: *** [GLib-2.0.gir] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of gobject-introspection: ########## Error running make -j 2  *** [27/116]

Same for gst-python and most python modules, pango (gtk-doc), gtk-mac-integration, librsvg, gtk-engines, gtk-quartz-engine ..
It tries to build pycairo-python2.6 - that looks wrong.
Looks like we want to force python to build earlier. I found that out later, after much time wasted. Most python modules then build OK.
Maybe just a _gtk_osx_use_jhbuild_python = True line?
Or maybe I was meant to build meta-osx-xpra-deps and not just a plain build?

Last edited 10 months ago by Antoine Martin (previous) (diff)

comment:20 Changed 10 months ago by Antoine Martin

At some point we will have to switch the existing build environment to a locked version, and move forward with the trunk / 64-bit / newer build env.
Until then, r12959 fixes building ffmpeg but this may also get in the way of supporting webcam capture in the future (we can't have both..).

comment:21 Changed 10 months ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

comment:22 Changed 8 months ago by Antoine Martin

FYI: bumped to openssl 1.1.0 in r13481, which requires a newer perl version than the one available on my 10.5.x VM, so then I had to figure out how to build the latest stable perl there:

tar -zxvf perl-5.24.0.tar.gz 
cd perl-5.24.0
./Configure -des -Dprefix={JHBUILD_PREFIX} \
    -Accflags="-arch i686 -I{JHBUILD_PREFIX}/include/ -L{JHBUILD_PREFIX}/lib/" \
    -Aldflags="-arch i686 -I{JHBUILD_PREFIX}/include/ -L{JHBUILD_PREFIX}/lib/"
make && make install

Bonus: this version seems to install cleanly without needing a chmod.
Downside: python-cryptography needs updating to 1.5 to support openssl 1.1 : https://cryptography.io/en/latest/changelog/. (done in r13482)

@smo: if that works for you, please add perl to the moduleset.

comment:23 Changed 8 months ago by Smo

Okay I ran into the perl issue as well.

I will add this to the moduleset but should we require it or leave it as an optional thing on older systems that need it?

comment:24 Changed 8 months ago by Antoine Martin

Hopefully we will be moving to a newer 64-bit build system soon, so this should be optional.

comment:25 Changed 8 months ago by Smo

r13692:13695 adds perl

Was some issues with the configure line since it is non standard. Not sure if I need to add the cc and ld flags so I left them out for now.

If the new openssl seems to install properly i'll see if I can remove some of the hacks from the moduleset for installing it in the first place.

comment:26 Changed 7 months ago by Smo

Tried with the latest changes to the moduleset on 64 bit. Seems to get further now. I got this error today

build/temp.macosx-10.9-x86_64-2.7/_openssl.c:415:10: fatal error: 'openssl/opensslv.h' file not found
#include <openssl/opensslv.h>
         ^
1 error generated.
error: command '/usr/bin/gcc' failed with exit status 1
*** Error during phase build of python-cryptography: ########## Error running python setup.py build *** [106/119]

Confirmed that openssl is installed.

comment:27 Changed 7 months ago by Antoine Martin

Milestone: 1.02.0
Priority: majorblocker

Using 10.10+ as a build platform will also allow us to fix #404 and #1305

comment:28 Changed 7 months ago by Antoine Martin

Tried again as part of #1340:

  • got an error with librsvg not finding gdk-pixbuf gir
  • ffmpeg complained about missing libvpx
  • occasionally had to switch things around with code-select: jhbuild requires the developer path but compiling things require the command line tools.. PITA - how do get get both to work at the same time?

comment:29 Changed 6 months ago by Antoine Martin

Also a blocker for #1340.

comment:31 Changed 4 months ago by Antoine Martin

After moving to a newer build host, we can start using a number of new APIs and do:

  • #776
  • webcam support

etc

comment:32 Changed 4 months ago by Antoine Martin

I have pointed my 10.5.x build VM to the stable 1.0.x branch after merging the latest set of safe updates (r14622).
Now we can start "breaking" things again to make 64-bit builds work properly, I am building on 10.10.x with xcode 7.2.1:

After that:

  • jhbuild buildone -f openssl to make sure it is rebuilt before python (this should probably be in the moduleset instead)
  • jhbuild build -f - and wait a while (see minor fixes below)
  • make a signed build in one swoop (based on the instructions in ticket:1340#comment:19):
    security unlock-keychain -p THEACCOUNTPASSWORD
    export CODESIGN_KEYNAME="Antoine Martin"
    ./make-all.sh
    
  • the resulting DMG / PKG are not much smaller than before: http://xpra.org/beta/osx

Things left TODO:

  • openssl is still a pain to build + install, still requires
    chmod +x ${JHBUILD_PREFIX}/lib/libssl.* ${JHBUILD_PREFIX}/lib/libcrypto.*
    
  • the gi bindings are missing and we really need those for gstreamer.. but [ https://github.com/anholt/libepoxy libepoxy] won't build (could be related to Adds support for OS X without X11), and it is required by GTK3, which is needed by the gobject-introspection module... I still managed to build it with jhbuild buildone -f pygobject3
  • gstreamer 1.10.x (version bumped in r14627, fix for gst-plugins-bad in r14631): downloads fail because they redirect to https and we get a certificate issue, again (as found in comment:19)
  • speex codec is missing, needs adding to moduleset?
  • pyobjc-core moaned:
    error: SDK root '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk' does not exist
    

Not sure why it has the same path 3 times as the command it runs to get that path only returns it once, but running like this allows me to move on:

export SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
python ./setup.py build
python ./setup.py install
  • IIRC, some versions of glib / gi were held back for compatibility reasons (with what?) - can we move forward again?

Note: there is a weird setup issue on my 10.10.x system which forces me to switch to command line tools to ensure the build finds gcc / clang but switch back to "developer" mode to launch jhbuild:

  • sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer
  • sudo xcode-select -switch /Library/Developer/CommandLineTools
Last edited 4 months ago by Antoine Martin (previous) (diff)

comment:33 Changed 4 months ago by Antoine Martin

Updates:

  • r14648 removes libdir from gtkglext
  • r14647 fixes mpfr download link, r14649 uses a reliable link (one that will not expire when the next release comes out), r14650 (undoes an unwanted change)
  • r14646 opus update to 1.1.3
  • r14645 fixes openssl install (no longer installed every time or needing chmod)
  • r14651 + r14652 + r14653: use "1.0" suffix for gstreamer modules, enable gobject-introspection
Last edited 4 months ago by Antoine Martin (previous) (diff)

comment:34 Changed 4 months ago by Antoine Martin

Some of these updates have been applied to the v1.0.x branch: r14657 (libpng), r14658 (mpfr), r14659 (opus)


TODO:

  • pygobject3 dependency? (build gtk3, surely that can be fixed?)
  • gi / glib updates?

Will ignore: pyobjc and other build env weirdness for now.

Last edited 4 months ago by Antoine Martin (previous) (diff)

comment:35 Changed 3 months ago by Smo

Everything builds for me on this machine

sw_vers && xcodebuild -version
ProductName:	Mac OS X
ProductVersion:	10.12.1
BuildVersion:	16B2555
Xcode 8.2.1
Build version 8C1002
Last edited 3 months ago by Smo (previous) (diff)

comment:36 Changed 3 months ago by Smo

Also works on this system

sw_vers && xcodebuild -version
ProductName:    Mac OS X
ProductVersion: 10.11.1
BuildVersion:   15B42
Xcode 7.1
Build version 7B91b

I put nasm in the modules part of the config because libjpeg-turbo will need it.
using this config
~/.jhbuildrc-custom

_gtk_osx_use_jhbuild_python = True
setup_sdk(target="10.9", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"

modules = [ "openssl", "python", "yasm", "nasm", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine","gtk-mac-integration-python", "meta-osx-xpra-deps"]

#change moduleset  
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"

comment:37 Changed 3 months ago by Antoine Martin

Resolution: fixed
Status: newclosed

Works for me too (doing another full rebuild from scratch to check), minus the weirdness of having to run xcodeselect on this build system. (PITA but not a showstopper)

We can follow up in #1404.
The next release will be 64-bit only for OSX! (and targetting 10.9+ only)

Last edited 3 months ago by Antoine Martin (previous) (diff)
Note: See TracTickets for help on using tickets.