Xpra: Ticket #840: 64-bit builds for OSX using moduleset

Split from #533. See ticket:533#comment:77

Minor fixes + tweaks already done: r9089, r9090.

Things that need fixing:

That's it!



Tue, 21 Apr 2015 07:41:48 GMT - Antoine Martin: description changed

FWIW: I've uploaded a beta 0.15.0 build which works quite well, see http://xpra.org/beta/osx/x86_64.

@afarr: it would be worth playing with it a bit to see if there are any other issues - stability issues (or even improvements), or if there are any differences in performance, etc..


Tue, 21 Apr 2015 08:15:46 GMT - Antoine Martin: owner changed

See also:

@smo: you may want to post the errors to the pyobjc mailing list, that's the blocker. The rest is relatively easy or optional.


Thu, 23 Apr 2015 04:46:35 GMT - Antoine Martin:

Lots of builds updates, see ticket:533#comment:79 and later comments. In particular: r9123 which removes lots of i386


Thu, 17 Sep 2015 15:55:58 GMT - Antoine Martin:

See also #984


Thu, 24 Sep 2015 22:06:04 GMT - Smo:

Updated in r10674 from upstream. We may still need to make further updates to this for our needs.


Tue, 27 Oct 2015 21:16:04 GMT - Smo:

This is still on hold until we can get gstreamer up to date.


Thu, 29 Oct 2015 18:41:08 GMT - Smo: owner changed

Can we reschedule this for a further release. Unless you think 64bit builds are not worth the bother and we can just close this.


Fri, 30 Oct 2015 01:33:18 GMT - Antoine Martin: owner, milestone changed

Re-scheduling. Worth doing, maybe after the next LTS so we have both 32-bit and 64-bit.


Thu, 05 Nov 2015 23:50:39 GMT - Smo:

Tried this myself today on osx 10.9.5 didn't get as far as I hoped some weird issue here.

(cd /Users/user/gtk/inst/_jhbuild/root-tango-icon-theme/Users/user/gtk/inst/share/icons/Tango/16x16 && /Users/user/gtk/inst/libexec/icon-name-mapping -c actions)
Bizarre copy of UNKNOWN in scalar assignment at /Users/user/gtk/inst/lib/perl5/5.16.2/darwin-thread-multi-2level/XML/Parser/Expat.pm line 125.
make[3]: *** [install-data-local] Error 25
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install-recursive] Error 1
*** Error during phase install of tango-icon-theme: ########## Error running make install DESTDIR=/Users/user/gtk/inst/_jhbuild/root-tango-icon-theme *** [41/97]

If I run this utility I also get the same issue.

/Users/user/gtk/inst/libexec/icon-name-mapping -c actions
Bizarre copy of UNKNOWN in scalar assignment at /Users/user/gtk/inst/lib/perl5/5.16.2/darwin-thread-multi-2level/XML/Parser/Expat.pm line 125.

Doesn't appear to be a 64bit issue really just some sort of broken perl module?


Wed, 02 Dec 2015 23:01:12 GMT - Smo:

Building on 10.11.1 with xcode 7.1 and these were the ones that failed

*** the following modules were not built *** [97/97]
libvpx sdl gst-plugins-bad-xpra pygtkglext meta-osx-xpra-deps

sdl was missing headers for X libvpx needs updating to 1.5.0 which builds fine need to update moduleset

gst-plugins-bad-xpra failed to build with an error about ssl header files we might be able to disable this

In file included from gstapexraop.c:33:
./gstapexraop.h:33:10: fatal error: 'openssl/rand.h' file not found
#include <openssl/rand.h>
         ^
In file included from gstapexplugin.c:29:
In file included from ./gstapexsink.h:32:
./gstapexraop.h:33:10: fatal error: 'openssl/rand.h' file not found
#include <openssl/rand.h>

pygtkglext failed with this error

gdkglext.override:36:10: fatal error: 'GL/gl.h' file not found
#include <GL/gl.h>

Thu, 03 Dec 2015 03:11:15 GMT - Antoine Martin:


Thu, 03 Dec 2015 20:10:40 GMT - Smo:

gtk/gdkgl/gdkglext.c change to #include <OpenGL/gl.h> afterwards this builds no problem.

I'm not sure about having multiple modulesets at the moment I include it by url in customrc file but one of the files that gets included also has a url we include from svn trunk


Sat, 27 Feb 2016 13:38:57 GMT - Antoine Martin:

This should allow us to get libyuv on osx: ticket:973#comment:3


Sun, 10 Apr 2016 06:22:20 GMT - Antoine Martin: milestone changed

Should be a priority for the next milestone.


Tue, 07 Jun 2016 21:27:26 GMT - Smo:

Still having issues trying to do this now I think its an issue with jhbuild during the configure phase of x264 it produces this stacktrace

*** Configuring x264 *** [72/116]
Traceback (most recent call last):
  File "/Users/user/.local/bin/jhbuild", line 32, in <module>
    jhbuild.main.main(sys.argv[1:])
  File "/Users/user/Source/jhbuild/jhbuild/main.py", line 155, in main
    rc = jhbuild.commands.run(command, config, args, help=lambda: print_help(parser))
  File "/Users/user/Source/jhbuild/jhbuild/commands/__init__.py", line 183, in run
    return cmd.execute(config, args, help)
  File "/Users/user/Source/jhbuild/jhbuild/commands/__init__.py", line 52, in execute
    return self.run(config, options, args, help)
  File "/Users/user/Source/jhbuild/jhbuild/commands/base.py", line 268, in run
    return build.build()
  File "/Users/user/Source/jhbuild/jhbuild/frontends/buildscript.py", line 163, in build
    error, altphases = module.run_phase(self, phase)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/__init__.py", line 420, in run_phase
    method(buildscript)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/autotools.py", line 217, in do_configure
    cmd = self._get_configure_cmd(buildscript)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/autotools.py", line 130, in _get_configure_cmd
    cmd = self.static_analyzer_pre_cmd(buildscript) + template % vars
KeyError: u'libdir'

Thu, 16 Jun 2016 15:18:20 GMT - Antoine Martin:

Any progress? We can skip x264 and do it by hand if need be.

Blocker for #1231.


Thu, 16 Jun 2016 22:17:20 GMT - Smo:

The problem is with libdir replacement in those strings maybe this isn't needed anymore since if I remove it I can compile this package through jhbuild.

I will commit fix for this shortly to trunk. Looks like a similar issue with ffmpeg as well.


Fri, 17 Jun 2016 05:13:17 GMT - Smo:

my current .jhbuildrc-custom for el capitan machine

setup_sdk(target="10.9", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "python", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine","gtk-mac-integration-python",
"gstreamer", "gst-plugins-base", "gst-plugins-good", "meta-gstreamer-1.0",
"meta-osx-xpra-deps"]
#skip iconv and add some switches to turn off stuff that was breaking
skip.append("libiconv")
#skip.append("expat")
append_autogenargs('gstreamer', '--enable-introspection=no')
append_autogenargs('gst-plugins-base', '--enable-introspection=no')
append_autogenargs('gst-plugins-good', '--enable-introspection=no --disable-deinterlace')
append_autogenargs('gst-plugins-bad-1.0', '--enable-opengl=no --disable-apple_media')
#change moduleset
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"

Fri, 17 Jun 2016 15:15:36 GMT - Antoine Martin:

This does not look ready to me, or I need some more precise instructions.

Following, more or less: https://winswitch.org/dev/macosx.html but with the new customrc on a 10.10 system. Found some useful commands along the way:

$ sw_vers -productVersion
10.10.5
$ xcrun --show-sdk-version
10.11
$ xcrun --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk

And now for the (long) list of problems:

Probably the same cause as this other error:

Making all in doc
/Users/osx/gtk/inst/etc/xml/catalog:1: parser error : Document is empty
I/O error : Attempt to load network entity http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd
(..)

And another related one (as I found out much later..):

GISCAN   GLib-2.0.gir
Traceback (most recent call last):
  File "./g-ir-scanner", line 53, in <module>
    from giscanner.scannermain import scanner_main
  File "./giscanner/scannermain.py", line 29, in <module>
    import tempfile
  File "/Users/osx/gtk/inst/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/Users/osx/gtk/inst/lib/python2.7/io.py", line 51, in <module>
    import _io
ImportError: No module named _io
make[2]: *** [GLib-2.0.gir] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of gobject-introspection: ########## Error running make -j 2  *** [27/116]

Same for gst-python and most python modules, pango (gtk-doc), gtk-mac-integration, librsvg, gtk-engines, gtk-quartz-engine .. It tries to build pycairo-python2.6 - that looks wrong. Looks like we want to force python to build earlier. I found that out later, after much time wasted. Most python modules then build OK. Maybe just a _gtk_osx_use_jhbuild_python = True line? Or maybe I was meant to build meta-osx-xpra-deps and not just a plain build?


Mon, 04 Jul 2016 09:25:49 GMT - Antoine Martin:

At some point we will have to switch the existing build environment to a locked version, and move forward with the trunk / 64-bit / newer build env. Until then, r12959 fixes building ffmpeg but this may also get in the way of supporting webcam capture in the future (we can't have both..).


Tue, 12 Jul 2016 16:52:22 GMT - Antoine Martin: milestone changed

Milestone renamed


Sat, 27 Aug 2016 04:18:07 GMT - Antoine Martin:

FYI: bumped to openssl 1.1.0 in r13481, which requires a newer perl version than the one available on my 10.5.x VM, so then I had to figure out how to build the latest stable perl there:

tar -zxvf perl-5.24.0.tar.gz
cd perl-5.24.0
./Configure -des -Dprefix={JHBUILD_PREFIX} \
    -Accflags="-arch i686 -I{JHBUILD_PREFIX}/include/ -L{JHBUILD_PREFIX}/lib/" \
    -Aldflags="-arch i686 -I{JHBUILD_PREFIX}/include/ -L{JHBUILD_PREFIX}/lib/"
make && make install

Bonus: this version seems to install cleanly without needing a chmod. Downside: python-cryptography needs updating to 1.5 to support openssl 1.1 : https://cryptography.io/en/latest/changelog/. (done in r13482)

@smo: if that works for you, please add perl to the moduleset.


Fri, 09 Sep 2016 00:39:12 GMT - Smo:

Okay I ran into the perl issue as well.

I will add this to the moduleset but should we require it or leave it as an optional thing on older systems that need it?


Fri, 09 Sep 2016 02:13:47 GMT - Antoine Martin:

Hopefully we will be moving to a newer 64-bit build system soon, so this should be optional.


Tue, 13 Sep 2016 19:33:56 GMT - Smo:

r13692:13695 adds perl

Was some issues with the configure line since it is non standard. Not sure if I need to add the cc and ld flags so I left them out for now.

If the new openssl seems to install properly i'll see if I can remove some of the hacks from the moduleset for installing it in the first place.


Tue, 20 Sep 2016 22:48:36 GMT - Smo:

Tried with the latest changes to the moduleset on 64 bit. Seems to get further now. I got this error today

build/temp.macosx-10.9-x86_64-2.7/_openssl.c:415:10: fatal error: 'openssl/opensslv.h' file not found
#include <openssl/opensslv.h>
         ^
1 error generated.
error: command '/usr/bin/gcc' failed with exit status 1
*** Error during phase build of python-cryptography: ########## Error running python setup.py build *** [106/119]

Confirmed that openssl is installed.


Tue, 27 Sep 2016 09:47:05 GMT - Antoine Martin: priority, milestone changed

Using 10.10+ as a build platform will also allow us to fix #404 and #1305


Thu, 13 Oct 2016 16:49:21 GMT - Antoine Martin:

Tried again as part of #1340:


Fri, 28 Oct 2016 10:02:41 GMT - Antoine Martin:

Also a blocker for #1340.


Wed, 02 Nov 2016 02:16:30 GMT - Antoine Martin:

gstreamer 1.10: https://gstreamer.freedesktop.org/releases/1.10/.


Fri, 16 Dec 2016 19:45:05 GMT - Antoine Martin:

After moving to a newer build host, we can start using a number of new APIs and do:

etc


Wed, 28 Dec 2016 08:42:03 GMT - Antoine Martin:

I have pointed my 10.5.x build VM to the stable 1.0.x branch after merging the latest set of safe updates (r14622). Now we can start "breaking" things again to make 64-bit builds work properly, I am building on 10.10.x with xcode 7.2.1:

After that:


Things left TODO:

Not sure why it has the same path 3 times as the command it runs to get that path only returns it once, but running like this allows me to move on:

export SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
python ./setup.py build
python ./setup.py install

Note: there is a weird setup issue on my 10.10.x system which forces me to switch to command line tools to ensure the build finds gcc / clang but switch back to "developer" mode to launch jhbuild:


Fri, 30 Dec 2016 03:15:18 GMT - Antoine Martin:

Updates:


Fri, 30 Dec 2016 04:08:57 GMT - Antoine Martin:

Some of these updates have been applied to the v1.0.x branch: r14657 (libpng), r14658 (mpfr), r14659 (opus)


TODO:

Will ignore: pyobjc and other build env weirdness for now.


Sat, 14 Jan 2017 05:56:46 GMT - Smo:

Everything builds for me on this machine

sw_vers && xcodebuild -version
ProductName:	Mac OS X
ProductVersion:	10.12.1
BuildVersion:	16B2555
Xcode 8.2.1
Build version 8C1002

Sat, 14 Jan 2017 06:02:40 GMT - Smo:

Also works on this system

sw_vers && xcodebuild -version
ProductName:    Mac OS X
ProductVersion: 10.11.1
BuildVersion:   15B42
Xcode 7.1
Build version 7B91b

I put nasm in the modules part of the config because libjpeg-turbo will need it. using this config ~/.jhbuildrc-custom

_gtk_osx_use_jhbuild_python = True
setup_sdk(target="10.9", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "openssl", "python", "yasm", "nasm", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine","gtk-mac-integration-python", "meta-osx-xpra-deps"]
#change moduleset
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"

Sat, 14 Jan 2017 06:21:36 GMT - Antoine Martin: status changed; resolution set

Works for me too (doing another full rebuild from scratch to check), minus the weirdness of having to run xcodeselect on this build system. (PITA but not a showstopper)

We can follow up in #1404. The next release will be 64-bit only for OSX! (and targetting 10.9+ only)


Mon, 21 Oct 2019 02:37:33 GMT - Antoine Martin:

webcam: see #1231


Sat, 23 Jan 2021 05:07:34 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/840