Xpra: Ticket #533: migrate osx build to using jhbuild and modulesets



Fri, 14 Mar 2014 22:32:45 GMT - Smo: status changed

Started work on this here is the first start. Hoping these can be added to svn somewhere.

Still needs work for dependencies however they all seem to build nicely. Only one patch was needed for "nasm" to support make install DESTDIR= or jhbuild complains.

I'm attaching the first files to this ticket.


Fri, 14 Mar 2014 22:33:31 GMT - Smo: attachment set

Xpra Moduleset


Fri, 14 Mar 2014 22:33:37 GMT - Smo: attachment set


Mon, 17 Mar 2014 19:33:23 GMT - Smo:

Added these in r5829

After doing

jhbuild bootstrap
jhbuild build
jhbuild buildone gstreamer
jhbuild buildone gst-plugins-base
jhbuild buildone gst-plugins-good

You should be able to now cd to trunk/osx/jhbuild

Run jhbuild -m ./xpra.modules build meta-osx-xpra-deps

This should build and install everything else necessary to build xpra..

Currently it is missing (py)gtkglext which is the next step to be worked on.

Optional if you want subversion updated and installed you can try this as well

jhbuild -m ./xpra.modules build meta-xpra-subversion


Fri, 21 Mar 2014 11:14:27 GMT - Antoine Martin:

Very good stuff, what a timesaver overall! I've updated the winswitch osx build page to refer to this moduleset.

For reference here's a link to the jhbuild directory


Only found a few minor things that we want to tweak / note:


Tue, 08 Apr 2014 23:42:58 GMT - Smo:

I've had mixed results with not using --insecure or -k with curl in the end I ended up having to add the flag to the tarball.py within jhbuild to get by some issues.


Mon, 14 Apr 2014 18:28:56 GMT - Smo:


Tue, 15 Apr 2014 06:07:06 GMT - Antoine Martin:

New updates:

For ffmpeg, I am now using a better, more limited configure line (also on win32):

./configure --prefix=${JHBUILD_PREFIX} \
    --cpu=i686 --enable-runtime-cpudetect --enable-pic --enable-memalign-hack \
    --enable-static --enable-shared --enable-gpl \
    --disable-everything \
    --enable-swscale --enable-libx264 --enable-decoder=h264 \
    --enable-libvpx --enable-decoder=vp8 --enable-decoder=vp9 --enable-decoder=hevc \
    --disable-protocol=tcp --disable-protocol=rtp \
    --disable-filter=aformat --disable-filter=crop --disable-filter=setpts \
    --disable-filter=anull --disable-filter=format --disable-filter=trim \
    --disable-filter=atrim --disable-filter=null \
    --disable-programs --disable-avfilter

It took a lot of fiddling and trial and error:


Tue, 15 Apr 2014 09:28:07 GMT - Smo:

Added patch for gtkglext in r6096

Can use it by doing something like this

git clone git://git.gnome.org/gtkglext
cd gtkglext
curl -O http://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtkglext.patch
patch -p1 <xpra-gtkglext.patch
autoreconf -fiv
./configure --prefix=${JHBUILD_PREFIX}
make && make install

This isn't ideal but jhbuild doesn't allow me to apply a patch when pulling source from git only when using tarballs

Next is pygtkglext.


Tue, 15 Apr 2014 19:50:59 GMT - Smo:

Added

python-macholib
python-modulegraph
python-altgraph

in r6098

As I was unable to build without these perhaps easy_install pulls these in for us normally?

After compiling and installing these I was able to build and package xpra.

Installing pygtkglext is still quite a hack as the installing portion usually fails and requires you to install them manually into site-packages/


Fri, 18 Apr 2014 09:33:44 GMT - Antoine Martin:

New ones:

These are going to keep accumulating until this ticket is closed..

As for the question above: macholib, modulegraph and altgraph are normally pulled in automatically when installing py2app via easy_install.


Thu, 01 May 2014 20:51:56 GMT - Smo:


Fri, 09 May 2014 04:40:21 GMT - Antoine Martin: attachment set

patch for building git version of pygtkglext


Fri, 09 May 2014 04:45:52 GMT - Antoine Martin:

The patch above builds pygtkglext for me using:

PYTHONPATH=$JHBUILD_PREFIX/share/pygobject/2.0/:$PYTHONPATH python2.7 \
    ./setup.py build

We cannot use setup.py for installing (see #226 for details), so we install it with rsync (ugly):

rsync -rpvlgto build/lib.macosx*/gtk/* $JHBUILD_PREFIX/lib/python2.7/site-packages/gtk-2.0/gtk/

Note: be aware of ticket:563#comment:14 when choosing which version to use (git tree vs 1.1 snapshots)


Fri, 09 May 2014 05:13:10 GMT - Antoine Martin: attachment set

1.2 snapshot for osx found here: http://lists.gnu.org/archive/html/bug-gnubg/2011-04/msg00006.html


Wed, 14 May 2014 10:20:09 GMT - Antoine Martin:

Newer versions should be updated in the moduleset:

Note to smo: work is going to keep piling on until you close this ticket...


Thu, 15 May 2014 18:20:19 GMT - Smo:

updated versions in r6479

Compiling this whole moduleset to make sure its still working. I will use the patch that I made against gtkglext this time but should I be using this new hack that you attached? Have you tried this?


Fri, 16 May 2014 04:26:57 GMT - Antoine Martin:

The 1.2.0 patched version I posted is a newer version with some osx patches already applied, in particular the gtk>=2.20 compatibility patches. I haven't found any regressions, so I think we should use this rather than patching an older version ourselves.


Tue, 27 May 2014 08:07:32 GMT - Antoine Martin:

New:


Tue, 03 Jun 2014 03:35:59 GMT - Antoine Martin:

And more updates will continue to accumulate until this ticket is closed..

Today:


Tue, 10 Jun 2014 05:15:10 GMT - Smo:

PyOpenGL version update in r6691 Python 2.7.7 I also changed the url in .jhbuildrc-custom and recompiled everything with no issues.

I have some modifications that work well for this as well that should be updated in the winswitch osx dev page.

Can we host gtkglext and the patch for pygtkglext somewhere on xpra.org so that I can put those into the moduleset as well?

Was your pygtkglext patch for the tarball?


Tue, 10 Jun 2014 05:21:39 GMT - Antoine Martin:

Can we host gtkglext and the patch for pygtkglext somewhere on xpra.org so that I can put those into the moduleset as well?


Sure, point me to them and I'll upload them.

Was your pygtkglext patch for the tarball?


Which one?


Tue, 10 Jun 2014 07:08:52 GMT - Smo:

They were both attachments to this ticket that you posted before however as I see now the pygtkglext patch is for git which won't work with jhbuild.

We can however put the hacked copy of gtkglext in the moduleset and that removes one more thing.


Tue, 10 Jun 2014 07:55:51 GMT - Antoine Martin:

Done: http://xpra.org/src/gtkglext-1.2.0.osx-hack.tar.gz


Wed, 11 Jun 2014 06:38:52 GMT - Smo:

Added gtkglext-1.2.0.osx-hack.tar.gz in r6720

had to configure with --enable-gtk-doc or make install failed with missing html doc file


Wed, 11 Jun 2014 15:26:22 GMT - Antoine Martin:

I've had to pull the 0.12.4 OSX build because pygtkglext is now completely broken (not that I could easily have noticed since I don't have GL in my macosx vms...), rebuilding it against the latest gtk+ update does not seem sufficient to make it run, see #593. You can easily get a crash just by trying to import gtkgl:

python -c "from gtk.gtkgl import *"

The error seems to be the one mentioned here Fatal Python error: can't initialize module gtk.gtkgl but I can't see anything wrong, this same source code used to build and run fine...


Thu, 12 Jun 2014 04:55:39 GMT - Antoine Martin:

I've just reviewed the GTK+ changes since 2.24.16, and we want quite a few of those (many are also relevant to win32 rebuilds discussed in ticket:263#comment:10):


Mon, 14 Jul 2014 23:25:50 GMT - Smo:

r6897 updates many libraries

I had issues trying to rebuild after this as gst-plugins-ugly links against x264

jhbuild -m ./xpra.modules uninstall gst-plugins-ugly-xpra
jhbuild -m ./xpra.modules -f buildone gst-plugins-ugly-xpra

Should be enough to get it building properly again.


Thu, 24 Jul 2014 08:47:13 GMT - Antoine Martin: priority changed

It's taken a while but I think we will soon be able to close this ticket!

Please update as discussed:

Finally, you can find three patches which allow us to build (py)gtkglext from git which I have added to svn in r6940 + r6941:

If needed for the moduleset, we can host the patches in a more canonical location, or even make a patched source archive with the patches already applied.

Note: the pygtkglext patch is not so different from the previous version (which may still work?), and I've seen weird things happen as I rebuilt the libraries. I believe jhbuild leaves some crud behind and we must forcibly remove the build directories before trying to rebuild a library. Something to keep in mind.. since I don't see a way of forcing jhbuild to completely remove a build directory.

It may be worthwhile adding a switch somewhere so we can get debug builds for all those libraries. Not sure exactly how to add -g to the CFLAGS however.


Once that's all done and tested, please re-assign to me so I can close this.


Fri, 25 Jul 2014 01:40:30 GMT - Antoine Martin:

Dammit, looks like we're not the first ones to hit this "invalid drawable" bug (recorded in ticket:563#no4 - which is closed, so moving here):

But for gtkgl on osx, the best I found is this (which has a dead link in it - it should point here instead: GtkGLExt OS X Quartz hack patch):

I think I see it just by running gl_check which should help in testing a fix.

Running the examples in gtkglext works without major problems, but the examples in pygtkglext show the invalid drawable problem. For the record, we can also build pygtkglext-1.1.0 with the patch above, ignoring the changes to autogen.sh and then using:

CFLAGS="-I/usr/X11/include" ./configure --prefix=${JHBUILD_PREFIX}

But the bug remains..

We can also build gtkglext with debug enabled:

    --enable-debug=[no/minimum/yes]

Fri, 25 Jul 2014 05:56:58 GMT - Antoine Martin: priority changed

Edited comment:25 with instructions I have tested repeatedly from scratch, this seem to fix the "invalid drawable" bug from ticket:563#no4, now tracked in ticket:593#comment:4 (at least it works on my 10.5 and 10.6 test machines so far)

Let's try to close this ticket whilst things work OK! (so raising to blocker)

If jhbuild doesn't allow us to apply the quartztagfix patch after the autoreconf step, we can just generate our own archive from git and stick it in svn.


Wed, 06 Aug 2014 04:19:12 GMT - Antoine Martin: attachment set

gtkglext git snapshot


Wed, 06 Aug 2014 04:19:47 GMT - Antoine Martin: attachment set

pygtkglext git snaptshot


Thu, 07 Aug 2014 19:07:37 GMT - Smo:

I've updated most of the stuff from comment:25

I've had to make a copy of the modulesets-stable to our source control so that I can modify them to update a few modules. We can contribute this upstream when we are satisfied everything is working.

r7184 brings in all the necessary changes to our moduleset r7186 updates many libraries to current versions

I still have to build gtkglext and pygtkglext by hand but the instructions work nicely along with the patches. We may have to make source tarballs for these to apply the patches.

I've tried to find a nicer way to fix autoconf for gtkglext so we don't have to do the second patch but didn't figure it out yet.

I've tested building all of these on an mac mini running osx 10.6.8 64bit hence all the build switches I commited for 32bit builds

I am now in the process of building this all again on a mac mini running 10.5.8 32bit but I assume it will work even easier than the other machine.

So again we are very close to closing this ticket but still some work needs to be done.

This is what my .jhbuildrc-custom looks like at the moment on the 32bit machine

module_autogenargs['cairo'] = '--enable-ft=no'
setup_sdk(target="10.5", sdk_version="10.5", architectures=["i386"])
os.environ["CC"] = "/usr/bin/gcc-4.2"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "python",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg",
"meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine",
"gtk-mac-integration-python" ]
#skip iconv and add some switches to turn off stuff that was breaking
skip.append("libiconv")
append_autogenargs('gstreamer', '--enable-introspection=no')
append_autogenargs('gst-plugins-base', '--enable-introspection=no')
append_autogenargs('gst-plugins-good', '--enable-introspection=no --disable-deinterlace')
#change moduleset
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"

Thu, 07 Aug 2014 19:59:03 GMT - Smo:

More problems it doesn't use bootstrap.modules from that location it only uses it from local ~/Source/jhbuild/modulesets/bootstrap.modules

Which is troublesome as we need to update gtk-doc so that we can build new glib if memory serves right I will also need another package to build new gtk-doc

I checked bootstrap.modules against the one locally and the only change was the change I made so its safe to use this one to edit for now.

Unfortunatly I need to just copy over the one that exists with ours to do this not ideal

cd ~/Source/jhbuild/modulesets && curl -O http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/bootstrap.modules

Then run jhbuild bootstrap


Thu, 07 Aug 2014 20:33:47 GMT - Smo:

Ran into another problem

gtk-doc requires itstool but itstool requires python

configure: error: itstool not found
*** Error during phase configure of gtk-doc: ########## Error running ./configure --prefix /Users/blank/gtk/inst --libdir '/Users/blank/gtk/inst/lib' --disable-scrollkeeper --with-xml-catalog=$JHBUILD_PREFIX/etc/xml/catalog   *** [16/19]
jhbuild buildone python
jhbuild buildone libxml2
jhbuild buildone itstool
jhbuild bootstrap

lets me get past this point but I guess python, libxml2 and itstool need to move to bootstrap? Or build gtk-doc without itstool?

After this I get

*** success *** [19/19]

That is enough to move on to the next steps


Sat, 09 Aug 2014 04:40:51 GMT - Antoine Martin:

Please see also:


Tue, 19 Aug 2014 17:17:09 GMT - Smo:

I'll add the md5sums to the missing packages. Looks like we still need a patch for GTK for dual monitor support I found that out today.

As for (py)gtkglext I really think we should probably just make snapshots of the repos and host our own .tar.bz2 for those 2 so we can use them with jhbuild.

What would be a good place to stick these?


Wed, 20 Aug 2014 16:26:19 GMT - Antoine Martin:


Thu, 21 Aug 2014 05:19:12 GMT - Smo:

I found a way to have jhbuild pull our patch and run it after autoreconf however I can't seem to build from the gtkglext git snapshot in comment:33

I always end up with a build failure like this

i686-apple-darwin9-gcc-4.2.1: .libs/gdkglversion.o: No such file or directory
i686-apple-darwin9-gcc-4.2.1: .libs/gdkglinit.o: No such file or directory
i686-apple-darwin9-gcc-4.2.1: .libs/gdkglquery.o: No such file or directory
i686-apple-darwin9-gcc-4.2.1: .libs/gdkglconfig.o: No such file or directory
i686-apple-darwin9-gcc-4.2.1: .libs/gdkglcontext.o: No such file or directory
i686-apple-darwin9-gcc-4.2.1: .libs/gdkgldrawable.o: No such file or directory
i686-apple-darwin9-gcc-4.2.1: .libs/gdkglpixmap.o: No such file or directory
i686-apple-darwin9-gcc-4.2.1: .libs/gdkglwindow.o: No such file or directory
i686-apple-darwin9-gcc-4.2.1: .libs/gdkglglext.o: No such file or directory
make[4]: *** [libgdkglext-quartz-1.0.la] Error 1
make[3]: *** [all-recursive] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1

I followed the instructions from comment:25 just to be sure and it works no problem so i'm thinking maybe that snapshot is bad?


Thu, 21 Aug 2014 05:20:50 GMT - Antoine Martin:

I doubt the snapshot is bad, but it's possible.

Those errors are about object files, which are meant to be generated during the build, so a bad snapshot shouldn't be causing this unless it messed with the build scripts.


Wed, 27 Aug 2014 11:58:17 GMT - Antoine Martin:

RELEASE: ORC 0.4.22 released


Thu, 28 Aug 2014 14:44:25 GMT - Antoine Martin:

https://pypi.python.org/pypi/Pillow/2.5.3 low priority: only CVE entries for things we don't use.


Thu, 28 Aug 2014 21:25:35 GMT - Smo: attachment set

gtkglext snapshot from git works for building osx with patches


Thu, 28 Aug 2014 21:46:00 GMT - Smo:

r7472 adds gtkglext and pygtkglext

The source tarball for gtkglext is one I attached to this ticket so this should probably change.

I made this one by doing this

git clone git://git.gnome.org/gtkglext && cd gtkglext
git archive --prefix=gtkglext-1.2.0/ -o ../gtkglext-1.2.0.tar HEAD
cd .. && bzip2 gtkglext-1.2.0.tar

Seems to build now with that snapshot i'm not sure what the difference is. Feel free to make a new one and just provide me with the url and I will update it.

I will now delete my ~/gtk directory and start this all from scratch hopefully now it will just build everything we need.


Fri, 29 Aug 2014 05:49:45 GMT - Antoine Martin:

I've uploaded your snapshot in the source dir: http://xpra.org/src/gtkglext-1.2.0.tar.bz2


Sun, 14 Sep 2014 15:21:26 GMT - Antoine Martin:

Note: Cython should be updated to 0.21.x (I have reported fairly big bug, they may well do a point release before too long).

I have created a separate ticket for OSX GTK3: #680.


Wed, 17 Sep 2014 01:00:10 GMT - Smo:

Many new changes seems to work quite well now. Still needs some polish but pretty happy with it now.

You'll need the .jhbuildrc-custom from r7651 which includes our modulesets from by url.

r7642 adds gtk-doc from bootstrap.modules to our moduleset as we need itstool and libxml2 which requires python to build it now.

r7648 adds a patch for liboil to fix autreconf from breaking found it here https://github.com/GNOME/frogr/tree/master/osx/jhbuild/patches

r7652 Cython 0.21

You need to overwrite ~/Source/jhbuild/modulesets/bootstrap.modules with ours I haven't found any way to use ours instead.

If everything is in place I'm able to do

jhbuild bootstrap
jhbuild build
jhbuild build meta-subversion-xpra

bootstrap has 17 steps and build has 88

I had to modify ~/Source/jhbuild/jhbuild/versioncontrol/tarball.py to add -k to curl as a lot a few https tarballs were failing since the certificates on this old mac are outdated.


Wed, 17 Sep 2014 01:00:30 GMT - Smo: owner, status changed


Wed, 17 Sep 2014 08:08:47 GMT - Antoine Martin:

My best guess at how to run this build from start to finish (took a few tries to get every step) - to be used as documentation template? There should be at least a README file telling you what to do:

I got some errors, but I think they were due to the fact that I had originally missed some of the undocumented steps.


Build started at: Wed 17 Sep 09:42:47 ICT 2014

Build finished at: Wed Sep 17 16:46:12 ICT 2014

A fair amount of this time was spent downloading stuff again on my slow DSL line (as I had wiped the whole home directory to start clean), but also re-doing things, and fixing cert errors...


Comments:

Others can wait for #680: I've added the info there.


Wed, 17 Sep 2014 09:59:43 GMT - Antoine Martin: owner changed

It looks bad because of the amount of pending issues, but really it isn't. One more round and we should be able to close this... and move on to #680 and #640.

I've made a beta build (with some version updates on top, as per above) which works fine: http://xpra.org/beta/osx/x86/


Wed, 17 Sep 2014 10:03:24 GMT - Antoine Martin: attachment set

not fully tested moduleset updates


Thu, 18 Sep 2014 23:39:40 GMT - Smo:

Applied your patch uploaded the modulesets to a test webserver and tested them. Just a couple issues.

https://github.com/atgreen/libffi/issues/128

configure: error: Either a previously installed pkg-config or "glib-2.0 >= 2.16" could not be found. Please set GLIB_CFLAGS and GLIB_LIBS to the correct values or pass --with-internal-glib to configure to use the bundled copy.

things that were updated in r7693

I tested this like you did by updating a few versions then I recompiled the whole lot.

Looks good the client will require some testing to make sure nothing breaks as this is quite a big update.

Next I will add the winswitch stuff that is left i'm just not sure if I want to make a winswitch.modules and include it or just add it to the current moduleset.

I will write out some simplified instructions and attact them to this ticket. Your instructions from comment:43 are roughly correct except you don't have to copy all the .modules files they are included via urls


Fri, 19 Sep 2014 02:34:02 GMT - Antoine Martin:

Notes:


Thu, 02 Oct 2014 22:50:40 GMT - Smo:

I'm having issues trying to add the winswitch stuff can't link with this nxcomp library

ld: in /Users/user/gtk/inst/lib/libXcomp.so, can't link with bundle (MH_BUNDLE) only dylibs (MH_DYLIB)

I've been following the steps http://winswitch.org/dev/macosx.html for the winswitch stuff.

I will try a few more things.


Tue, 04 Nov 2014 21:56:19 GMT - Smo:

r8054

I needed to use the autotools module to do some hacks to download a 2nd tarball and extract it plus make a link.

This might not be the smartest way to do this perhaps I can just have it download a simply makefile that does all this stuff instead of hacking it into jhbuild that way satisfying both problems.

I will add in gtk-mac-bundler soon as well as it doesn't seem to matter where it is actually extracted to it simply makes a helper script in ~/.local/bin pointing to it. With a working patch this should install cleanly.


Mon, 01 Dec 2014 20:08:06 GMT - Antoine Martin:

More updates needed (I've already done most of them outside the moduleset):

more?


Thu, 11 Dec 2014 19:15:01 GMT - Antoine Martin:


Tue, 16 Dec 2014 22:08:49 GMT - Smo:

gdb is still problematic to add to the moduleset since it requires one extra step to make it work after installing


Tue, 16 Dec 2014 22:31:35 GMT - Smo:

New serf uses scons build system which is problematic to adding to moduleset as well.

I believe I came across this before so unless we absolutely need the latest version I don't think we should update.


Wed, 17 Dec 2014 04:12:00 GMT - Antoine Martin:

the changeset was r8250.

FWIW: I'm still on ffmpeg 2.4 (2.4.4 as of this writing), not switching to 2.5 yet.


Thu, 18 Dec 2014 09:15:50 GMT - Antoine Martin:

ORC 0.4.23 released


Thu, 12 Feb 2015 21:52:22 GMT - Smo:

r8653 brings in many changes from upstream git repo

https://git.gnome.org/browse/gtk-osx/tree/modulesets-stable

Few were left behind if we were using newer versions like for gmp and python 2.7.9

Still needs testing which I'm rebuilding right now will update with instructions if successful


Thu, 19 Feb 2015 21:51:48 GMT - Smo:

r8668 Newer versions of gtk-mac-integration will not build on osx < 10.6

r8671 Host patch for gtk-mac-integration as the old one seems to be missing

r8673 removes 2 modules we had put in that are now bootstrap modules

r8676 adds python-nose as we didn't have this and is needed now by lz4

r8678 we don't use scons for building serf

r8683 add gtk-mac-bundler to dependencies for meta-osx-xpra-deps this install works now

r8684 added proper old catched patch for gtk-mac-integration


Thu, 19 Feb 2015 21:56:08 GMT - Smo:

Will update soon with new jhbuildrc-custom and a few tweaks.

Haven't found a way to override bootstrap.modules yet so I typically copy over it with ours.

Also I typically have to also make a change in jhbuild/versioncontrol/tarball.py to make curl use the '-k' option as certificate checking is a bit old and often fails when downloading files.

Another bug I discovered is that you want to build everything outside of jhbuild shell so that you use the system python for calling jhbuild. If you do not the ssl module in python 2.7.9 now checks certificates by default and this also causes problems as they use urllib2 to download and cache patches.


Fri, 20 Feb 2015 02:57:57 GMT - Smo:

Here are some rough instructions should be good for updating the winswitch osx dev page if this all works.

curl -k -O https://git.gnome.org/browse/gtk-osx/plain/gtk-osx-build-setup.sh
sh gtk-osx-build-setup.sh
export PATH=$PATH:~/.local/bin/
curl -k -o ~/Source/jhbuild/modulesets/bootstrap.modules http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/bootstrap.modules
sed -i -e 's/^setup_sdk/#setup_sdk/g' .jhbuildrc-custom
cat << EOF >> .jhbuildrc-custom
setup_sdk(target="10.5", sdk_version="10.5", architectures=["i386"])
os.environ["CC"] = "/usr/bin/gcc-4.2"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "python", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine","gtk-mac-integration-python",
"gstreamer", "gst-plugins-base", "gst-plugins-good",
"meta-osx-xpra-deps"]
#skip iconv and add some switches to turn off stuff that was breaking
skip.append("libiconv")
append_autogenargs('gstreamer', '--enable-introspection=no')
append_autogenargs('gst-plugins-base', '--enable-introspection=no')
append_autogenargs('gst-plugins-good', '--enable-introspection=no --disable-deinterlace')
#change moduleset
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
EOF
jhbuild bootstrap
jhbuild build
jhbuild build meta-subversion-xpra
jhbuild shell

Thu, 19 Mar 2015 18:30:23 GMT - Smo: owner changed

Another day another fun time with osx.

Found that I was missing libz.1.2.8.dylib not because of de-duplication but because new jhbuild modulesets build it instead of using the system one.

After wasting my type trying to debug and fix the de-duplication part of the make-app.sh script I found that I just needed to add this to Xpra.bundle so that it was actually included.

Now I've run into further problems with libjpeg-turbo and PIL which I will have to solve now.

2015-03-19 10:58:21,680 draw error
Traceback (most recent call last):
  File "/Users/test/gtk/inst/lib/python2.7/site-packages/xpra/client/ui_client_base.py", line 1961, in _do_draw
  File "/Users/test/gtk/inst/lib/python2.7/site-packages/xpra/client/client_window_base.py", line 423, in draw_region
  File "/Users/test/gtk/inst/lib/python2.7/site-packages/xpra/client/window_backing_base.py", line 476, in draw_region
  File "/Users/test/gtk/inst/lib/python2.7/site-packages/xpra/client/gtk2/window_backing.py", line 35, in paint_image
  File "/Users/test/gtk/inst/lib/python2.7/site-packages/xpra/client/window_backing_base.py", line 227, in paint_image
  File "PIL/Image.pyc", line 644, in tobytes
    one of :py:attr:`PIL.Image.NEAREST` (use nearest neighbour),
  File "PIL/ImageFile.pyc", line 191, in load
  File "PIL/Image.pyc", line 413, in _getdecoder
    the image.
IOError: decoder jpeg not available

Could be that PIL wasn't built with jpeg support as I can see the library is packaged but I am looking for any advice you can give me to debug this.


Thu, 19 Mar 2015 18:32:59 GMT - Smo:

Here is output from forcing PIL to rebuild. Looks like jpeg is compiled in so perhaps just missing something.

--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 2.5.3
platform     darwin 2.7.9 (default, Feb 19 2015, 11:53:52)
             [GCC 4.2.1 (Apple Inc. build 5577)]
--------------------------------------------------------------------
--- TKINTER support available
--- JPEG support available
*** OPENJPEG (JPEG2000) support not available
--- ZLIB (PNG/ZIP) support available
--- LIBTIFF support available
--- FREETYPE2 support available
*** LITTLECMS2 support not available
--- WEBP support available
*** WEBPMUX support not available
--------------------------------------------------------------------

Sun, 22 Mar 2015 05:25:17 GMT - Antoine Martin:

To build pycups on OSX 10.5.x, I had to add the following missing defines to cupsmodule.c before building:

#define CUPS_FORMAT_AUTO     "application/octet-stream"
#define CUPS_FORMAT_COMMAND  "application/vnd.cups-command"
#define CUPS_FORMAT_PDF      "application/pdf"
#define CUPS_FORMAT_POSTSCRIPT "application/postscript"
#define CUPS_FORMAT_RAW      "application/vnd.cups-raw"
#define CUPS_FORMAT_TEXT     "text/plain"
#define CUPS_GET_DOCUMENT   0x4027

It fails at runtime on 10.5. (I think that the version of cups in 10.5.x is just too old. I guess we could try to build a newer version of cups and bundle that too.. but at that point, we might as well make our own OS - so we just won't support 10.5 for printing)

$ python -c "import cups"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/MacAdmin/gtk/inst/lib/python2.7/site-packages/cups.so, 2): Symbol not found: _cupsCreateJob
  Referenced from: /Users/MacAdmin/gtk/inst/lib/python2.7/site-packages/cups.so
  Expected in: dynamic lookup

It does run with 10.6 onwards though, see ticket:598#comment:34.

PS: in 0.15, you also need rencode as a separate module.


Wed, 25 Mar 2015 05:38:14 GMT - Smo:

r8828 added configure switch for libjpeg-turbo to turn on v8 compatibility which now works with python-pillow

PyObjcTools? seems to be missing __init__.py in ~/gtk/inst/lib/python2.7/site-packages/PyObjCTools which causes py2app to not bundle it.

You can see the file here though which should be there https://bitbucket.org/ronaldoussoren/pyobjc/src/f4334b2f9f2d0b4a940eb9d0a6f1be98d5db9d02/pyobjc-framework-Cocoa/Lib/PyObjCTools/__init__.py?at=default

Not sure why this is missing will have to look into this some more. Once I added this file everything is packaged again.

My previous builds were always missing this stuff and I believe it may have been causing issues.


Wed, 25 Mar 2015 19:21:37 GMT - Smo:

r8835 adds patch for pygobject to get rid of annoying messages as noted here

http://winswitch.org/dev/macosx-manual.html

That is the last of the cleanup and everything seems to be running smoothly now


Thu, 26 Mar 2015 20:01:56 GMT - Smo:

Can't add patches for disutils stuff in jhbuild so I may have to add pycups some other way if I want to patch it to include these constants.

Builds fine when I add them same as you.


Sat, 28 Mar 2015 07:11:25 GMT - Antoine Martin: owner changed

I have gone through the steps in comment:58.

I've hit the following https certificate issues (this is a 10.5.x VM), worked around by adding --insecure to the curl download command from a shell:

(which means that it took me a lot longer than 3 hours to build because of the manual steps)

Also, the versions of many packages is way too old: netifaces, Pillow, Cython, numpy, etc.. (and probably many others too), see comment:43 (which must be out of date by now, but more up to date than the moduleset). Please go through them all as I did before and ensure we use the most up to date version, unless there is a good reason (Twisted had problems with winswitch - but we'll need the latest version to get the ssh fix anyway, so the fix will have to be in winswitch for that one)

I've seen the missing pyobjc warning, PIL is having problems too (module has no attribute 'Image'), and obviously pycups is also missing. If this helps, I've uploaded this beta osx x86 build as Xpra-0.15.0-r8825-MODULESET.dmg - beware: because of the issues above, it can't be used for anything useful.


Mon, 30 Mar 2015 18:29:55 GMT - Smo: owner changed

I can go through and add update all packages to their latest version that is not a problem.

The libjpeg problem I think I see the issue is that libjpeg.*.dylib needs to be added to Xpra.bundle and possible libz.*.dylib

Svn is marking this file as a binary file for some reason so I missed it when I was doing a diff.

Can you try adding those 2 and making another build that will probably fix up the PIL issue.

As I said in comment:62 pyobjctools seems to be missing init.py in site-packages after it is installed.


Tue, 31 Mar 2015 02:40:06 GMT - Smo: owner changed

r8873:r8889 updates versions in this moduleset

I will have to rebuild and test it all again to make sure everything is fine.


Tue, 31 Mar 2015 08:42:50 GMT - Antoine Martin:

For the record (and for my own reference), I have updated my OSX build machine with the latest:

Not updating pyopengl - will wait for the 3.1.1 proper.

The win32 build VM needed most of the same updates (added difficulty: both the python 2.7 and python 3.4 environments), done them too.

After that I uploaded a new beta build made from those environments, this may help in comparing with what you're getting (or with tickets such as #598).


FWIW: the OSX "moduleset" based builds are not usable at this point. So I've uploaded both a "MODULESET" build and a "normal" one in the osx beta area. PS: Some NS Autorelease Pool errors (in both builds now), somewhat similar to __NSAutoreleaseNoPool error … - just leaking. I haven't seen those before, or at least not recently, so this may be related to the update of pyobjc. But I will try the fix suggested in that link.


Thu, 02 Apr 2015 18:30:36 GMT - Smo:

r8903 rolled back pyobjc stuff to older versions however still seeing these kinds of errors

_NSAutoreleaseNoPool(): Object 0x1803a660 of class NSCFDictionary autoreleased with no pool in place - just leaking

I figured it would be an updated version causing this problem. the link in comment:68 said something about matplotlib but we don't use this. Could the problem be with numpy?


Thu, 02 Apr 2015 18:50:39 GMT - Smo: owner changed

r8904 rolls back numpy to 1.8.1 and still I get the same error messages as comment:69

Not sure where to look from here maybe we do need to fix this properly?


Thu, 02 Apr 2015 19:37:53 GMT - Smo:

Tried my client with --speaker=off

No more NSAutoreleaseNoPool messages. Must be something new in sound that causes this?


Fri, 03 Apr 2015 12:12:31 GMT - Antoine Martin: attachment set

my attempts at getting rid of the warnings


Fri, 03 Apr 2015 13:51:28 GMT - Antoine Martin:

The patch above doesn't help... As per Working with threads, it sounds like this is a threading issue but we already process all the packets in the main thread... not sure what to do (see ticket:669#comment:24).


Sat, 04 Apr 2015 16:58:15 GMT - Antoine Martin:

New update: libvpx 1.4.0, see #832. (download location has changed)


Sun, 05 Apr 2015 03:50:31 GMT - Antoine Martin:

I don't see libvorbis anywhere in the moduleset, but this one has been bumped to 1.3.5 last month.


Sun, 05 Apr 2015 04:53:34 GMT - Antoine Martin:

Pillow 2.8.1 has been released...


Mon, 20 Apr 2015 18:25:13 GMT - Antoine Martin: owner, status changed

@smo: FWIW, I am giving this a go on a clean OSX 10.9 (aka Mavericks) + Xcode 6.2 and the command line tools. Using these changes to .jhbuildrc-custom:

setup_sdk(target="10.9", sdk_version="10.9", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"

It's going to take a while... but so far so good! (the bootstrap stage finished without problems - yay!)


Tue, 21 Apr 2015 04:12:57 GMT - Antoine Martin: attachment set

blatant bug caught by newer versions of gcc / clang


Tue, 21 Apr 2015 05:19:47 GMT - Antoine Martin:

Hurdles (not mentioning the dozens of packages which spit out thousands of warnings because this is using clang instead of gcc):

This is fixed by applying attachment/ticket/533/quartz-style-fix.patch. Which we should apply in any case, using an uninitialized variable like this is a bug. It is probably fixed upstream already, so maybe we just need a newer version of the library.

That's because the configure line looks like this:

./configure --prefix ${JHBUILD_PREFIX} --libdir ${JHBUILD_PREFIX}/lib --build=i386-darwin CC="gcc -arch i386" CXX="g++ -arch i386"

Removing the CC and CXX definitions and using --build=x86_64-darwin allows it to build (completely removing --build= might also work). @smo: do we really need those overrides for the i386 case? Why? If so, how can we make this conditional?

Similar problem, solved by just adding --build=x86_64-darwin to the configure line. Then you hit this problem: https://trac.macports.org/browser/trunk/dports/multimedia/libtheora/files/patch-configure.diff?rev=118032.

Same problem. Since I am building on 10.9, I used --target=x86_64-darwin13-gcc. That's libvpx 1.3, we should try 1.4 to see if that builds ok.

Same problem.

Same problem. --cpu=i686 should be --cpu=x86_64

Lots of answers: Can't install PIL after Mac OS X 10.9, but what I did was simply:

ln -sf /System/Library/Frameworks/Tk.framework/Versions/8.5/Headers/X11 ${JHBUILD_PREFIX}/include/

Which is nice because it doesn't require root. But the build then fails with:

./src/video/quartz/SDL_QuartzVideo.h:94:5: error: unknown type name 'CGDirectPaletteRef'

Which is fixed using this: http://hg.libsdl.org/SDL/rev/e9466ead70e5. This looks like a cleaner fix: http://www.emaculation.com/doku.php/compiling_sheepshaver_basilisk#tidbits. (not tested). Then it will moan about missing X11/Xmd.h, and we don't care about this X11 video stuff so just add this to configure: --disable-video-x11.

Same arch problem. And then a new obscure one after changing arch to x86_64:

tmp-invert_limb_table.s:46:1: error: unknown directive
.hidden __gmpn_invert_limb_table
^
make[2]: *** [invert_limb_table.lo] Error 1

I tried the latest development version to see if that fixes the problem, sadly not. The solution turns out to be quite simple: just take out the whole arch thing and let the configure script guess what to do.

Note: I didn't wait to see if it was going to build, because it's wrong to build for i386... Same thing: just removed arch completely to build it.

Obscure errors:

error: missing ',' between enumerators
error: missing ',' between enumerators
In file included from Modules/objc/block_support.m:11:
In file included from Modules/objc/pyobjc.h:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:52:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSOperation.h:6:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/sys/qos.h:128:4: error: redefinition of
      enumerator '__AVAILABILITY_INTERNAL__MAC_10_10'
                        __QOS_CLASS_AVAILABLE_STARTING(__MAC_10_10, __IPHONE_8_0) = 0x19,
                        ^
...

I give up on this one, and the ones that depend on it (pyobjc cocoa, pyobjc quartz). We need it so I will have to find a fix.

Solved in a similar way to the X11 header problem:

ln -sf /System/Library/Frameworks/OpenGL.framework/Versions/A/Headers ${JHBUILD_PREFIX}/include/GL

@smo: a lot of the problems are because of the configure arguments, can't we use append_autogenargs for this? Then have one custom file for i386 and one for x86_64?

Small nitpick: when installing gtk-mac-bundler, it doesn't really install anything but links back to where the source is. Which means that when cleaning gtk/source... we break it. It would be worth copying to gtk/inst/lib and running install from there.

Important: we do need some of those updates. Pillow 2.8 for example.


Tue, 21 Apr 2015 08:13:49 GMT - Antoine Martin: owner, status changed

I installed pycups using easy_install, because I didn't see in the moduleset.. Then I hit disk space issues... made some space. Then I hit a PIL.Image packaging issue, needed to add libz to the bundle list, fixed in r9089. I am moving the 64-bit only issues to #840 (not many at all - at least so far), link to a beta 64-bit build there. Then I bumped a bunch of libraries: r9091 + r9092.

@smo: I am happy with this and I think we can finally close the ticket. I'm not sure if I will be using it for the 0.15.x release, but going forward I will definitely be using it for 64-bit builds (#840). Either close and follow up there, or re-assign to me if you have more fixes / updates.


Thu, 23 Apr 2015 02:14:09 GMT - Smo:

r9116 Many version updates from upstream r9119 libffi version downgrade r9120 + r9121 gtk-mac-integration version downgrade r9122 ffmpeg downgrade to latest in 2.5 branch since 2.6 requires newer libvpx and this won't build on osx 10.5


Thu, 23 Apr 2015 02:22:19 GMT - Smo:

r9123 remove i386 build options since they are only needed when compiling 32bit on 64bit systems.

I've tested these being removed on a 32bit system and all is fine without them.

This should help us moving to ticket #840. There are some version of libs that we can build on newer systems that we cannot for 32bit builds on 10.5 so we may have to split this up a bit going forward.

I tried to keep it as easy to update from upstream as possible so we can help test their stable builds as well.

Pycups requires a patch on osx 10.5 which is not possible with distutils instructions through jhbuild I may be able to hack this up though.


Thu, 23 Apr 2015 02:33:41 GMT - Smo:

r9124 commited the patch you put here for gtk-quartz-engine can't hurt r9125 add that patch for the moduleset


Thu, 23 Apr 2015 02:35:33 GMT - Smo:

I'm going to rebuild the lot from scratch again and if I don't have any serious issues with the 32bit builds I will close this and move on to the 64bit ticket.


Thu, 23 Apr 2015 04:46:29 GMT - Antoine Martin:

@smo:


Thu, 23 Apr 2015 07:10:18 GMT - Smo:

the downgrade to apr and apr-util in r9116 wasn't intentional I guess I missed that one when I was looking at it. This is probably something that should be fixed as apr and apr-util are now in bootstrap so they aren't necessary in xpra.modules. I will look into this further.

I will have to try ffmpeg 2.6.1 again and see what the error was it was related to vp9.

I'm also in the process of cleaning this up a bit. Made some mess when I was dealing with patches.


Thu, 23 Apr 2015 07:30:14 GMT - Smo:

My bad about ffmpeg 2.6.1 those were warnings and jhbuild wasn't printing out the actual error which was

LD      libavformat/libavformat.56.dylib
HTML    doc/ffmpeg-utils.html
HTML    doc/ffmpeg-scaler.html
HTML    doc/ffmpeg-resampler.html
Unknown open() mode ':encoding(utf-8-strict)' at /usr/bin/texi2html line 11111.
make: *** [doc/ffmpeg-utils.html] Error 255
make: *** Waiting for unfinished jobs....
Unknown open() mode ':encoding(utf-8-strict)' at /usr/bin/texi2html line 11111.
make: *** [doc/ffmpeg-scaler.html] Error 255
Unknown open() mode ':encoding(utf-8-strict)' at /usr/bin/texi2html line 11111.
make: *** [doc/ffmpeg-resampler.html] Error 255

adding --disable-doc to the configure line seems to fix this issue so I will commit those fixes


Thu, 23 Apr 2015 22:48:38 GMT - Smo:

Still a couple things to do here.


Fri, 24 Apr 2015 11:41:02 GMT - Antoine Martin: milestone set

(just setting milestone)


Thu, 30 Apr 2015 08:08:10 GMT - Antoine Martin:

Tested with a completely clean user account on my 10.5.8 VM and got this error:

configure: error: Oops, mp_limb_t is 32 bits, but the assembler code
in this configuration expects 64 bits.
You appear to have set $CFLAGS, perhaps you also need to tell GMP the
intended ABI, see "ABI and ISA" in the manual.
*** Error during phase configure of gmplib: ########## \
    Error running ./configure --prefix /Users/osx/gtk/inst --libdir '/Users/osx/gtk/inst/lib'    *** [71/97]

Fixed by re-adding --build=i386-darwin. That's odd, is anything else building for 64 bit?? @smo: we may have to re-add this one and workaround it for 64-bit.

Another really annoying problem is that the apr downloads seem to disappear with new releases (see http://www.eu.apache.org/dist/apr/), which is a really bad idea for those building from source like us, maybe we should mirror those? Otherwise we run the risk of not being able to build things in the future. I've used apr 1.5.2 for building and updated the moduleset, also updated subversion to 1.8.13


Thu, 30 Apr 2015 14:56:25 GMT - Antoine Martin:

I have just uploaded an osx beta build (tagged with r9198) made from this moduleset environment. @afarr: testing it wouldn't hurt, making sure there are no regressions from previous "non moduleset" builds.

PS: unless problems are reported against these builds, from now on all the 0.15 builds (and later) will be made from this "moduleset" environment.


Fri, 01 May 2015 03:02:07 GMT - Antoine Martin:

Just noticed an important update missing: gtk 2.24.27, see:

There are some crashes fixed in these releases.


Thu, 07 May 2015 20:53:19 GMT - Smo: status changed; resolution set

r9279 updates gtk to 2.24.27 builds fine

I'm going to close this now while I can and move on to 64bit builds.


Fri, 31 Jul 2015 17:20:32 GMT - Antoine Martin:

@smo: FYI r10091 bumped x264 to a newer stable snapshot. This caused hard to decipher problems at package time because some dylibs still referred to the old one. (libx264.142.dylib vs libx264.146.dylib)

Fixed by rebuilding gstreamer with:

jhbuild buildone -f

I wished jhbuild had the equivalent of revdep-rebuild under Gentoo. There might be a way of cooking up a check script using otool to verify all the dylibs found in gtk/inst/lib can be loaded. Until then, we'll fix things when they break and hope the breakage manifests itself at package time rather than at runtime..

Similar dependency problem in #971.

Follow up for 64-bit in #840.


Sun, 10 Jan 2016 13:03:08 GMT - Antoine Martin:

@smo: FYI lots of updates in r11631.


Thu, 16 Feb 2017 17:16:46 GMT - Antoine Martin:

Note: the pygtkglext patch should be updated to support "get_depth" which we commented out: #1443


Sat, 23 Jan 2021 04:58:34 GMT - migration script:

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/533