xpra icon
Bug tracker and wiki

Opened 3 years ago

Closed 9 months ago

Last modified 9 months ago

#641 closed task (fixed)

OSX .pkg creation scrips

Reported by: Scott Bennett Owned by: Smo
Priority: blocker Milestone: 0.17
Component: packaging Version: trunk
Keywords: Cc:

Description

Add OSX .pmdoc (Package Definition) with relative paths

Add make-pkg.sh script

Optionally add to make-all.sh

Change History (36)

comment:1 Changed 3 years ago by Scott Bennett

Formatted the generated xml files so they can be maintained by hand.

Figured out how to make the paths in relative to pwd

Xml files can be regenerated from scratch by:

  1. Use packagemaker.app on OSX 10.5 or 10.6 (possibly 10.7)
  2. running through a code formatting utility (it all comes out on one very long line)
  3. Replace you absolute paths from /your/path/to/trunk/osx/image/Xpra.app to image/Xpra.app

Script (make-pkg.sh) must be run from the osx directory after running make-app.sh

.pmdoc and make-pkg.sh are tested and committed to trunk.

comment:2 Changed 3 years ago by Scott Bennett

Owner: changed from Antoine Martin to Scott Bennett
Status: newassigned

comment:3 Changed 3 years ago by Antoine Martin

Looks to me like you've added those scripts and files in (was r 7336 - now removed)

We have to do something about the 01xpra-contents.xml: it cannot be kept in svn as it is, it weighs more than the whole xpra source tree combined:

$ du -sk osx/xpra.pmdoc/
4656	osx/xpra.pmdoc/
$ du -sk src/xpra/
3580	src/xpra/

We need some kind of script so we can generate it on the fly instead.

Last edited 3 years ago by Antoine Martin (previous) (diff)

comment:4 Changed 3 years ago by Scott Bennett

Good point. Let's remove from SVN and host on a web server in a tar.gz and link to ticket for now if anyone wants it. I'll keep the task open and work on a script to generate this file for the next release.

Last edited 3 years ago by Antoine Martin (previous) (diff)

comment:5 Changed 3 years ago by Scott Bennett

RE: (was in r 7337 - now removed)

  1. Removed large XML files and replaced with a single tar.bz2
  2. Added extract if-not-exists to make-pkg.sh

File size was reduced to 70K is this is still to big, let me know and I'll remove from SVN and post on web server with link.

Last edited 3 years ago by Antoine Martin (previous) (diff)

comment:6 Changed 3 years ago by Antoine Martin

There were two big (literally) problems which forced me to completely rebuild the svn repository to remove those last two commits:

  • we don't want the 4.5MB file in the repository history at all (makes checkout unnecessarily expensive in disk space and bandwidth, same for bisection, etc.)
  • even having a compressed (tar.bz2) version of it is not the answer, it only hides the real problem. Having every path in this file is not just horrible (thanks Apple!), it's also unmaintainable and would also cause unnecessary churn in the repo

This file is generated by some tools, we need the commands and scripts used so we can generate it on the fly.
Surely, a bit of sed or perl can do this? Otherwise, maybe we can just move to 10.6.x or 10.7.x as build platforms and use this tutorial Making OS X Installer Packages like a Pro - Xcode4 Developer ID Mountain Lion ready pkg. Having #533 finally closed would help.

Important: you will need to re-checkout the project since its history has changed. (safer than selectively pruning)

comment:7 Changed 2 years ago by Antoine Martin

Once that's done, it should be relatively straightforward to do: Submitting a Python App to the Mac App Store:

  • sandboxing: probably means we need to change the paths for config files and sockets? (PITA but not terribly hard) - see #888
  • code-signed: easy-ish (see #984)
  • deprecated apis: none that I know of? (but maybe some pyobjc?)
  • ppc code: none
  • large app icon: easy
  • app store category: easy

We need this to be able to do #407.

Last edited 20 months ago by Antoine Martin (previous) (diff)

comment:8 Changed 2 years ago by Antoine Martin

As part of #888, I will probably do a large part of what is required for App Sandboxing.
See also Migrating an App to a Sandbox
(FWIW: this signing + sandboxing doesn't seem very useful, attackers can sign their apps too!)
The main difficulty is going to be the unix domain sockets, which require special handling and permissions..

Some pointers:

So I don't see a way of keeping the sockets private without creating our own user directory as /var/tmp/$UID or even /var/tmp/$UID-Xpra..

Last edited 2 years ago by Antoine Martin (previous) (diff)

comment:9 Changed 22 months ago by Antoine Martin

Milestone: future0.16

scheduling for 0.16

comment:10 Changed 20 months ago by Antoine Martin

Priority: minorblocker

@sbennett: please attach the make-pkg.sh script.

Best reference I found:

Last edited 20 months ago by Antoine Martin (previous) (diff)

comment:11 Changed 16 months ago by Antoine Martin

Excellent link which seems to allow us to sign PKG files without needing any of apple's tools: Signing a Mac OS X package on Linux.

comment:12 Changed 15 months ago by Antoine Martin

Owner: changed from Scott Bennett to Antoine Martin
Status: assignednew

See also: Flat Package Format - The missing documentation


Assuming this gets resolved, we will then want to generate the pmdoc file rather than run a GUI tool every time. But this is Apple, they don't have a clue: their own packages are a complete mess, clearly generated by hand every time, with different spellings half the time, absolutely no consistency anywhere - that's why: you can't expect to automate something that they're not capable of automating themselves.
End of rant.


This ticket is a blocker for doing printer forwarding with osx shadow servers (#391)

Last edited 15 months ago by Antoine Martin (previous) (diff)

comment:13 Changed 15 months ago by Antoine Martin

Milestone: 0.160.17
Status: newassigned

comment:14 Changed 15 months ago by Antoine Martin

Found a new link: bomutils: bomutils are a set of tools to create Mac OS X installer packages on foreign OSes (windows, linux, etc.)..
Including a tutorial.

Other relevant links: How to build a Mac OS X Installer Package from these files? (the answer on flattening could be useful), Porting PackageMaker command line build installer to pkgbuild.

I can build a package using this command line (done on a 10.8 system after building the app on the usual 10.5 system):

/Developer/usr/bin/packagemaker \
    --title "Xpra" \
    --root ./image/ \
    --version 0.17 \
    --filter "\.DS_Store" \
    --root-volume-only \
    --domain system \
    --verbose \
    --no-relocate \
    --target 10.5 \
    --id org.xpra.pkg \
    --out Xpra.pkg

It takes a very very long time, even on a fast i7 with superfast SSD to match - whatever it is doing, it's not IO, just wasting CPU cycles? Interestingly, the "Activity Monitor" shows that "PackageMaker?" is the only non-64bit process running.. (hovering at 8% CPU usage)
The resulting pkg does absolutely nothing when you run it, it claims that 360MB of disk space will be used, shows a shiny progress bar - but does absolutely zilch.

Trying again with pkgbuild:

pkgbuild --root ./image \
    --version 0.17.0
    --install-location "/Applications" \
    --identifier "org.xpra.pkg" \
    Xpra.pkg

This one works fine, tested on 10.5 and 10.8.
Let's merge this, and try to get file association working too #407, and maybe also the cups backend (see ticket:391#comment:9).

comment:15 Changed 15 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo
Status: assignednew

r12373 will now spit out a PKG. I have uploaded one made with this script.

bomutils was the way to go: it can be automated easily enough via a simple shell script and it works on both OSX and Linux.

This needs to be added to the moduleset:

  • cpio (Apple's version is too old or just incompatible with the command line used in the script)
  • bomutils
  • xar >= 1.6.1 (if we want signing)

Now that we have everything in place: a "flat" pkg, a newer build of openssl (#876), we may be able to do the signing as per comment:11.

@smo: if this package works for you, please merge those new packages into the moduleset and try to do the signing - then I can pay the apple tax and get my packages signed too.

Last edited 15 months ago by Antoine Martin (previous) (diff)

comment:16 Changed 15 months ago by Antoine Martin

Related changes in r12376 for #391.

We should now be able to deal with #407 (session file association).

Also found some good information on "flat" packages: http://matthew-brett.github.io/docosx/flat_packages.html.

Last edited 15 months ago by Antoine Martin (previous) (diff)

comment:17 Changed 14 months ago by Smo

cpio added to moduleset r12530

comment:18 Changed 14 months ago by Smo

Having issues adding xar and I think we pull python-rencode from this same source.

I don't think there is much to do on these old systems other than update curl?

curl -k --continue-at - -L https://github.com/downloads/mackyle/xar/xar-1.6.1.tar.gz
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

comment:19 Changed 14 months ago by Antoine Martin

We already build ssl, so building our own version of curl should be fine.

comment:20 Changed 14 months ago by Smo

The problem with building our own version of curl is jhbuild uses curl to download stuff so we have an issue.

Maybe we can just host these 2 source packages ?

comment:21 Changed 14 months ago by Smo

Having another issue with curl this time getting openssl which we will need to build new curl. This is becoming increasingly harder on this old outdated machine to maintain.

curl -k --continue-at - -L http://www.openssl.org/source/openssl-1.0.2h.tar.gz - o /Users/user/gtk/source/pkgs/openssl-1.0.2h.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   316  100   316    0     0    888      0 --:--:-- --:--:-- --:--:--     0
curl: (35) Unknown SSL protocol error in connection to www.openssl.org:443
*** Error during phase checkout of openssl: ########## Error running curl -k --continue-at - -L http://www.openssl.org/source/openssl-1.0.2h.tar.gz -o /Users/user/gtk/source/pkgs/openssl-1.0.2h.tar.gz *** [80/115]

comment:22 Changed 14 months ago by Antoine Martin

Openssl can be downloaded via ftp:

$ curl --version
curl 7.16.4 (i386-apple-darwin9.0) libcurl/7.16.4 OpenSSL/0.9.7l zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps 
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz 
osx-2:~ osx$ curl -O ftp://ftp.openssl.org/source/openssl-1.0.2h.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5150k  100 5150k    0     0   614k      0  0:00:08  0:00:08 --:--:--  975k

comment:23 Changed 13 months ago by Smo

r12770 changes to ftp for openssl

r12771 adds xar

comment:24 Changed 11 months ago by Antoine Martin

I do have a developer account which should allow me to sign the packages, but Is it possible to have GPL software in the Mac App Store? could be a problem.

comment:25 Changed 11 months ago by Smo

Another ssl issue downloading Xar

curl -k --continue-at - -L https://github.com/downloads/mackyle/xar/xar-1.6.1.tar.gz -o /Users/build/gtk/source/pkgs/xar-1.6.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129    0   129    0     0    178      0 --:--:-- --:--:-- --:--:--   183
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) SSL peer handshake failed, the server most likely requires a client certificate to connect
*** Error during phase checkout of xar: ########## Error running curl -k --continue-at - -L https://github.com/downloads/mackyle/xar/xar-1.6.1.tar.gz -o /Users/build/gtk/source/pkgs/xar-1.6.1.tar.gz *** [1/1]

I can't seem to find any mirror

comment:26 Changed 11 months ago by Antoine Martin

Once you have openssl, you can build curl. Isn't that enough?
If needed, we can download / update the ca cert list to be able to download from github.

comment:27 Changed 10 months ago by Smo

r13386 adds curl

I can't seem to find the default location or how to set the default for the certificates so this is what i've done inside jhbuild shell to make certificates work

export SSL_CERT_FILE="$HOME/gtk/inst/etc/ssl/cacert.pem" 
curl -o ${SSL_CERT_FILE} -k https://curl.haxx.se/ca/cacert.pem

Afterwards I can download/build xar with jhbuild buildone xar from within jhbuild shell.

Last edited 10 months ago by Antoine Martin (previous) (diff)

comment:28 Changed 10 months ago by Smo

r13387 fixes checkout location for bomutils

I still have an issue with this one when I build with this error

make install DESTDIR=/Users/build/gtk/inst/_jhbuild/root-bomutils
install -d /Users/build/gtk/inst/_jhbuild/root-bomutils/usr/bin
install -d /Users/build/gtk/inst/_jhbuild/root-bomutils/usr/share/man/man1
install -m 0755 build/bin/mkbom build/bin/dumpbom build/bin/lsbom build/bin/ls4mkbom /Users/build/gtk/inst/_jhbuild/root-bomutils/usr/bin
install -m 0644 build/man/mkbom.1.gz build/man/dumpbom.1.gz build/man/lsbom.1.gz build/man/ls4mkbom.1.gz /Users/build/gtk/inst/_jhbuild/root-bomutils/usr/share/man/man1
*** Error during phase install of bomutils: Module failed to install into DESTDIR u'/Users/build/gtk/inst/_jhbuild/root-bomutils-broken' *** [1/1]

It appears to support DESTDIR and i've checked the makefile so i'm not sure why jhbuild doens't like this?

If I copy the binaries to their proper location ~/gtk/inst/bin I can build the pkg no problem

I'll have someone test the pkg to make sure all is well.

Now we seem to have 3 issues to fix with jhbuild openssl, curl and bomutils

comment:29 Changed 10 months ago by Smo

Owner: changed from Smo to Antoine Martin

comment:30 Changed 10 months ago by maxmylyn

Smo asked me to paste the error here, unfortunately it is extremely vague and un-helpful:

Aug 18 12:21:20 Max-Trashcan installd[243]: PackageKit: Install Failed: Error Domain=PKInstallErrorDomain Code=112 "An error occurred while running scripts from the package “Xpra-0.17.5-r13398.pkg”." UserInfo={NSFilePath=./postinstall, NSURL=file://localhost/Users/max/Downloads/Xpra-0.17.5-r13398.pkg#base.pkg, PKInstallPackageIdentifier=org.xpra.pkg, NSLocalizedDescription=An error occurred while running scripts from the package “Xpra-0.17.5-r13398.pkg”.} {
     NSFilePath = "./postinstall";
     NSLocalizedDescription = "An error occurred while running scripts from the package \U201cXpra-0.17.5-r13398.pkg\U201d.";
     NSURL = "file://localhost/Users/max/Downloads/Xpra-0.17.5-r13398.pkg#base.pkg";
     PKInstallPackageIdentifier = "org.xpra.pkg";
 }

Also, the machine on which I tested this is OSX 10.11.5.

comment:31 Changed 10 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

The mistake here is to think that the Apple mothership will let you do what you want, even if you have root on your device. Because "security" of course.

The postinstall error is caused by Apple's System Integrity Protector. See How to fix permission issues on Homebrew in OS X El Capitan? for details.

  • r13411 uses /usr/local/bin for our scripts instead of /usr/bin
  • r13412 removes launch agent support (#1169) until I can figure out a workaround for 10.11.x

Both should be backported. I can now install the PKG on 10.11.x again.

comment:32 Changed 10 months ago by Smo

Owner: changed from Smo to Antoine Martin

Got this lovely error updating openssl today

*** Configuring openssl *** [81/118]
./Configure --prefix=/Users/build/gtk/inst --openssldir=/Users/build/gtk/inst/etc/ssl shared  darwin-i386-cc && make install
Perl v5.10.0 required--this is only v5.8.8, stopped at ./Configure line 12.
*** Error during phase configure of openssl: ########## Error running ./Configure --prefix=/Users/build/gtk/inst --openssldir=/Users/build/gtk/inst/etc/ssl shared  darwin-i386-cc && make install *** [81/118]

Trying to close this ticket but seems now I need to build perl on this old system?

comment:33 Changed 10 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo

Yes, perl is needed on older build systems: ticket:840#comment:22

Last edited 9 months ago by Antoine Martin (previous) (diff)

comment:34 Changed 9 months ago by Smo

jhbuild meta-osx-xpra-pkgtools

This will build all that is necessary to make the pkg.

There is still an issue with bomutils so you must copy these.

Last edited 9 months ago by Smo (previous) (diff)

comment:35 Changed 9 months ago by Smo

Resolution: fixed
Status: newclosed

problem with bomutils has been fixed in r13828

Everything works as it should now.

comment:36 Changed 9 months ago by Antoine Martin

Follow up for signing and appstore: #1340.

Note: See TracTickets for help on using tickets.