xpra icon
Bug tracker and wiki

Opened 11 months ago

Closed 2 months ago

Last modified 2 months ago

#1340 closed task (fixed)

OSX PKG signing

Reported by: Antoine Martin Owned by: J. Max Mena
Priority: major Milestone: 2.0
Component: platforms Version: trunk
Keywords: osx signing pkg Cc:

Description (last modified by Antoine Martin)

Follow up form #641. (see also #840)

Links:

Once the certificate have magically appeared in Xcode and the keychain (..), we can sign the Xpra.app bundle, the PKG and DMG files:

codesign --deep --force --verify --verbose --sign "Developer ID Application: Antoine Martin (XXXXXXXXXX)" Xpra.app
codesign --deep --force --verify --verbose --sign "Developer ID Application: Antoine Martin (XXXXXXXXXX)" Xpra.dmg
productsign --sign "Developer ID Installer: Antoine Martin (XXXXXXXXXX)" Xpra.pkg Xpra-signed.pkg

The signature can be verified with:

codesign --verify --verbose=2 Xpra.app
spctl --assess --verbose=4 --type=execute Xpra.app
spctl --assess --verbose=4 --type=execute Xpra.dmg
spctl --assess --verbose=4 --type=install Xpra-signed.pkg

But this fails for signing "Xpra.app" if I keep the Xpra_NoDock.app nested inside the Xpra.app, it complains:

image/Xpra.app: invalid Info.plist (plist or signature have been modified)
In subcomponent: /Users/osx/Xpra-svn/trunk/osx/image/Xpra.app/Contents/Xpra_NoDock.app

Removing the nested app fixes this, but there must be something I can change in the Info.plist (or elsewhere?) to be able to bundle it.

@smo: what am I missing?

Attachments (6)

osx-non-relocatable.patch (1.1 KB) - added by Antoine Martin 11 months ago.
to make the package non relocatable
osx-make-appstore-pkg.sh (4.5 KB) - added by Antoine Martin 11 months ago.
work in progress to generate a PKG compatible with the appstore
osx-make-appstore-pkg-v2.sh (6.2 KB) - added by Antoine Martin 11 months ago.
updated patch
appstore.png (78.2 KB) - added by Antoine Martin 11 months ago.
appstore view
ticket1340_xpra_1-0-1-r14723-pkg-double-click-install-response_10-12-and-10-10-5.png (84.5 KB) - added by alas 8 months ago.
unknown developer error when installing .pkg
1340 unsigned 2.1.png (50.5 KB) - added by J. Max Mena 3 months ago.

Download all attachments as: .zip

Change History (35)

comment:1 Changed 11 months ago by Antoine Martin

Description: modified (diff)
Owner: changed from Antoine Martin to Smo

(edit: added verify for PKG - full error message)

comment:3 Changed 11 months ago by Antoine Martin

Owner: changed from Smo to Antoine Martin
Status: newassigned

Despite the hopelessly unhelpful and misleading error message from the codesign tool (..), I've narrowed it down through trial and error (mostly error):

  • removing all symlinks from the subapp passes
  • removing Contents/MacOS from the subapp passes
  • using the Info.plist from the main app passes

Bringing the Info.plist closer to the main one "fixes" things, done in r14152.

Along the way I found that changing CFBundleExecutable is not possible in the subapp!? It's fortunate that we don't use this executable directly.

Now onto app submission. Let's hope it is as much fun as this, not.

comment:4 Changed 11 months ago by Antoine Martin

r14154 + r14155 adds hooks to do the signing as part of the app / dmg / pkg scripts.

So until I move to a newer build host, the process is:

  • build the app (currently on a 10.5.8 VM):
    ./make-app.sh
    
  • copy the "Xpra.app" over to the machine doing the signing, ie:
    rsync -rplogt --delete "image/Xpra.app" osx@osx1010:/Users/osx/Xpra-svn/trunk/osx/image/
    
  • login to the signing VM and unlock the keychain:
    security unlock-keychain -p THEACCOUNTPASSWORD
    

(see User interaction is not allowed: trying to sign an OSX app using codesign)

  • run the remaining part of the build:
    cd Xpra-svn/trunk/osx
    export CODESIGN_KEYNAME="Antoine Martin"
    ./sign-app.sh && ./make-DMG.sh && ./make-PKG.sh 
    

Appstore fun:

  • follow steps in comment:2
  • install ApplicationLoader 3.0
  • watch in despair as the tool fails with yet another set of cryptic messages - for which there does not seem to be any reference or documentation anywhere, some have been fixed (ie: r14160 for "ITMS-90242"), most have not:
    • "ITMS-90251" : The product archive is invalid. Javascript may not be used in the installer.
    • "ITMS-90230" : Invalid product archive metadata. Error in keyPath [product-metadata.product-identifier]...
    • "ITMS-90230" : Invalid product archive metadata. Error in keyPath [product-metadata.product-version]...
    • "ITMS-90237" : The product archive package's signature is invalid. Ensure that it is signed with your "3rd Party Mac Developer Installer" certificate
    • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Path "Applications/Xpra?.app/Contents/Resources/lib/python2.7/numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap" in Payload is missing from Bom.
    • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Bundle at "Appplications/Xpra?.app" has identifier "Xpra" in PackageInfo? but "org.xpra.Xpra" in Payload
    • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Bundle at "Appplications/Xpra?.app" has version CFBundleVersion="1.3.0" in PackageInfo?, but CFBundleShortVersionString="1.0", CFBundleVersion="1.0" in Payload
    • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Bundle at "Appplications/Xpra?.app/Contents/Xpra_NoDock.app" in Payload not declared in PackageInfo?
    • "ITMS-90000" : Bad Bundle Executable. You must include a valid CFBundleExecutable key in your bundle's information property list file
    • "ITMS-90000" : Unsupported toolchain. Packages submitted to the App Store must be created either through Xcode, or using the productbuild tool..
Last edited 11 months ago by Antoine Martin (previous) (diff)

comment:5 Changed 11 months ago by Antoine Martin

More fixes:

What remains:

  • for javascript: About Distribution Definition Files: Important: For applications distributed on the App Store, use Xcode to make your installer package. If you need to specify system requirements, use the Pre-install Requirements Property List in the Xcode build settings. For more details, see the productbuild man page. - not sure how to fix this one
  • product-metadata issues: Distribution XML Reference: product - sadly, adding <product id="org.xpra.Xpra" version="$VERSION" /> to the Distribution file makes things worse, Application Loader then claims not to find the file you click on: The file \U201cXpra-1-1.0-r14164\U201d couldn\U2019t be opened because there is no such file (this error is just garbage)
  • CFBundleExecutable could be caused by the lack of signature on "Xpra_Launcher"?
    $ ls -al@ image/Xpra.app/Contents/MacOS/
    total 24
    drwxr-xr-x   5 osx  staff   170 14 Oct 20:00 .
    drwxr-xr-x  10 osx  staff   340 14 Oct 20:03 ..
    -rwxr-xr-x@  1 osx  staff  2752 10 Oct 21:44 PythonExecWrapper
    	com.apple.cs.CodeDirectory	 141 
    	com.apple.cs.CodeRequirements	 180 
    	com.apple.cs.CodeSignature	8527 
    -rwxr-xr-x@  1 osx  staff   399 10 Oct 21:44 Xpra
    	com.apple.cs.CodeDirectory	 128 
    	com.apple.cs.CodeRequirements	 164 
    	com.apple.cs.CodeSignature	8526 
    -rwxr-xr-x   1 osx  staff   292 14 Oct 20:00 Xpra_Launcher
    

See also:

Last edited 11 months ago by Antoine Martin (previous) (diff)

comment:6 Changed 11 months ago by Antoine Martin

Alternatively using productbuild:

productbuild --component ./image/Xpra.app \
    /Applications \
    ./image/Xpra-1.0.pkg  \
    --sign "3rd Party Mac Developer Installer"

The resulting installer package lacks any icons or text but seems to install OK.
Only one error remains for "Application Loader" upload for this one: ITMS-90259: Bad Bundle Executable...

Or even:

pkgbuild --analyze --root ./image/Xpra.app ./image/XpraComponents.plist
#edit XpraComponents.plist and set {{{BundleIsRelocatable}}} to false
pkgbuild --root ./image/Xpra.app \
    --component-plist ./image/XpraComponents.plist \
    --install-location "/Applications" \
    ./Xpra.pkg
#generate the Distribution.xml file:
productBuild --synthesize --package ./Xpra.pkg ./Distribution.xml
# modify it using the same data as found in our make-PKG.sh script, and:
# change "#base.pkg" to "#Xpra.pkg"
# add the product id and version line
productbuild --distribution ./Distribution.xml --package-path . \
    --sign "3rd Party Mac Developer Installer" \
    ./Xpra-Installer.pkg

Same "bad bundle executable" error, and also missing the 512x512 icons...


More links:

Last edited 11 months ago by Antoine Martin (previous) (diff)

comment:7 Changed 11 months ago by Antoine Martin

I can get a little bit further by:

  • replacing the "MacOS" shell scripts with a python binary (temporarily) - scripts should work though, as long as they are more than 28 characters long (this is not a joke: http://superuser.com/a/463673/58851)
  • deleting the "no dock" sub-app (temporarily)
  • deleting the "Helpers" (temporarily)
  • signing all "dylib", "so", "MacOS/*" and "Resources/bin" files using:
    codesign --force --verbose --sign "3rd Party Mac Developer Application" --entitlements ./Xpra.entitlements  THEFILE
    

The entitlements file was added in r14166. (it's not clear if the same entitlements should be used for all the files: I doubt the system is capable of isolating the code running in libraries from its entrypoint)

  • signing the "image/Xpra.app" after that

Then you need to make an upload to itunes... which takes a lot of time and effort. And then you get some more errors via email! (this is some kind of bad joke) Invalid Signature: (..) satisfies its Designated Requirement test-requirement: code failed to satisfy specified code requirement(s) - could they have worded it more ambiguously perhaps?
Before I started signing the "so" / "dylib" / etc, the error was different for those: code object is not signed at all In architecture: i386).
Rinse and repeat until you find the right combination.
But you can't delete versions on there, so you have to bump the CFBundleVersion every time. How convenient.


More links:

Changed 11 months ago by Antoine Martin

Attachment: osx-non-relocatable.patch added

to make the package non relocatable

Changed 11 months ago by Antoine Martin

Attachment: osx-make-appstore-pkg.sh added

work in progress to generate a PKG compatible with the appstore

comment:8 Changed 11 months ago by Antoine Martin

A big part of the difficulty is that it's non-trivial to embed shell scripts... Because shiny. Or something. (and also because it takes way too long to test anything)

With the patch attached, the codesign step shows:

(..)
image/Xpra.app/Contents/Helpers/Bug_Report: replacing existing signature
image/Xpra.app/Contents/Helpers/Bug_Report: signed generic [Bug_Report]
(..)

Note that is uses "generic" and not "signed Mach-O thin (i386)" like it does for all the binaries. And then much much later (..), you get an email (..) saying the build cannot be uploaded:

org.xpra.Xpra.pkg/Payload/Xpra.app: code object is not signed at all
In subcomponent: org.xpra.Xpra.pkg/Payload/Xpra.app/Contents/Helpers/Bug_Report

See:

Maybe we can change the wrappers so that the "MacOS" binary executes the real "Python" executable (renamed if needed) with the script as argument instead. (and move the scripts somewhere else..)

Still TODO:

  • fix sound subapp
  • postinstall script with symlinks to /usr/bin
  • agent

etc.

Last edited 11 months ago by Antoine Martin (previous) (diff)

comment:9 Changed 11 months ago by Antoine Martin

A gem in http://prod.lists.apple.com/archives/xcode-users/2013/Oct/msg00067.html:
In addition to my entitlements file I needed to create an info.plist file and embed it into the binary by adding the following linker flag:

-sectcreate __TEXT __info_plist SandboxUnixCommand/SandboxUnixCommand-Info.plist

Well, obviously... ahem. Magic TEXT section values!
More info here: Gimmedebugah: how to embedded a Info.plist into arbitrary binaries

Last edited 11 months ago by Antoine Martin (previous) (diff)

Changed 11 months ago by Antoine Martin

Attachment: osx-make-appstore-pkg-v2.sh added

updated patch

comment:10 Changed 11 months ago by Antoine Martin

Lots more nasty tricks to beat this mess into shape:

  • r14167 + r14174: dynamically populate version / revision / buildno during build
  • r14168: fix typo
  • r14169 + r14171 + r14172 + r14173 + r14176 + r14180: move things aroung to make codesign happy (some of these may not be mandatory - trial and error..)
  • r14170: we don't use the webcam yet, remove the entitlement
  • r14175: we need this weird entitlement!?
  • r14177: fix code stripping (backport in r14182)
  • r14178: fix "Bug Report" launch script (backport in r14181)
  • r14183: appstore packaging script

comment:11 Changed 11 months ago by Antoine Martin

At last a package that both runs and uploads to the appstore:

Changed 11 months ago by Antoine Martin

Attachment: appstore.png added

appstore view

comment:12 Changed 11 months ago by Antoine Martin

as shown on the https://itunesconnect.apple.com/ appstore control page:
appstore view

Last edited 11 months ago by Antoine Martin (previous) (diff)

comment:13 Changed 11 months ago by Antoine Martin

Sound sub-app fixes:

  • r14223: move binaries to "/Resources/bin"
  • r14224 + r14226: run the "sandbox inherit" version of the sound binary
  • r14225, r14222: minor related tweaks

comment:14 Changed 11 months ago by Antoine Martin

Some path fixes and updates:

  • r14266 (backport in r14267)
  • r14268
  • r14269: generate an SSL cert (but postinstall script is still not used in appstore version..)

Ideally, we would store our preferences in /Library/Preferences, but:

  • there is no API to get the path to this directory (sigh), see NSSearchPathDirectory
  • we are only supposed to use XML plist files in there? (sigh)

Then, we could handle Migrating an App to a Sandbox and have the preferences moved for us to the user sandbox.

Last edited 7 months ago by Antoine Martin (previous) (diff)

comment:15 Changed 11 months ago by Antoine Martin

Milestone: 1.02.0

I don't think this is going to happen anytime soon. Re-scheduling.

Their complaints after reviewing the app:

  • load session file dialog can trigger errors if the folders are not accessible due to sandboxing (ie: the home folder is not), not much we can do here since the file selection dialog is GTK's - we could just remove the load and save options for apple users (simplicity at its best), or use a different file selection tool (native somehow?), or remove the sanbox restrictions? (yuk)
  • name of the app doesn't match ("Launcher" vs "Xpra") and "Quit" is "Disconnect Launcher" - no biggie, this could be changed easily enough
  • we require sandboxing exceptions:
    • com.apple.coreservices.launchservicesd is needed to for pyobjc "coreservices" - not much we can do about this - maybe try to silence / ignore the errors this causes, or report the pyobjc issue upstream
    • com.apple.systemuiserver.screencapture is needed for using xpra as a server - they unhelpfully suggested: We encourage you to investigate other ways of implementing the desired functionality.
  • we use CTFontCopyDefaultCascadeList which is a non-public API, see pango coretext: use public function to obtain cascade list if available - looks like rebuilding on 10.8 or newer should fix this: #840

comment:16 Changed 10 months ago by Antoine Martin

Milestone: 2.01.0
Summary: OSX PKG signing - appstoreOSX PKG signing

Moving the appstore packaging to a new ticket: #1366

This purpose of this ticket is now only about signing the PKG.
At least some of the latest PKG builds should be signed... I should make it easier to know which PKG are signed and which ones are not. (requires using a newer VM to sign.. which won't boot right now on my systems.. even with the Intel one..)

I'm not entirely sure what the actual difference is for users when installing signed vs unsigned packages. I've paid the fee for the privilege of having Apple sign the key. Does this make it easier to install the PKG on OSX versions with the more stringent security settings?

comment:17 Changed 10 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas
Status: assignednew

Success. I have fixed my signing 10.10.x VM (it needs vbox 5.0 to boot, fails with 5.1 - fixes for running 5.0 on Fedora 25: Error: nothing provides libvpx.so.3()(64bit) on Fedora 25 and Runtime/r0drv/linux: 4.8 mod_timer_pinned fix).

Uploaded both DMG and PKG r14467 builds signed with my developer key.
@afarr: do these signed builds help with installation on newer versions of macosx? (you may need to turn the "stricter installation checks" setting back on again)

comment:18 Changed 10 months ago by alas

Owner: changed from alas to Antoine Martin

Odd.

I have an r14467 .dmg installed in my usual odd place - deleting and re-installing into the more usual /Applications directory works as usual, and likewise with the r14492 .dmg.

Double clicking on them, however, they fail the "Verifying 'Xpra'" validation test, "because it is from an unidentified developer".

I can't find any sign of the r14467 pkg though, or .dmg, in either the /dists or the /beta directories (so, where did I get the r14467 .dmg I already have?).

Trying to install the r14492 .pkg I get the usual security/unsigned (unknown developer) issue, but I'm not sure you signed that one.

I think you'll need to post another signed .pkg or even .dmg for me to check.

comment:19 Changed 10 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas

1.0 should be signed, just released today.
(the other ones disappeared because I needed to free some diskspace)


Note to self since this is not obvious, this is how we make the signed DMG / PKG:

  • run ./make-app.sh on the build VM
  • copy the Xpra.app to the signing VM
  • unlock the keys on the singing VM: security unlock-keychain -p THEACCOUNTPASSWORD
  • run ./make-DMG.sh && ./make-PKG.sh on the signing VM
Last edited 10 months ago by Antoine Martin (previous) (diff)

comment:20 Changed 10 months ago by alas

Owner: changed from alas to Antoine Martin

Checking in xpra.org/beta/osx ... the 1.0 r14492 pkg (http://xpra.org/beta/osx/Xpra-1.0-r14492.pkg) as well as the xpra.pkg (http://xpra.org/beta/osx/Xpra.pkg) both, once downloaded, trigger a security warning when trying to install (“Xpra.pkg” can’t be opened because it is from an unidentified developer.).

They indicate that they were posted/updated 11/27.

Handing this back to you... and chuckling that my username seems to have become 'alas' (I feel three breaths closer to Elizabethan-dom ... three breathless breaths, whatever that means).

Last edited 9 months ago by Antoine Martin (previous) (diff)

comment:21 Changed 9 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas

1.0 was released, meaning it is no longer in the beta area! Try here instead: http://xpra.org/dists/osx/x86/.
(and the files in the beta area will soon disappear altogether to make way for 2.0 builds)

As for 'alas', that's your "name" in your account settings. The trac update now shows the name rather username. You should be able to change it yourself from your account profile - but it does have a nice ring to it, doesn't it?

Last edited 9 months ago by Antoine Martin (previous) (diff)

comment:22 Changed 9 months ago by Antoine Martin

1.0.1 is also signed.

Changed 8 months ago by alas

unknown developer error when installing .pkg

comment:23 Changed 8 months ago by alas

Err — alas... I seem to be triggering the same "unknown developer" error with the new 1.0.1 (Xpra-1.0.1-r14723.pkg) that I was mentioning getting with 1.0 in comment:20 (which I also repeated with Xpra-1.0-r14502).

I got the errors with the new 10.12.1 super-shiny mac OS, and then confirmed I'm also getting it with a 10.10.5 significantly-less-shiny mac.

Same message with 1.0 and 1.0.1, on 10.12.1 & 10.10.5.

unknown developer error when installing .pkg

Apple must have more hoops for you?

comment:24 Changed 8 months ago by alas

Owner: changed from alas to Antoine Martin

comment:25 Changed 6 months ago by Antoine Martin

Milestone: 1.02.0
Owner: changed from Antoine Martin to alas

Does it work any better with the beta 2.0 builds?
Those are built and signed on the same system.

comment:26 Changed 3 months ago by J. Max Mena

Owner: changed from alas to Antoine Martin

Sadly, the latest 2.1 betas are still complaining about unidentified developers.

Changed 3 months ago by J. Max Mena

Attachment: 1340 unsigned 2.1.png added

comment:27 Changed 3 months ago by Antoine Martin

Owner: changed from Antoine Martin to alas

The certificate expired a few months back so I stopped signing the builds, you may want to try a build made around the time of comment:25 instead. (2.0.x)

comment:28 Changed 3 months ago by Antoine Martin

Owner: changed from alas to J. Max Mena

comment:29 Changed 2 months ago by J. Max Mena

Resolution: fixed
Status: newclosed

Retried with a build from mid March, and still got the error, and that was the oldest 2.0 build available in the betas folder (dists had nothing relevant unless I missed something that obvious..which is possible).

Either way, based on the conversations we've had, I'm going to close this - if we want to revisit OSX signing, we can re-open this, but in the meantime, closing.

(As an aside, I think we need a "crickets" resolved option, but that's just me)

Last edited 2 months ago by J. Max Mena (previous) (diff)
Note: See TracTickets for help on using tickets.