xpra icon
Bug tracker and wiki

Opened 3 years ago

Last modified 9 days ago

#678 new task

building all the dependencies from source on win32

Reported by: Antoine Martin Owned by: Smo
Priority: blocker Milestone: 2.0
Component: platforms Version: trunk
Keywords: win32 Cc:

Attachments (11)

libyuv-mjpeg.patch (331 bytes) - added by Antoine Martin 3 months ago.
(incomplete) patch for compiling libyuv with mingw
platform-win32.diff (10.0 KB) - added by Antoine Martin 3 months ago.
temporary patching for build with mingw, without any pywin32 modules
PKGBUILD (3.7 KB) - added by Antoine Martin 3 months ago.
modified build file for python-gobject2 with the gio-types patch
PKGBUILD.2 (1.3 KB) - added by Antoine Martin 3 months ago.
libpng 1.6.27 build file
dsextras-gcc.patch (1.1 KB) - added by Antoine Martin 3 months ago.
patch for dsextras to support newer gcc versions
shell32.patch (2.8 KB) - added by Antoine Martin 2 months ago.
use cython to access shell functions
shell32.pyx (3.4 KB) - added by Antoine Martin 2 months ago.
failed attempt at implementing IPropertyStore access using a C++ cython module
websockify-0.8.0-win32.tar.xz (15.6 KB) - added by Antoine Martin 2 months ago.
modified websockify for win32
python-lzo-envpath-fix.patch (910 bytes) - added by Antoine Martin 2 months ago.
we still need to patch python-lzo for 1.11 (fixed in trunk)
libyuv-nojpeg.patch (1.0 KB) - added by Antoine Martin 2 months ago.
don't link jpeg with libyuv
libyuv.pc (184 bytes) - added by Antoine Martin 2 months ago.
32-bit pkgconfig file

Download all attachments as: .zip

Change History (61)

comment:1 Changed 3 years ago by Antoine Martin

Note: this is required for ticket:263#comment:12.

comment:2 Changed 2 years ago by Antoine Martin

Description: modified (diff)

(added new links found on the mailing list)

comment:4 Changed 20 months ago by Antoine Martin

Milestone: 0.16future

GTK3 is not looking good.

comment:5 Changed 17 months ago by Antoine Martin

Milestone: future0.17
Priority: majorcritical
Summary: building GTK3 and all the dependencies from source on win32building all the dependencies from source on win32

All GTK3 things will now be tracked in #640, we still want to do this for GTK2.

comment:6 Changed 16 months ago by Antoine Martin

A solution based on hexchat's https://github.com/codekiddy2/Visual-Studio-gtkmm/wiki.
Also: https://github.com/fanc999/gtk-msvc-projects - MSVC Projects to build GTK+ from Stock Installation of MSVC 2008-2013

See this thread: https://www.mail-archive.com/gtkmm-list@gnome.org/msg19515.html.

comment:7 Changed 12 months ago by Antoine Martin

Milestone: 0.170.18

comment:8 Changed 12 months ago by Antoine Martin

Milestone: 0.180.17

Becoming more urgent every day (missing out on fixes and new features added since 0.24.10! that's lots of them), ie: ticket:1131#comment:5

comment:9 Changed 12 months ago by Antoine Martin

The latest hexchat scripts can be found here: https://github.com/hexchat/gtk-win32, for GTK3 see ticket:640#comment:36

comment:10 Changed 11 months ago by Smo

Owner: changed from Smo to Antoine Martin

I've gone ahead with getting a machine ready to compile what we can with the hexchat scripts.

Not done yet but maybe we can get a headstart by using the precompiled binaries from hexchat for testing?

https://hexchat.readthedocs.org/en/latest/building.html

https://dl.hexchat.net/gtk-win32/vc14/x86/gtk-Win32.7z
https://dl.hexchat.net/gtk-win32/vc14/x64/gtk-x64.7z

comment:11 Changed 11 months ago by Antoine Martin

Milestone: 0.170.18
Owner: changed from Antoine Martin to Smo

Building GTK itself is documented here: https://github.com/hexchat/gtk-win32/blob/master/README.md.
It does include all the latest versions, which is great.
(re-scheduling)

comment:13 Changed 10 months ago by Antoine Martin

There are 3 tickets that should be consolidated into one: #917, #678 and #300

comment:14 Changed 9 months ago by Antoine Martin

We'll probably have to also make 64-bit builds in order to support newer CUDA and NVENC versions for shadowing (#558) as the latest sdks don't support 32-bit..

Preliminary steps:

  • hexchat build setup instructions, but install python in the more standard location (root of C drive) and choose Python 2.7 and 3.4 (3.5 is not supported yet by cx_freeze or the gi bindings)
  • install Microsoft Visual C++ Compiler for Python 2.7
  • install tortoisesvn for accessing the xpra source code (optional)
  • install pip for python 2.7
  • using pip, install most of the python wiki/Dependencies for both python 2.7 and 3.4 (the exact names to use may vary slightly in the pypi registry), some special cases:
    • py2exe 0.6.9 installed from exe for python 2.7, not needed with python 3.x
    • pywin32 installed from exe
  • problematic dependencies: lz4, lzo, pillow, cryptography
  • build x264, libyuv, vpx and ffmpeg and place them in XPRA_WIN32_BUILD_LIB_PREFIX

comment:15 Changed 9 months ago by Smo

From the hexchat build setup instructions I installed Visual Studio 2015 Community and didn't end up with what was needed for building.

Should install the 2nd one instead Visual C++ Build Tools 2015

It also installs less stuff and installs much faster.

comment:16 Changed 9 months ago by Smo

Also since it only uses msys2 for tools like patch etc.. it is not necessary to have the 64bit version

comment:17 Changed 9 months ago by Smo

Following instructions for hexchat I can now build both win32 and x64 builds of gtk and the other libraries needed for Hexchat.

Hexchat's build system does this by using vcproj files to facilitate building from the command line.

We will have to do the same for some of the libraries that you listed as problematic since we'd like to build these all with msvc as well.

comment:18 Changed 9 months ago by Antoine Martin

Priority: criticalblocker

Blocker for #1072 and #1230

Last edited 7 months ago by Antoine Martin (previous) (diff)

comment:19 Changed 9 months ago by Antoine Martin

Milestone: 0.181.0

Milestone renamed

comment:20 Changed 7 months ago by Antoine Martin

Milestone: 1.02.0

comment:21 Changed 6 months ago by Antoine Martin

We'll have to make sure everything runs obviously, so this should cover #628.

comment:22 Changed 3 months ago by Antoine Martin

Edited a brain fart: building the GTK bundles from source

Then I noticed that there are also some files that are supposed to build pygtk2 for win32: https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python2-pygtk
And jhbuild was used at least once for cross compiling: Cross Compiling Jhbuild.
And there's even a jhbuild for windows: http://afuera.me.uk/jhbuild-windows/

Another thing worth considering: do we want to continue to support 32-bit windows? Maybe just leave that for the 1.0.x branch and focus on 64-bit? (so we can use more modern things like AVX acceleration in openssl, NVENC, etc)

Last edited 3 months ago by Antoine Martin (previous) (diff)

comment:23 Changed 3 months ago by Antoine Martin

Owner: changed from Smo to Antoine Martin
Status: newassigned

Closed #300 and #917 as duplicates of this ticket.

I now have two machines with everything installed as per hexchat's building the GTK bundles from source. One 32-bit VM ("Windows 7 Pro x86") for regular builds and one 64-bit dedicated system ("Windows 7 Pro x64") with an Nvidia card for NVENC and hardware testing.

Note: I had to install "Visual Studio 2015 Community" and not "Visual C++ Build Tools 2015". The latter caused build failures, it seems that the directory layout does not match the expectations of the powershell build script.

Next we need to build a number of extra packages, some of those may be available via MSYS2 or from here: https://github.com/Alexpux/MINGW-packages/tree/master/, others may be found here: https://github.com/codekiddy2/Visual-Studio-gtkmm/wiki/Packages, OTOH:

  • gtkglext + pygtkgl
  • yasm / nasm
  • openssl
  • gmp / mpfr?
  • libjpeg-turbo?
  • gstreamer, its plugins and their dependencies - we can make progress without this first, and we may be able to get by using the binaries if needed - this may help: Build GStreamer on Windows: A Guide
  • x264 / ffmpeg / libvpx and their dependencies
  • python (probably easier to just call setuptools / pip): cython, rencode, netifaces, python-cryptography, pillow, numpy, lz4, lzo, pyopengl, pycuda, pynvml, gi, py2exe / cx_freeze
  • python win32: pywin32, comtypes, wmi
  • opencv?

How do we update this build script? fork it? (PITA) or maintain an overlay?
ie: we want a newer libpng.

The packaging scripts will also need to be modified to find (py)GTK in its new location.

Last edited 3 months ago by Antoine Martin (previous) (diff)

comment:24 Changed 3 months ago by Antoine Martin

For setting up a pure mingw development environment (here for 32-bit):

pacman -S base-devel msys2-devel mingw-w64-i686-toolchain

Since the packages exist in MSYS2, why can't we just use that? (and those are also built from source if needed - or we can tweak the PKGBUILD file).

Here's a list of versions found as of today here: https://github.com/Alexpux/MINGW-packages, looks like things get updated fairly regularly too:

  • pygtk2 2.24.0 and all its dependencies - this is already using the latest versions too:
    • gtk 2.24.31
    • glib 2.50.2
    • gdk-pixbuf 2.36.0
    • pygoject2 2.28.6
    • libjpeg-turbo 1.5.1
    • libpng 1.6.26 (needs updating)
  • winpthreads (useful for ffmpeg and others)
  • orc 0.4.26
  • yasm 1.2.0
  • x264 r2721.72d53ab
  • ffmpeg 3.2: will need tweaking to reduce the dependencies as per ticket:445#comment:4
  • opus 1.1.3, flac 1.3.1, wavpack 4.80.0, gmp 6.1.1, lame 3.99.5, libmad 0.15.1b, libmatroska 1.4.5: all present
  • gstreamer 1.8.3: we can trim the plugins during the packaging phase
  • gobject-introspection bindings 1.50.0
  • lz4 1.7.1
  • lzo2 2.09
  • xxhash 0.6.1
  • openssl 1.0.2j: no 1.1 yet, no biggie
  • python2 2.7.12: 2.7.13 should land soon I expect
  • python3 3.5.2

And it even includes the python packages:

  • numpy 1.11.1
  • pillow 3.3.0
  • pywin32 219
  • setuptools 28.6.0

extras we may want to use:

  • opencv 3.1.0
  • xpdf (tools only) 3.04
  • evince 3.22.1
  • ghostscript 9.20
  • switch to libssh / libssh2 instead of executing putty?

Good tip: MSVC and MinGW DLLs, or how we can generate lib / def files.

Last edited 3 months ago by Antoine Martin (previous) (diff)

comment:25 Changed 3 months ago by Antoine Martin

With the pure mingw approach:

We use it for:

  • win32con everywhere..
  • win32api.SetConsoleTitle
  • win32console in wait for input
  • win32api.EnumDisplayMonitors, win32api.MonitorFromWindow, win32api.GetMonitorInfo: gui screen detection
  • win32com.propsys (win32_propsys_set_group_leader): window hooks
  • win32api.GetSystemMetrics, gdi32.GetDeviceCaps, win32gui.GetDoubleClickTime: settings
  • win32api.GetWindowLong / win32gui.SetWindowLong: gui
  • win32api.ClipCursor: grabs
  • win32gui.FindWindow, win32api.SendMessage in show_desktop
  • win32api.MapVirtualKey, win32api.GetAsyncKeyState, win32api.GetKeyState, win32api.GetKeyboardLayoutList, win32api.GetKeyboardLayout, win32gui.SystemParametersInfo: keyboard
  • win32com.shell.SHGetFolderPath, win32api.RegOpenKey, win32api.RegQueryValueEx: paths, etc
  • win32print.EnumPrinters: printing
  • win32process.GetWindowThreadProcessId, win32gui.GetWindowText, win32gui.GetWindowRect, win32gui.EnumWindows, win32gui.GetDesktopWindow, win32gui.GetWindowDC, win32ui.CreateDCFromHandle, etc. for shadow
  • winxpgui and win32gui, win32gui.Shell_NotifyIcon, win32api.GetCursorPos: tray
  • win32ts.WTSRegisterSessionNotification / win32ts.WTSUnRegisterSessionNotification: events
  • comtypes webcam

It is a lot, but managing all the builds provided by MSYS2 ourselves is probably worse..
Most of these calls can be changed to ctypes relatively easily.
It may help to write unit tests for these functions to make sure we re-implement them correctly.

Easily installed with pacman:

  • yasm, lz4, lzo2, xxhash, openssl
  • python modules: python2-numpy, python2-pillow, cython2 (only at 0.24.1)

Then with setuptools, these python modules:

  • rencode, python-lz4, xxhash, cryptography, nvidia-ml-py, netifaces
  • comtypes, wmi (will these actually work without pywin32?)

Python modules that need fixing:

  • lzo (needs pointer to the C source code)
  • py2exe or cx_freeze
  • gtkglext + pygtkglext: the big ones, if that's too hard then we can look at GTK3 again (we have the GTK source so we can build cython extension if we need the speed)
  • pycuda (needs CUDA installed first)
Last edited 3 months ago by Antoine Martin (previous) (diff)

comment:26 Changed 3 months ago by Antoine Martin

Added:

  • subversion, cmake
  • python2-cx_Freeze
  • libvpx, x264, ffmpeg
  • gst-plugins-good, gst-plugins-bad, gst-plugins-ugly

Added using their own installer:

  • ghostscript
  • gsview

Added with easy_install:

  • PyOpenGL + PyOpenGL_accelerate

Needs sorting out:

  • uglifyjs + yuicompressor
  • libyuv (see patch and cmake . && make)
  • websockify (installed OK via easy_install, but needs our patches for win32)

With those changes and also some minor fixes to the build file: r14673, r14674, r14675, r14676
I can build xpra (as long as I replace win32con with a handmade copy) using:

python ./setup.py build_ext --inplace --without-enc_x265
python ./setup.py install_exe --install=./install-dir

The x265 plugin crashes hard, probably a bug in our code - we should either disable it or remove it altogether.

The resulting executables are almost usable, apart from the pywin32 bits missing.. (and pygtkglext)

Last edited 3 months ago by Antoine Martin (previous) (diff)

Changed 3 months ago by Antoine Martin

Attachment: libyuv-mjpeg.patch added

(incomplete) patch for compiling libyuv with mingw

comment:27 Changed 3 months ago by Antoine Martin

gtkglext and pygtkgl

Install:

  • gtk-doc
  • gobject-introspection
  • gtkglext

Download pygtkglext (the download links on the gnome gtkglext project site are dead):

git clone https://github.com/GNOME/pygtkglext
cd pygtkglext
#remove sanity check which would fail:
sed -i -e 's/if not pkgc.*/if False:/g' setup.py
#tell the build tools where to find codegen:
export PYTHONPATH=${MINGW_PREFIX}/share/pygobject/2.0/
#fix the pygtk dsextras class to support gcc 6:
wget "https://xpra.org/trac/raw-attachment/ticket/678/dsextras-gcc.patch"
patch -p1 -d ${MINGW_PREFIX} < dsextras-gcc.patch
#build pygtkglext:
python ./setup.py build
#install phase fails on "add_template_option", see #147 for details
#so do it by hand:
rsync -rplogtv build/lib.mingw-2.7/gtk/* ${MINGW_PREFIX}/lib/python2.7/site-packages/gtk-2.0/gtk/

With r14677 (detect mingw and use minor build tweaks) + r14678 (honour install directory for pyopengl modules), I get a working pygtkgl installation!

Last edited 3 months ago by Antoine Martin (previous) (diff)

Changed 3 months ago by Antoine Martin

Attachment: platform-win32.diff added

temporary patching for build with mingw, without any pywin32 modules

comment:28 Changed 3 months ago by Antoine Martin

Started the conversion from pywin32 to pure ctypes in r14683 (+fixup in r14684).
The build still runs as well as before on the old build system and we now get an almost usable client on the new mingw32 build system!

Still TODO:

  • cryptography packaging: complains about missing "appdirs".
  • gi not bundled, so gstreamer is not available - do we want to continue bundling python3 for sound? (could be more difficult now that python2 uses the same cx_freeze codepath?)
  • win32_events, keyboard info: lots of pywin32 to convert to ctypes
  • paths: registry access, SHGetFolderPath

etc

comment:29 Changed 3 months ago by Antoine Martin

A lot more done in r14686 + r14687.

Note done yet:

  • shadow server
  • named pipes
  • systray icon

Not done because more difficult as those need callbacks to enumerate:

  • GetKeyboardLayoutList
  • EnumDisplayMonitors
  • EnumPrinters

comment:30 Changed 3 months ago by Antoine Martin

More done in r14689 + r14690 + r14691 + r14692 + r14693.

Still TODO:

  • systray
  • named pipes
  • shadow server
  • win32 events listener

Changed 3 months ago by Antoine Martin

Attachment: PKGBUILD added

modified build file for python-gobject2 with the gio-types patch

Changed 3 months ago by Antoine Martin

Attachment: PKGBUILD.2 added

libpng 1.6.27 build file

comment:31 Changed 3 months ago by Antoine Martin

Looks like there's something missing from the PKGBUILD for (py)gtk.
I needed to do this to get pygtk to build:

mkdir /mingw32/include/gtk-2.0/gdk/win32/
cp ./mingw-w64-gtk2/src/gtk+-2.24.31/gdk/win32/gdkwin32keys.h \
      /mingw32/include/gtk-2.0/gdk/win32/

Then for python-gobject2, apart from the gio patch I also needed to skip the check phase: attachment/ticket/678/PKGBUILD.

We should also patch zlib seeing the recent CVEs that have been fixed in 1.2.10, but the large number of mingw patches make this non trivial.


Most of the missing bits for gstreamer packaging with python 2.7 (assuming we use python2 for sound):

rsync -rplogtv /mingw32/lib/python2.7/site-packages/gi install-dir/
rsync -rplogtv /mingw32/lib/girepository-1.0 install-dir/
rsync -rplogtv /mingw32/lib/gstreamer-1.0 install-dir/
Last edited 3 months ago by Antoine Martin (previous) (diff)

comment:32 Changed 3 months ago by Antoine Martin

Still needed to:

  • easy_install comtypes

Converted more code to ctypes: r14700 (cosmetic / for testing), r14701 (shadow server / named pipes), r14702 (don't use wmi)
Minor bugfixes: r14696, r14699 (win32 thread deadlock)

All remaining TODO (IIRC):

  • systray: should use a guid?
  • submit PKGBUILD updates upstream (zlib: maybe just wait for this one)
  • packaging: gstreamer 1.0 (with python 2 or 3?)
  • fix named pipes and shadow server bugs (may split into another ticket)
  • missing group leader API workaround (maybe use cython?)
  • 64-bit build system: document end-to-end setup, add CUDA and NVENC bits
Last edited 3 months ago by Antoine Martin (previous) (diff)

comment:33 Changed 3 months ago by Antoine Martin

Owner: changed from Antoine Martin to Smo
Status: assignednew

Some minor fixes were needed: r14713 (python3), r14714 (ignore mingw python3 generated files), r14715 (legacy win32 directory may not exist), r14716 (gcc on mingw compile fix).

Install "script":

#choose your target build arch (or do each one in turn):
#32-bit:
#export XPKG="mingw-w64-i686-"
#64-bit:
export XPKG="mingw-w64-x86_64-"
#most packages get installed here: (python, gtk, etc):
pacman --no-confirm -S ${XPKG}python2 ${XPKG}python2-pygtk ${XPKG}gtkglext
#media libraries (more than we actually need):
pacman --no-confirm -S ${XPKG}ffmpeg ${XPKG}gst-plugins-good ${XPKG}gst-plugins-bad ${XPKG}gst-plugins-ugly
#network layer libraries:
pacman --no-confirm -S ${XPKG}lz4 ${XPKG}lzo2 ${XPKG}xxhash
#python3 GStreamer bindings:
pacman --no-confirm -S ${XPKG}gst-python
#development tools and libs for building extra packages:
pacman --no-confirm -S base-devel ${XPKG}yasm ${XPKG}nasm subversion rsync gtk-doc ${XPKG}cmake ${XPKG}gcc ${XPKG}pkg-config ${XPKG}libffi
#python libraries and install and packaging tools:
pacman --no-confirm -S ${XPKG}python2-numpy ${XPKG}python2-pillow ${XPKG}cython2 ${XPKG}python2-setuptools ${XPKG}python2-cx_Freeze
#python3 versions (not all are really needed if just using python3 for sound):
pacman --no-confirm -S ${XPKG}python3-numpy ${XPKG}python3-pillow ${XPKG}cython ${XPKG}python3-cx_Freeze
#using easy-install for python libraries which are not packaged by mingw:
for x in rencode xxhash netifaces lz4 websocket-client comtypes PyOpenGL PyOpenGL_accelerate websockify cffi pycparser cryptography nvidia-ml-py; do
    easy-install-2.7 -U -Z $x
    easy-install-3.5 -U -Z $x
done

remaining issues (some important like pygtkglext, but not blockers):

  • cryptography failed to install unless its dependencies are installed separately first (cffi and pycparser)
  • pygtkglext: follow comment:27
  • libpng update: from source using attachment/ticket/678/PKGBUILD.2
  • pygobject rebuild: from source using attachment/ticket/678/PKGBUILD
  • install gsview and ghostscript separately (we could use ghostscript as provided by mingw, but gsview isn't.. now is a good time to deal with #1230)
  • websockify: still needs patching (not done at all - won't run out of the box..)
  • python3 builds need --with-server --with-shadow to help cx_Freeze, and the sound subapp doesn't work at all..

NVENC related installation issues (only available on 64-bit):

  • CUDA needs to be installed using its own EXE installer
  • NVENC SDK needs to be unzipped somewhere
  • pycuda doesn't find CUDA via easy-install

Build xpra into dist:

cd /e
svn co https://xpra.org/svn/Xpra
cd Xpra/trunk/src
python2 ./add_build_info.py
#x265 causes crashes!
BUILD_ARGS=--without-enc_x265
python2 ./setup.py build_ext --inplace ${BUILD_ARGS}
python2 ./setup.py install_exe --install=./dist
#for sound (still broken)
#python3 ./setup.py build_ext --inplace ${BUILD_ARGS}
#python3 ./setup.py install_exe --install=./dist/Sound --with-server --with-shadow

The resulting executables in dist should all run.

New minor TODO: the setup.py rebuilds all the cython extensions, even when not needed. (cython bug?)

@smo: please try this out, you will need a build system you can use as support for the old py2exe one will be removed soon.

Last edited 2 months ago by Antoine Martin (previous) (diff)

Changed 3 months ago by Antoine Martin

Attachment: dsextras-gcc.patch added

patch for dsextras to support newer gcc versions

Changed 2 months ago by Antoine Martin

Attachment: shell32.patch added

use cython to access shell functions

comment:34 Changed 2 months ago by Antoine Martin

For lack of a better place, some good links for debugging Cython / C++ / DLL issues on win32:

Turns out that the dependency on a Windows 8.1 Set that I was seeing on my Windows 7 build system (dependency on api-ms-win-core-libraryloader-l1-2-0.dll) was actually caused by the linker flag -lwindowsapp. Replacing it with -lole32 fixes the problem. Obviously. Not.

Changed 2 months ago by Antoine Martin

Attachment: shell32.pyx added

failed attempt at implementing IPropertyStore access using a C++ cython module

comment:35 Changed 2 months ago by Antoine Martin

The window grouping code (see #799) has (finally) been re-implemented using a C++ helper in r14795.
Notes:

  • I'm not sure it is even possible using a plain C cython module
  • using a C++ cython module is a pain, and I couldn't make it work
  • using a glue C++ file with a Cython module works
  • the windows types are a pain to use in cython, especially when so many functions make use of macros... which cython cannot use.

Documentation useful for writing native code:

Changed 2 months ago by Antoine Martin

modified websockify for win32

comment:36 Changed 2 months ago by Antoine Martin

  • r14800 silences console handler warnings during cleanup
  • r14801 packaging fix for newer versions of python-cryptography, backported in r14802
  • r14803 avoid unnecessary file copying during win32 build

New extra setup steps:

  • install python lzo, after applying this patch: attachment/ticket/678/python-lzo-envpath-fix.patch:
    pacman -S liblzo2-devel
    export LZO_DIR=$MINGW_PREFIX
    python2 ./setup.py install
    python3 ./setup.py install
    
  • websockify (should be upstreamed): untar these modified files attachment/ticket/678/websockify-0.8.0-win32.tar.xz over the 0.8.0 upstream source before running install
  • minify html5 client: easy_install-2.7 minify && easy_install-3.5 minify, install Java and add java.exe to your PATH before running the build
  • libyuv: filed an upstream ticket requesting info: build libyuv.dll on win32
  • webcam support, totally not obvious: the opencv package is missing some dependencies you have to install by hand:
    pacman -S ${XPKG}opencv ${XPKG}hdf5 ${XPKG}tesseract-ocr
    
  • ssh: we could also ship openssh's ssh.exe and sshpass.exe as an alternative to putty, it can be used with xpra attach --ssh=ssh.exe. The problem is that this would bring a large number of dependencies with it:
    cp /c/msys32/usr/bin/ssh.exe dist/
    cp /c/msys32/usr/bin/sshpass.exe dist/
    for x in ssh.exe sshpass.exe msys-gcc_s-1.dll msys-2.0.dll msys-ssp-0.dll msys-crypto-1.0.0.dll \
        msys-z.dll msys-gssapi-3.dll msys-heim* msys-krb5-26.dll msys-asn1-8.dll \
        msys-com_err-1.dll msys-roken-18.dll msys-roken-18.dll msys-crypt* \
        msys-wind-0.dll msys-hx509-5.dll msys-sqlite3*
        cp /c/msys32/usr/bin/$x dist/
    done
    

And also it doesn't have a GUI for prompting for username or password..
Maybe we can add this later as an option. And maybe this should be kept in a separate sub-directory.


Notes:

  • there is now a dedicated wiki page for setting up the win32 build env: wiki/Building/MSWindows
  • it would be nice to deal with #1231 so we can remove the opencv dependency
  • 64-builds are now tracked in #1413
  • "pacman -Syu" brings a nice set of updates: cython 0.25.2, libpng-1.6.28, gstreamer 1.10.2, gdk-pixbuf 2.36.3, flac 1.3.2, ffmpeg 3.2.2, lz4 1.7.5, numpy 1.11.2, etc
Last edited 2 months ago by Antoine Martin (previous) (diff)

Changed 2 months ago by Antoine Martin

we still need to patch python-lzo for 1.11 (fixed in trunk)

comment:37 Changed 2 months ago by Antoine Martin

r14808 adds a shell script (yes shell!) for generating the installer, just run this from a mingw shell:

./win32/PY27_MINGW_BUILD.sh

To get the EXE installer.

Still TODO:

  • remove some DLLs we should not be including "api-ms-win*"
  • trim the dlls (do we really need all the opencv bits?)
  • fix gdk pixbuf loader cache warning
  • replace MSI Wrapper with something better?
  • 64-bit tweaks (#1413)

comment:38 Changed 2 months ago by Antoine Martin

The dlls cannot be trimmed further, best to just move to a new webcam API: #1231.
r14811 removed the "api-ms-win-*dll" which came from Tortoise Plink, and fixes the pixbuf loader.
The latest windows beta upload was made using this mingw build, and it looks good! (still need to fix SIGINT hander?)

PS: just how awesome is mingw-w64? the latest update brings python-2.7.13 and zlib 1.2.11!

Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:39 Changed 2 months ago by Antoine Martin

Minor updates:

  • r14835: replace ctypes code with standard "winreg" module
  • r14832: fix shadow server screen size detection "dpi aware" (backport in r14834)
  • r14837 + r14838: directsound using ctypes

Note: the named pipes code is still broken, so you need to start shadow servers with "--bind=none" to workaround that.

Last edited 2 months ago by Antoine Martin (previous) (diff)

comment:40 Changed 2 months ago by Antoine Martin

libyuv

Here's how I figured out how to build the libyuv.dll:

git clone https://chromium.googlesource.com/libyuv/libyuv
cd libyuv
wget "https://www.xpra.org/trac/raw-attachment/ticket/678/libyuv-nojpeg.patch"
patch -p1 < ./libyuv-nojpeg.patch
cmake -G "MSYS Makefiles"
make
#build may fail on "convert.exe",
#if so then remove it from the Makefile,
#then run:

rm libyuv.a;make VERBOSE=1
#copy the long ar.exe command that generates "libyuv.a",
#and modify to build the DLL instead:
gcc -shared -o libyuv.dll \
    CMakeFiles/yuv.dir/source/compare.cc.obj \
    CMakeFiles/yuv.dir/source/compare_common.cc.obj \
    CMakeFiles/yuv.dir/source/compare_neon.cc.obj \
    CMakeFiles/yuv.dir/source/compare_neon64.cc.obj \
    CMakeFiles/yuv.dir/source/compare_gcc.cc.obj \
    CMakeFiles/yuv.dir/source/compare_win.cc.obj \
    CMakeFiles/yuv.dir/source/convert.cc.obj \
    CMakeFiles/yuv.dir/source/convert_argb.cc.obj \
    CMakeFiles/yuv.dir/source/convert_from.cc.obj \
    CMakeFiles/yuv.dir/source/convert_from_argb.cc.obj \
    CMakeFiles/yuv.dir/source/convert_jpeg.cc.obj \
    CMakeFiles/yuv.dir/source/convert_to_argb.cc.obj \
    CMakeFiles/yuv.dir/source/convert_to_i420.cc.obj \
    CMakeFiles/yuv.dir/source/cpu_id.cc.obj \
    CMakeFiles/yuv.dir/source/mjpeg_decoder.cc.obj \
    CMakeFiles/yuv.dir/source/mjpeg_validate.cc.obj \
    CMakeFiles/yuv.dir/source/planar_functions.cc.obj \
    CMakeFiles/yuv.dir/source/rotate.cc.obj \
    CMakeFiles/yuv.dir/source/rotate_any.cc.obj \
    CMakeFiles/yuv.dir/source/rotate_argb.cc.obj \
    CMakeFiles/yuv.dir/source/rotate_common.cc.obj \
    CMakeFiles/yuv.dir/source/rotate_mips.cc.obj \
    CMakeFiles/yuv.dir/source/rotate_msa.cc.obj \
    CMakeFiles/yuv.dir/source/rotate_neon.cc.obj \
    CMakeFiles/yuv.dir/source/rotate_neon64.cc.obj \
    CMakeFiles/yuv.dir/source/rotate_gcc.cc.obj \
    CMakeFiles/yuv.dir/source/rotate_win.cc.obj \
    CMakeFiles/yuv.dir/source/row_any.cc.obj \
    CMakeFiles/yuv.dir/source/row_common.cc.obj \
    CMakeFiles/yuv.dir/source/row_mips.cc.obj \
    CMakeFiles/yuv.dir/source/row_msa.cc.obj \
    CMakeFiles/yuv.dir/source/row_neon.cc.obj \
    CMakeFiles/yuv.dir/source/row_neon64.cc.obj \
    CMakeFiles/yuv.dir/source/row_gcc.cc.obj \
    CMakeFiles/yuv.dir/source/row_win.cc.obj \
    CMakeFiles/yuv.dir/source/scale.cc.obj \
    CMakeFiles/yuv.dir/source/scale_any.cc.obj \
    CMakeFiles/yuv.dir/source/scale_argb.cc.obj \
    CMakeFiles/yuv.dir/source/scale_common.cc.obj \
    CMakeFiles/yuv.dir/source/scale_mips.cc.obj \
    CMakeFiles/yuv.dir/source/scale_msa.cc.obj \
    CMakeFiles/yuv.dir/source/scale_neon.cc.obj \
    CMakeFiles/yuv.dir/source/scale_neon64.cc.obj \
    CMakeFiles/yuv.dir/source/scale_gcc.cc.obj \
    CMakeFiles/yuv.dir/source/scale_win.cc.obj \
    CMakeFiles/yuv.dir/source/video_common.cc.obj
#then we can install everything by hand:
#(a better way would be to work out how to tell cmake about our prefix)
rsync -rplogtv include/libyuv* ${MINGW_PREFIX}/include/
cp libyuv.a  ${MINGW_PREFIX}/lib/
cp libyuv.dll ${MINGW_PREFIX}/bin/
#add a pkgconfig file so the library can be found by build tools:
#(this .pc version is for mingw 32-bit):
wget "https://www.xpra.org/trac/raw-attachment/ticket/678/libyuv.pc"
mv libyuv.pc ${MINGW_PREFIX}/lib/pkgconfig/

For actually building our csc_libyuv module on 64-bit, you will also need r14853 .
Then you can verify that the libyuv codec has been built properly by running the "Encoding_info.exe" tool.

Last edited 2 months ago by Antoine Martin (previous) (diff)

Changed 2 months ago by Antoine Martin

Attachment: libyuv-nojpeg.patch added

don't link jpeg with libyuv

Changed 2 months ago by Antoine Martin

Attachment: libyuv.pc added

32-bit pkgconfig file

comment:41 Changed 8 weeks ago by Antoine Martin

Note: we may need to submit this patch to mingw for libvpx: gstreamer libvpx segfaults on Windows/x86 / upstream: libvpx crashes when encoding on x86_32.

comment:42 Changed 8 weeks ago by Smo

$ python2
Python 2.7.13 (default, Jan 17 2017, 13:56:44)  [GCC 6.3.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import gi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named gi

turns out I was missing this one

pacman -S mingw64/mingw-w64-x86_64-python2-gobject

comment:44 Changed 7 weeks ago by Antoine Martin

Note: the system update warns you about shortcuts going stale, I just created new ones by dragging "mingw32.exe" / "mingw64.exe" to the desktop.

comment:45 Changed 6 weeks ago by Antoine Martin

The systray code has been moved to its own ticket: #1434

comment:46 Changed 6 weeks ago by Antoine Martin

r15053 uninstalls xpra (if present) before installing a new version. This needs to be backported. Without this change, installing an older version over a 2.x installation breaks everything. (not sure why innosetup doesn't handle this properly)

comment:47 Changed 2 weeks ago by Antoine Martin

Interesting read: win32api handle incompatible with ctypes?.
Following at least some this advice, we now use WinDLL: r15244

Last edited 2 weeks ago by Antoine Martin (previous) (diff)

comment:48 Changed 11 days ago by Antoine Martin

Note: for keeping the python libraries up to date, just re-run this again as needed:

for x in rencode xxhash netifaces lz4 websocket-client comtypes PyOpenGL PyOpenGL_accelerate websockify cffi pycparser cryptography nvidia-ml-py; do
    easy_install-2.7 -U -Z $x
    easy_install-3.5 -U -Z $x
done

python-lz4 0.9.0 doesn't build on win32, not the first breakage in this release: #1464.

comment:49 Changed 10 days ago by Antoine Martin

r15307 moves the wiki instructions to shell scripts, including an update script for the python libraries, wiki updated: wiki/Building/MSWindows.

comment:50 Changed 9 days ago by Antoine Martin

Note: See TracTickets for help on using tickets.