Split from #90 (see also #640). Similar to #533 for OSX.
Note: cross compiling is not an option because of the Cython extensions which seem to confuse mingw..
Some links:
Note: this is required for ticket:263#comment:12.
(added new links found on the mailing list)
GTK+3 build environment for Win32/Win64 - temporary fork for upstreaming
GTK3 is not looking good.
All GTK3 things will now be tracked in #640, we still want to do this for GTK2.
A solution based on hexchat's https://github.com/codekiddy2/Visual-Studio-gtkmm/wiki. Also: https://github.com/fanc999/gtk-msvc-projects - MSVC Projects to build GTK+ from Stock Installation of MSVC 2008-2013
See this thread: https://www.mail-archive.com/gtkmm-list@gnome.org/msg19515.html.
Becoming more urgent every day (missing out on fixes and new features added since 0.24.10! that's lots of them), ie: ticket:1131#comment:5
The latest hexchat scripts can be found here: https://github.com/hexchat/gtk-win32, for GTK3 see ticket:640#comment:36
I've gone ahead with getting a machine ready to compile what we can with the hexchat scripts.
Not done yet but maybe we can get a headstart by using the precompiled binaries from hexchat for testing?
https://hexchat.readthedocs.org/en/latest/building.html
https://dl.hexchat.net/gtk-win32/vc14/x86/gtk-Win32.7z https://dl.hexchat.net/gtk-win32/vc14/x64/gtk-x64.7z
Building GTK itself is documented here: https://github.com/hexchat/gtk-win32/blob/master/README.md. It does include all the latest versions, which is great. (re-scheduling)
Build GStreamer on Windows: A Guide
There are 3 tickets that should be consolidated into one: #917, #678 and #300
We'll probably have to also make 64-bit builds in order to support newer CUDA and NVENC versions for shadowing (#558) as the latest sdks don't support 32-bit..
Preliminary steps:
XPRA_WIN32_BUILD_LIB_PREFIX
From the hexchat build setup instructions I installed Visual Studio 2015 Community and didn't end up with what was needed for building.
Should install the 2nd one instead Visual C++ Build Tools 2015
It also installs less stuff and installs much faster.
Also since it only uses msys2 for tools like patch etc.. it is not necessary to have the 64bit version
Following instructions for hexchat I can now build both win32 and x64 builds of gtk and the other libraries needed for Hexchat.
Hexchat's build system does this by using vcproj files to facilitate building from the command line.
We will have to do the same for some of the libraries that you listed as problematic since we'd like to build these all with msvc as well.
Milestone renamed
We'll have to make sure everything runs obviously, so this should cover #628.
Edited a brain fart: building the GTK bundles from source
Then I noticed that there are also some files that are supposed to build pygtk2 for win32: https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python2-pygtk And jhbuild was used at least once for cross compiling: Cross Compiling Jhbuild. And there's even a jhbuild for windows: http://afuera.me.uk/jhbuild-windows/
Another thing worth considering: do we want to continue to support 32-bit windows? Maybe just leave that for the 1.0.x branch and focus on 64-bit? (so we can use more modern things like AVX acceleration in openssl, NVENC, etc)
Closed #300 and #917 as duplicates of this ticket.
I now have two machines with everything installed as per hexchat's building the GTK bundles from source. One 32-bit VM ("Windows 7 Pro x86") for regular builds and one 64-bit dedicated system ("Windows 7 Pro x64") with an Nvidia card for NVENC and hardware testing.
Note: I had to install "Visual Studio 2015 Community" and not "Visual C++ Build Tools 2015". The latter caused build failures, it seems that the directory layout does not match the expectations of the powershell build script.
Next we need to build a number of extra packages, some of those may be available via MSYS2 or from here: https://github.com/codekiddy2/Visual-Studio-gtkmm/wiki/Packages, OTOH:
How do we update this build script? fork it? (PITA) or maintain an overlay? ie: we want a newer libpng.
The packaging scripts will also need to be modified to find (py)GTK in its new location.
For setting up a pure mingw development environment (here for 32-bit):
pacman -S base-devel msys2-devel mingw-w64-i686-toolchain
Since the packages exist in MSYS2, why can't we just use that? (and those are also built from source if needed - or we can tweak the PKGBUILD file).
Here's a list of versions found as of today here: https://github.com/Alexpux/MINGW-packages, looks like things get updated fairly regularly too:
r2721.72d53ab
And it even includes the python packages:
extras we may want to use:
Good tip: MSVC and MinGW DLLs, or how we can generate lib / def files.
With the pure mingw approach:
We use it for:
win32con
everywhere..
win32api.SetConsoleTitle
win32console
in wait for input
win32api.EnumDisplayMonitors
, win32api.MonitorFromWindow
, win32api.GetMonitorInfo
: gui screen detection
win32com.propsys
(win32_propsys_set_group_leader): window hooks
win32api.GetSystemMetrics
, gdi32.GetDeviceCaps
, win32gui.GetDoubleClickTime
: settings
win32api.GetWindowLong
/ win32gui.SetWindowLong
: gui
win32api.ClipCursor
: grabs
win32gui.FindWindow
, win32api.SendMessage
in show_desktop
win32api.MapVirtualKey
, win32api.GetAsyncKeyState
, win32api.GetKeyState
, win32api.GetKeyboardLayoutList
, win32api.GetKeyboardLayout
, win32gui.SystemParametersInfo
: keyboard
win32com.shell.SHGetFolderPath
, win32api.RegOpenKey
, win32api.RegQueryValueEx
: paths, etc
win32print.EnumPrinters
: printing
win32process.GetWindowThreadProcessId
, win32gui.GetWindowText
, win32gui.GetWindowRect
, win32gui.EnumWindows
, win32gui.GetDesktopWindow
, win32gui.GetWindowDC
, win32ui.CreateDCFromHandle
, etc. for shadow
winxpgui
and win32gui
, win32gui.Shell_NotifyIcon
, win32api.GetCursorPos
: tray
win32ts.WTSRegisterSessionNotification
/ win32ts.WTSUnRegisterSessionNotification
: events
comtypes
webcam
It is a lot, but managing all the builds provided by MSYS2 ourselves is probably worse.. Most of these calls can be changed to ctypes relatively easily. It may help to write unit tests for these functions to make sure we re-implement them correctly.
Easily installed with pacman:
Then with setuptools, these python modules:
Python modules that need fixing:
Added:
Added using their own installer:
Added with easy_install:
PyOpenGL
+ PyOpenGL_accelerate
Needs sorting out:
cmake . && make
)
With those changes and also some minor fixes to the build file: r14673, r14674, r14675, r14676 I can build xpra (as long as I replace win32con with a handmade copy) using:
python ./setup.py build_ext --inplace --without-enc_x265 python ./setup.py install_exe --install=./install-dir
The x265 plugin crashes hard, probably a bug in our code - we should either disable it or remove it altogether.
The resulting executables are almost usable, apart from the pywin32 bits missing.. (and pygtkglext)
(incomplete) patch for compiling libyuv with mingw
Install:
Download pygtkglext (the download links on the gnome gtkglext project site are dead):
git clone https://github.com/GNOME/pygtkglext cd pygtkglext #remove sanity check which would fail: sed -i -e 's/if not pkgc.*/if False:/g' setup.py #tell the build tools where to find codegen: export PYTHONPATH=${MINGW_PREFIX}/share/pygobject/2.0/ #fix the pygtk dsextras class to support gcc 6: wget "https://xpra.org/trac/raw-attachment/ticket/678/dsextras-gcc.patch" patch -p1 -d ${MINGW_PREFIX} < dsextras-gcc.patch #build pygtkglext: python ./setup.py build #install phase fails on "add_template_option", see #147 for details #so do it by hand: rsync -rplogtv build/lib.mingw-2.7/gtk/* ${MINGW_PREFIX}/lib/python2.7/site-packages/gtk-2.0/gtk/
With r14677 (detect mingw and use minor build tweaks) + r14678 (honour install directory for pyopengl modules), I get a working pygtkgl installation!
temporary patching for build with mingw, without any pywin32 modules
Started the conversion from pywin32 to pure ctypes in r14683 (+fixup in r14684). The build still runs as well as before on the old build system and we now get an almost usable client on the new mingw32 build system!
Still TODO:
SHGetFolderPath
etc
A lot more done in r14686 + r14687.
Note done yet:
Not done because more difficult as those need callbacks to enumerate:
GetKeyboardLayoutList
EnumDisplayMonitors
EnumPrinters
More done in r14689 + r14690 + r14691 + r14692 + r14693.
Still TODO:
modified build file for python-gobject2 with the gio-types patch
libpng 1.6.27 build file
Looks like there's something missing from the PKGBUILD for (py)gtk. I needed to do this to get pygtk to build:
mkdir /mingw32/include/gtk-2.0/gdk/win32/ cp ./mingw-w64-gtk2/src/gtk+-2.24.31/gdk/win32/gdkwin32keys.h \ /mingw32/include/gtk-2.0/gdk/win32/
Then for python-gobject2, apart from the gio patch I also needed to skip the check phase: attachment/ticket/678/PKGBUILD.
We should also patch zlib seeing the recent CVEs that have been fixed in 1.2.10, but the large number of mingw patches make this non trivial.
Most of the missing bits for gstreamer packaging with python 2.7 (assuming we use python2 for sound):
rsync -rplogtv /mingw32/lib/python2.7/site-packages/gi install-dir/ rsync -rplogtv /mingw32/lib/girepository-1.0 install-dir/ rsync -rplogtv /mingw32/lib/gstreamer-1.0 install-dir/
Still needed to:
Converted more code to ctypes: r14700 (cosmetic / for testing), r14701 (shadow server / named pipes), r14702 (don't use wmi) Minor bugfixes: r14696, r14699 (win32 thread deadlock)
All remaining TODO (IIRC):
Some minor fixes were needed: r14713 (python3), r14714 (ignore mingw python3 generated files), r14715 (legacy win32 directory may not exist), r14716 (gcc on mingw compile fix).
Install "script":
#choose your target build arch (or do each one in turn): #32-bit: #export XPKG="mingw-w64-i686-" #64-bit: export XPKG="mingw-w64-x86_64-" #most packages get installed here: (python, gtk, etc): pacman --no-confirm -S ${XPKG}python2 ${XPKG}python2-pygtk ${XPKG}gtkglext #media libraries (more than we actually need): pacman --no-confirm -S ${XPKG}ffmpeg ${XPKG}gst-plugins-good ${XPKG}gst-plugins-bad ${XPKG}gst-plugins-ugly #network layer libraries: pacman --no-confirm -S ${XPKG}lz4 ${XPKG}lzo2 ${XPKG}xxhash #python3 GStreamer bindings: pacman --no-confirm -S ${XPKG}gst-python #development tools and libs for building extra packages: pacman --no-confirm -S base-devel ${XPKG}yasm ${XPKG}nasm subversion rsync gtk-doc ${XPKG}cmake ${XPKG}gcc ${XPKG}pkg-config ${XPKG}libffi #python libraries and install and packaging tools: pacman --no-confirm -S ${XPKG}python2-numpy ${XPKG}python2-pillow ${XPKG}cython2 ${XPKG}python2-setuptools ${XPKG}python2-cx_Freeze #python3 versions (not all are really needed if just using python3 for sound): pacman --no-confirm -S ${XPKG}python3-numpy ${XPKG}python3-pillow ${XPKG}cython ${XPKG}python3-cx_Freeze #using easy-install for python libraries which are not packaged by mingw: for x in rencode xxhash netifaces lz4 websocket-client comtypes PyOpenGL PyOpenGL_accelerate websockify cffi pycparser cryptography nvidia-ml-py; do easy-install-2.7 -U -Z $x easy-install-3.5 -U -Z $x done
remaining issues (some important like pygtkglext, but not blockers):
--with-server --with-shadow
to help cx_Freeze, and the sound subapp doesn't work at all..
NVENC related installation issues (only available on 64-bit):
Build xpra into dist:
cd /e svn co https://xpra.org/svn/Xpra cd Xpra/trunk/src python2 ./add_build_info.py #x265 causes crashes! BUILD_ARGS=--without-enc_x265 python2 ./setup.py build_ext --inplace ${BUILD_ARGS} python2 ./setup.py install_exe --install=./dist #for sound (still broken) #python3 ./setup.py build_ext --inplace ${BUILD_ARGS} #python3 ./setup.py install_exe --install=./dist/Sound --with-server --with-shadow
The resulting executables in dist
should all run.
New minor TODO: the setup.py rebuilds all the cython extensions, even when not needed. (cython bug?)
@smo: please try this out, you will need a build system you can use as support for the old py2exe one will be removed soon.
patch for dsextras to support newer gcc versions
use cython to access shell functions
For lack of a better place, some good links for debugging Cython / C++ / DLL issues on win32:
Turns out that the dependency on a Windows 8.1 Set that I was seeing on my Windows 7 build system (dependency on api-ms-win-core-libraryloader-l1-2-0.dll
) was actually caused by the linker flag -lwindowsapp
. Replacing it with -lole32
fixes the problem. Obviously. Not.
failed attempt at implementing IPropertyStore access using a C++ cython module
The window grouping code (see #799) has (finally) been re-implemented using a C++ helper in r14795. Notes:
Documentation useful for writing native code:
modified websockify for win32
New extra setup steps:
pacman -S liblzo2-devel export LZO_DIR=$MINGW_PREFIX python2 ./setup.py install python3 ./setup.py install
easy_install-2.7 minify && easy_install-3.5 minify
, install Java and add java.exe to your PATH before running the build
pacman -S ${XPKG}opencv ${XPKG}hdf5 ${XPKG}tesseract-ocr
xpra attach --ssh=ssh.exe
. The problem is that this would bring a large number of dependencies with it:
cp /c/msys32/usr/bin/ssh.exe dist/ cp /c/msys32/usr/bin/sshpass.exe dist/ for x in ssh.exe sshpass.exe msys-gcc_s-1.dll msys-2.0.dll msys-ssp-0.dll msys-crypto-1.0.0.dll \ msys-z.dll msys-gssapi-3.dll msys-heim* msys-krb5-26.dll msys-asn1-8.dll \ msys-com_err-1.dll msys-roken-18.dll msys-roken-18.dll msys-crypt* \ msys-wind-0.dll msys-hx509-5.dll msys-sqlite3* cp /c/msys32/usr/bin/$x dist/ done
And also it doesn't have a GUI for prompting for username or password.. Maybe we can add this later as an option. And maybe this should be kept in a separate sub-directory.
Notes:
pacman -Syu
" brings a nice set of updates: cython 0.25.2, libpng-1.6.28, gstreamer 1.10.2, gdk-pixbuf 2.36.3, flac 1.3.2, ffmpeg 3.2.2, lz4 1.7.5, numpy 1.11.2, etc
we still need to patch python-lzo for 1.11 (fixed in trunk)
r14808 adds a shell script (yes shell!) for generating the installer, just run this from a mingw shell:
./win32/PY27_MINGW_BUILD.sh
To get the EXE installer.
Still TODO:
The dlls cannot be trimmed further, best to just move to a new webcam API: #1231. r14811 removed the "api-ms-win-*dll" which came from Tortoise Plink, and fixes the pixbuf loader. The latest windows beta upload was made using this mingw build, and it looks good! (still need to fix SIGINT hander?)
PS: just how awesome is mingw-w64? the latest update brings python-2.7.13 and zlib 1.2.11!
Minor updates:
Note: the named pipes code is still broken, so you need to start shadow servers with "--bind=none
" to workaround that.
Here's how I figured out how to build the libyuv.dll:
git clone https://chromium.googlesource.com/libyuv/libyuv cd libyuv wget "https://www.xpra.org/trac/raw-attachment/ticket/678/libyuv-nojpeg.patch" patch -p1 < ./libyuv-nojpeg.patch cmake -G "MSYS Makefiles" make #build may fail on "convert.exe", #if so then remove it from the Makefile, #then run: rm libyuv.a;make VERBOSE=1 #copy the long ar.exe command that generates "libyuv.a", #and modify to build the DLL instead: gcc -shared -o libyuv.dll \ CMakeFiles/yuv.dir/source/compare.cc.obj \ CMakeFiles/yuv.dir/source/compare_common.cc.obj \ CMakeFiles/yuv.dir/source/compare_neon.cc.obj \ CMakeFiles/yuv.dir/source/compare_neon64.cc.obj \ CMakeFiles/yuv.dir/source/compare_gcc.cc.obj \ CMakeFiles/yuv.dir/source/compare_win.cc.obj \ CMakeFiles/yuv.dir/source/convert.cc.obj \ CMakeFiles/yuv.dir/source/convert_argb.cc.obj \ CMakeFiles/yuv.dir/source/convert_from.cc.obj \ CMakeFiles/yuv.dir/source/convert_from_argb.cc.obj \ CMakeFiles/yuv.dir/source/convert_jpeg.cc.obj \ CMakeFiles/yuv.dir/source/convert_to_argb.cc.obj \ CMakeFiles/yuv.dir/source/convert_to_i420.cc.obj \ CMakeFiles/yuv.dir/source/cpu_id.cc.obj \ CMakeFiles/yuv.dir/source/mjpeg_decoder.cc.obj \ CMakeFiles/yuv.dir/source/mjpeg_validate.cc.obj \ CMakeFiles/yuv.dir/source/planar_functions.cc.obj \ CMakeFiles/yuv.dir/source/rotate.cc.obj \ CMakeFiles/yuv.dir/source/rotate_any.cc.obj \ CMakeFiles/yuv.dir/source/rotate_argb.cc.obj \ CMakeFiles/yuv.dir/source/rotate_common.cc.obj \ CMakeFiles/yuv.dir/source/rotate_mips.cc.obj \ CMakeFiles/yuv.dir/source/rotate_msa.cc.obj \ CMakeFiles/yuv.dir/source/rotate_neon.cc.obj \ CMakeFiles/yuv.dir/source/rotate_neon64.cc.obj \ CMakeFiles/yuv.dir/source/rotate_gcc.cc.obj \ CMakeFiles/yuv.dir/source/rotate_win.cc.obj \ CMakeFiles/yuv.dir/source/row_any.cc.obj \ CMakeFiles/yuv.dir/source/row_common.cc.obj \ CMakeFiles/yuv.dir/source/row_mips.cc.obj \ CMakeFiles/yuv.dir/source/row_msa.cc.obj \ CMakeFiles/yuv.dir/source/row_neon.cc.obj \ CMakeFiles/yuv.dir/source/row_neon64.cc.obj \ CMakeFiles/yuv.dir/source/row_gcc.cc.obj \ CMakeFiles/yuv.dir/source/row_win.cc.obj \ CMakeFiles/yuv.dir/source/scale.cc.obj \ CMakeFiles/yuv.dir/source/scale_any.cc.obj \ CMakeFiles/yuv.dir/source/scale_argb.cc.obj \ CMakeFiles/yuv.dir/source/scale_common.cc.obj \ CMakeFiles/yuv.dir/source/scale_mips.cc.obj \ CMakeFiles/yuv.dir/source/scale_msa.cc.obj \ CMakeFiles/yuv.dir/source/scale_neon.cc.obj \ CMakeFiles/yuv.dir/source/scale_neon64.cc.obj \ CMakeFiles/yuv.dir/source/scale_gcc.cc.obj \ CMakeFiles/yuv.dir/source/scale_win.cc.obj \ CMakeFiles/yuv.dir/source/video_common.cc.obj #then we can install everything by hand: #(a better way would be to work out how to tell cmake about our prefix) rsync -rplogtv include/libyuv* ${MINGW_PREFIX}/include/ cp libyuv.a ${MINGW_PREFIX}/lib/ cp libyuv.dll ${MINGW_PREFIX}/bin/ #add a pkgconfig file so the library can be found by build tools: #(this .pc version is for mingw 32-bit): wget "https://www.xpra.org/trac/raw-attachment/ticket/678/libyuv.pc" mv libyuv.pc ${MINGW_PREFIX}/lib/pkgconfig/
For actually building our csc_libyuv module on 64-bit, you will also need r14853 . Then you can verify that the libyuv codec has been built properly by running the "Encoding_info.exe" tool.
don't link jpeg with libyuv
32-bit pkgconfig file
Note: we may need to submit this patch to mingw for libvpx: gstreamer libvpx crashes when encoding on x86_32.
$ python2 Python 2.7.13 (default, Jan 17 2017, 13:56:44) [GCC 6.3.0 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import gi Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named gi
turns out I was missing this one
pacman -S mingw64/mingw-w64-x86_64-python2-gobject
Added to wiki: https://xpra.org/trac/wiki/Building/MSWindows?action=diff&version=5
Note: the system update warns you about shortcuts going stale, I just created new ones by dragging "mingw32.exe" / "mingw64.exe" to the desktop.
The systray code has been moved to its own ticket: #1434
r15053 uninstalls xpra (if present) before installing a new version. This needs to be backported. Without this change, installing an older version over a 2.x installation breaks everything. (not sure why innosetup doesn't handle this properly)
Interesting read: win32api handle incompatible with ctypes?.
Following at least some this advice, we now use WinDLL
: r15244
Note: for keeping the python libraries up to date, just re-run this again as needed:
for x in rencode xxhash netifaces lz4 websocket-client comtypes PyOpenGL PyOpenGL_accelerate websockify cffi pycparser cryptography nvidia-ml-py; do easy_install-2.7 -U -Z $x easy_install-3.5 -U -Z $x done
python-lz4 0.9.0 doesn't build on win32, not the first breakage in this release: #1464.
r15307 moves the wiki instructions to shell scripts, including an update script for the python libraries, wiki updated: wiki/Building/MSWindows.
Minor fixes: r15610, r15609, r15613
See also #1523
See also #1528.
The 2.0.3 release is being held up by some MSYS2 breakage: some update broke numpy on 64-bit.
Breakage fixed, see link in comment:55.
Works for me. Not heard too many complaints about 2.x win32 builds either.
pygobject2 patch
Made a pull request (github's online editor is painful to use for patch files..): https://github.com/Alexpux/MINGW-packages/pull/2686
More regressions caused by the move to ctypes (these method signatures should be generated correctly once and for all): #1545, r16289. Python3 fix: r16281.
More pywin32 legacy code removed in r17317.
Some recent update broke setuptools (that whole thing is a terrible mess), you get ImportError: No module named 'pkg_resources._vendor.packaging'
.
More information on that here: No module named pkg_resources.
The solution I have used to get going again (and hope that MSYS2 fixes the package sooner or later), on 64-bit:
pacman -S mingw-w64-x86_64-python2-pip rm -fr /mingw64/lib/python2.7/site-packages/setuptools* pip install setuptools
FWIW pywin32 may get ported to MSYS2: https://github.com/msys2/MINGW-packages/pull/5615
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/678