Singularity with xpra, very close =D

[devin]

Hello there,

I'm trying to share an xpra server that is running in singularity with my local machine. the setup involves a couple jumps between this server and the local machine

local --> bastion host --> cluster --> active job with xpra server

I've setup the ssh command to be able to access this active job, but because we are arriving via ssh as a user who hasn't started a singularity instance with xpra in it (example singularity shell xpra.sif) command the xpra program and its setup environment aren't available.

A de-identified version of the results are here

Warning: vendor 'Intel Open Source Technology Center' is greylisted,
 you may want to turn off OpenGL if you encounter bugs
2020-08-20 16:45:40,653 Xpra GTK3 X11 client version 4.0.2-r26625 64-bit
2020-08-20 16:45:40,772  running on Linux Ubuntu 20.04 focal
2020-08-20 16:45:40,774  window manager is 'GNOME Shell'
2020-08-20 16:45:40,800 Warning: failed to import opencv:
2020-08-20 16:45:40,800  No module named 'cv2'
2020-08-20 16:45:40,801  webcam forwarding is disabled
2020-08-20 16:45:41,122 GStreamer version 1.16.2 for Python 3.8.2 64-bit
2020-08-20 16:45:41,123 Warning: no pulseaudio information available
2020-08-20 16:45:41,123  No module named 'distutils.spawn'
2020-08-20 16:45:41,280 No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
2020-08-20 16:45:41,552 Warning: vendor 'Intel Open Source Technology Center' is greylisted,
2020-08-20 16:45:41,552  you may want to turn off OpenGL if you encounter bugs
2020-08-20 16:45:41,657 OpenGL enabled with Mesa DRI Intel(R) HD Graphics 4000 (IVB GT2)
2020-08-20 16:45:41,792  keyboard settings: rules=evdev, model=pc105, layout=us
2020-08-20 16:45:41,795  desktop size is 1366x768 with 1 screen:
2020-08-20 16:45:41,795   :0.0 (361x203 mm - DPI: 96x96) workarea: 1294x741 at 72x27
2020-08-20 16:45:41,796     BOE LVDS-1 (309x173 mm - DPI: 112x112)
ssh user@bastion ssh cluster ssh user@activejob sh -c 'xpra initenv;if [ -x $XDG_RUNTIME_DIR/xpra/run-xpra ]; then $XDG_RUNTIME_DIR/xpra/run-xpra _proxy;elif which "xpra" > /dev/null 2>&1; then xpra _proxy;elif [ -x /usr/local/bin/xpra ]; then /usr/local/bin/xpra _proxy;elif [ -x ~/.xpra/run-xpra ]; then ~/.xpra/run-xpra _proxy;elif [ -x Xpra_cmd.exe ]; then Xpra_cmd.exe _proxy;else echo "no run-xpra command found"; exit 1; fi'
initenv: xpra: command not found
2020-08-20 16:45:48,680 no run-xpra command found

I just need some way to specify the singularity command singularity shell xpra.sif in between the ssh user@bastion ssh cluster ssh user@activejob and the sh -c xpra initenv... so that xpra actually exists. How can I make this happen? I feel like there's a ton of people who could start using xpra if this kind of simple support existed, I hope its an easy thing for y'all.

cheers,
Devin

[Antoine Martin]

Please see wiki/ReportingBugs.
ie: I don't know what commands you are using or how you installed xpra.

.. the xpra program and its setup environment aren't available.
(..)
no run-xpra command found

Means that xpra is not installed wherever it is that you are connecting to via ssh.
That's never going to work.

You may want to try:

• using openssh instead of paramiko: --ssh='ssh -v'
• adding ssh debugging: -d ssh
• look into ssh ProxyCommand, sshpass, etc.

[devin]

Sorry I left a lot out. Let me try to populate some of the things suggested in the bug report wiki.

the os for the local machine is ubuntu 20.04, the os for the active node is a singularity container built using ubuntu 20.04.

the full command to start the xpra server is

singularity shell xpra.sif;xpra start --start="./blender" --daemon=no

I've struggled to understand how xpra transforms the xpra attach ssh... command so I modified the /usr/lib/python3/dist-packages/xpra/scripts/main.py to override the full_ssh variable on line 1012 to be
full_ssh = ['ssh', 'user@bastion','ssh','cluster', 'ssh', 'user@activejob']
so the full command to attach the client is less illustrative than in other cases perhaps.

xpra attach --ssh="ssh -v" ssh://cluster?proxy=user@bastion

Sorry there's so much sketchy stuff associated with this ticket.

I'll definitely check out the -d ssh debugging command, and try to unpack the ProxyCommand and sshpass like you suggest.

Means that xpra is not installed wherever it is that you are connecting to via ssh.
That's never going to work.

Correct, if I have no way to introduce a small command before it tries to start up the xpra code then since its missing from the environment it shouldn't succeed.

However, this ssh command

ssh user@bastion ssh cluster ssh -t -t user@activejob 'singularity exec /home/u4/baylyd/Documents/test4.simg xpra --version'

returns
xpra v3.0.8-r25889

To me this shows its possible to give xpra commands to run within the environment, but the current way that attach works centers on using this after ssh login

sh -c 'xpra initenv;if [ -x $XDG_RUNTIME_DIR/xpra/run-xpra ]; then $XDG_RUNTIME_DIR/xpra/run-xpra _proxy;elif which "xpra" > /dev/null 2>&1; then xpra _proxy;elif [ -x /usr/local/bin/xpra ]; then /usr/local/bin/xpra _proxy;elif [ -x ~/.xpra/run-xpra ]; then ~/.xpra/run-xpra _proxy;elif [ -x Xpra_cmd.exe ]; then Xpra_cmd.exe _proxy;else echo "no run-xpra command found"; exit 1; fi'

---

So my question is really about how do I make this (which I believe is what xpra attach generates)

ssh user@bastion ssh cluster ssh user@activejob sh -c 'xpra initenv;if [ -x $XDG_RUNTIME_DIR/xpra/run-xpra ]; then $XDG_RUNTIME_DIR/xpra/run-xpra _proxy;elif which "xpra" > /dev/null 2>&1; then xpra _proxy;elif [ -x /usr/local/bin/xpra ]; then /usr/local/bin/xpra _proxy;elif [ -x ~/.xpra/run-xpra ]; then ~/.xpra/run-xpra _proxy;elif [ -x Xpra_cmd.exe ]; then Xpra_cmd.exe _proxy;else echo "no run-xpra command found"; exit 1; fi'

into this

ssh user@bastion ssh cluster ssh -t -t user@activejob 'singularity exec /home/u4/baylyd/Documents/test4.simg xpra initenv;if [ -x $XDG_RUNTIME_DIR/xpra/run-xpra ]; then $XDG_RUNTIME_DIR/xpra/run-xpra _proxy;elif which "xpra" > /dev/null 2>&1; then xpra _proxy;elif [ -x /usr/local/bin/xpra ]; then /usr/local/bin/xpra _proxy;elif [ -x ~/.xpra/run-xpra ]; then ~/.xpra/run-xpra _proxy;elif [ -x Xpra_cmd.exe ]; then Xpra_cmd.exe _proxy;else echo "no run-xpra command found"; exit 1; fi '

?

[Antoine Martin]

I've struggled to understand how xpra transforms..

Yeah, it's really not easy to follow!

xpra v3.0.8-r25889

That's out of date.

To me this shows its possible to give xpra commands to run within the environment, but the current way that attach works centers on using this after ssh login
sh -c 'xpra initenv;if [ -x $XDG_RUNTIME_DIR/xpra/run-xpra ]; then $XDG_RUNTIME_DIR/xpra/run-xpra _proxy;elif ...

That's true for --ssh=ssh (openssh), the paramiko backend (which should be the default) does something similar but without executing the whole thing as a single shell command.

However, this ssh command
ssh user@bastion ssh cluster ssh -t -t user@activejob 'singularity exec /home/u4/baylyd/Documents/test4.simg xpra --version'

Then can't you just do this:

xpra attach --ssh='ssh cluster ssh -t -t user@activejob' --remote-xpra="yourscript"

(or something like it using ssh and ProxyCommand)
And have yourscript just run singularity exec /home/u4/baylyd/Documents/test4.simg xpra $@?

[devin]

Oh yes, let me make sure the other xpra is up to date.

I'll give the last command a try. I'm really not sure how to create the correct xpra command, is it going to be

xpra attach --ssh='ssh cluster ssh -t -t user@activejob' --remote-xpra="yourscript" ssh://user@bastion

?

[Antoine Martin]

is it going to be ..

I'm not entirely sure. Try with -d ssh and tweak as needed.

[devin]

Oh, good reminder. Thanks so much! I'll reply with results or other snags.

[devin]

so for the --remote-xpra should I put in --remote-xpra="bash /path/to/my_script.sh" ? or should I leave out the bash, and the path?

[devin]

if I'm reading the results of the -d ssh correctly then I think the --remote-xpra is still placing the command in the wrong spot?

xpra attach --ssh='ssh user@bastion ssh -t -t cluster' --remote-xpra="bash /home/u4/baylyd/xpra_start.sh" -d ssh ssh://user@activejob
Warning: vendor 'Intel Open Source Technology Center' is greylisted,
 you may want to turn off OpenGL if you encounter bugs
2020-08-24 09:50:54,697 Xpra GTK3 X11 client version 4.0.2-r26625 64-bit
2020-08-24 09:50:54,844  running on Linux Ubuntu 20.04 focal
2020-08-24 09:50:54,848  window manager is 'GNOME Shell'
2020-08-24 09:50:54,874 Warning: failed to import opencv:
2020-08-24 09:50:54,874  No module named 'cv2'
2020-08-24 09:50:54,874  webcam forwarding is disabled
2020-08-24 09:50:55,309 GStreamer version 1.16.2 for Python 3.8.2 64-bit
2020-08-24 09:50:55,311 Warning: no pulseaudio information available
2020-08-24 09:50:55,311  No module named 'distutils.spawn'
2020-08-24 09:50:55,518 No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
2020-08-24 09:50:55,888 Warning: vendor 'Intel Open Source Technology Center' is greylisted,
2020-08-24 09:50:55,888  you may want to turn off OpenGL if you encounter bugs
2020-08-24 09:50:56,031 OpenGL enabled with Mesa DRI Intel(R) HD Graphics 4000 (IVB GT2)
2020-08-24 09:50:56,045 parse_ssh_string(ssh user@bastion ssh -t -t cluster)
2020-08-24 09:50:56,209 executing ssh command: "ssh" "user@bastion" "ssh" "-t" "-t" "cluster" "-l" "user" "-T" "activejob" "sh -c 'xpra initenv;if [ -x bash /home/u4/user/xpra_start.sh ]; then bash /home/u4/user/xpra_start.sh _proxy;else echo "no run-xpra command found"; exit 1; fi'"

It looks like it still tries to run the xpra initenv first?

I haven't actually seen what it does because the parsing of the ssh is still off. In the example invocation above xpra attach --ssh='ssh user@bastion ssh -t -t cluster' --remote-xpra="bash /home/u4/baylyd/xpra_start.sh" -d ssh ssh://user@activejob it takes the last part activejob and tries to use it as a command ? I guess I can see the parsed ssh_string being formed that way "-T" "activejob", but I have no idea how else to form this correct ssh command.

unsurprisingly we get the error
bash: activejob: command not found

[Antoine Martin]

so for the --remote-xpra should I put in ...

Just try and see what works.
remove bash, it should not be needed if the file is executable and will cause problems with the shell command that is generated.
The full path is needed unless the file is somewhere on the default $PATH.

.. is still placing the command in the wrong spot?

Hmm.

It looks like it still tries to run the xpra initenv first?

Yes and that's fine. Any failures there won't be fatal.

[devin]

Ok I'll try it without the bash in the --remote-xpra hopefully that'll fix the program not running. Worst case, I'm going to try putting in the command I'm hoping to supply in the net/ssh.py file around line 876 just so I know its even possible once the formatting is right.

[devin]

I'm still running into the xpra command not found.
initenv: xpra: command not found

unless I go in and modify this line in net/ssh.py

876         full_string = "ssh user@bastion ssh cluster ssh user@activejob -T 'singularity exec /home/user/Documents/test4.simg xpra initenv;/usr/bin/xpra _proxy'"
877         child = Popen(full_string.split(" "), stdin=PIPE, stdout=PIPE, **kwargs)

then this fails with

Warning: failed to create script directory '/run/user/25391/xpra':
 [Errno 2] No such file or directory: '/run/user/25391/xpra'
 ($XDG_RUNTIME_DIR has not been created?)
bash: /usr/bin/xpra: No such file or directory

but if I use a script that modifies $PATH by adding my user's home directory and sets $XDG_RUNTIME_DIR to that then I wind up seeing

xpra initialization error:
 cannot find any live servers to connect to

and I think this has to do with the singularity instance that I'm starting upon executing xpra attach ... on my local machine can't see the process that I started during a different login.

To test this I made a script

export PATH=$PATH:/home/u4/user
export XDG_RUNTIME_DIR=/home/u4/user

cd /home/u4/user/Documents/blender-2.83.5-linux64
xpra start --start="./blender" --daemon=no &


sleep 15s


xpra initenv
/usr/bin/xpra _proxy

and I start it running with ssh user@bastion ssh cluster ssh user@activejob -T 'singularity exec /home/u4/user/Documents/test4.simg bash /home/u4/user/xpra_init_commands.sh'

and it boots up with the expected xpra output for starting a server, or atleast the output that I've seen othertimes that xpra worked between machines

X.Org X Server 1.19.6
Release Date: 2017-12-20
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-168-generic x86_64 Ubuntu
Current Operating System: Linux activejob 
3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1127.13.1.el7.x86_64 root=UUID=2a097d5a-e636-46f0-8ee8-255f0dfc7b1a ro
Build Date: 14 November 2019  06:20:00PM
xorg-server 2:1.19.6-1ubuntu4.4 (For technical support please see http://www.ubuntu.com/support) 
Current version of pixman: 0.34.0
	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(++) Log file: "/home/u4/baylyd/xpra/Xorg.S73392.log", Time: Mon Aug 24 11:20:09 2020
(++) Using config file: "/etc/xpra/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
2020-08-24 11:20:11,548 Warning: failed to create socket directory '/run/user/25391/xpra'
2020-08-24 11:20:11,548  [Errno 2] No such file or directory: '/run/user/25391/xpra'
2020-08-24 11:20:11,549 Warning: cannot create socket '/run/user/25391/xpra/activejob':
2020-08-24 11:20:11,549  [Errno 2] No such file or directory
2020-08-24 11:20:11,549  ($XDG_RUNTIME_DIR has not been created?)
2020-08-24 11:20:11,549 cannot create group socket '/run/xpra/activejob'
2020-08-24 11:20:11,549  [Errno 30] Read-only file system
2020-08-24 11:20:11,549 Warning: no local server sockets,
2020-08-24 11:20:11,549  forward-xdg-open cannot be enabled
2020-08-24 11:20:11,550  non-embedded ssh connections will not be available

(xpra:73392): Gtk-WARNING **: 11:20:11.613: Locale not supported by C library.
	Using the fallback 'C' locale.
2020-08-24 11:20:11,681 pointer device emulation using XTest
2020-08-24 11:20:11,769 serving html content from '/usr/share/xpra/www'
2020-08-24 11:20:11,782 Warning: no local sockets defined,
2020-08-24 11:20:11,782  disabling printer forwarding
2020-08-24 11:20:12,844 OpenGL is supported on display ':8'
2020-08-24 11:20:12,845  using 'llvmpipe (LLVM 9.0, 128 bits)' renderer
2020-08-24 11:20:12,890 D-Bus notification forwarding is available
2020-08-24 11:20:12,900 pulseaudio not started: 'pulseaudio' command not found

(Xpra-Audio-query:73487): Gtk-WARNING **: 11:20:13.199: Locale not supported by C library.
	Using the fallback 'C' locale.
2020-08-24 11:20:13,223 GStreamer version 1.14.5 for Python 3.6.9 64-bit

(Xpra:73392): Gtk-CRITICAL **: 11:20:13.226: gtk_widget_realize: assertion 'widget->priv->anchored || GTK_IS_INVISIBLE (widget)' failed
2020-08-24 11:20:13,237 xpra is ready.
2020-08-24 11:20:13,238 xpra GTK3 X11 version 3.0.8-r25889 64-bit
2020-08-24 11:20:13,294 Warning: webcam forwarding is disabled
2020-08-24 11:20:13,294  the virtual video directory '/sys/devices/virtual/video4linux' was not found
2020-08-24 11:20:13,295  make sure that the 'v4l2loopback' kernel module is installed and loaded
2020-08-24 11:20:13,295 found 0 virtual video devices for webcam forwarding
2020-08-24 11:20:13,299 Warning: cannot watch for application menu changes without pyinotify:
2020-08-24 11:20:13,299  No module named 'pyinotify'
2020-08-24 11:20:13,315  uid=25391 (baylyd), gid=339 (student)
2020-08-24 11:20:13,315  running with pid 73392 on Linux Ubuntu 18.04 bionic
2020-08-24 11:20:13,316  connected to X11 display :8 with 24 bit colors
2020-08-24 11:20:13,342 Error parsing xdg menu data:
2020-08-24 11:20:13,342  ParsingError in file '/etc/xdg/menus/kde-debian-menu.menu', File not found
2020-08-24 11:20:13,342  this is either a bug in python-xdg,
2020-08-24 11:20:13,342  or an invalid system menu configuration
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5007:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM default
AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': No such file or directory
2020-08-24 11:20:13,472 503.8GB of system memory

sadly it also ends with the same result.

xpra initialization error:
 cannot find any live servers to connect to

I'm not sure what's causing this. Anything I can do to provide more information? I think we are making progress!
--edit--
Oops I forgot to ask, do you think I'm better off using xpra start from my local machine? Is it straightforward to convert the existing attach command we are discussing with a new start command? it could be what is needed to make sure the server is connected up right.

[devin]

I'll mention that I continued to work on this today with the xpra start command instead.

I'm still running into the case where after quite some time where it looks like things might work I eventually get this error and the local machine errors out

xpra initialization error:
 failed to identify the new server display!
2020-08-24 15:38:48,404 Error: failed to receive anything, not an xpra server?
2020-08-24 15:38:48,404   could also be the wrong protocol, username, password or port
2020-08-24 15:38:48,404   or the session was not found
2020-08-24 15:38:48,405 Connection lost

I'll explain what I've done at this point. I've modified the net/ssh.py line 876 to be the complete ssh command I need

        full_string = "ssh user@bastion ssh cluster ssh user@activejob -T 'singularity exec /home/u4/user/Documents/test5.simg bash /home/u4/user/xpra_init_commands.sh'"
        child = Popen(full_string.split(" "), stdin=PIPE, stdout=PIPE, **kwargs)

this file xpra_init_commands.sh consists of the following

export PATH=$PATH:/home/u4/user
export XDG_RUNTIME_DIR=/home/u4/user


xpra initenv
$XDG_RUNTIME_DIR/xpra/run-xpra _proxy_start  --start="./blender" :99 --daemon=no

when I start the xpra start command certain things about the beginning command are ignored ultimately when the full_string command is used, but I'm including the command for completeness
xpra start --start="./blender" --ssh='** all the ssh info you've seen before ** ' ssh:user@activejob
we get this result

Warning: vendor 'Intel Open Source Technology Center' is greylisted,
 you may want to turn off OpenGL if you encounter bugs
2020-08-24 15:33:39,542 Xpra GTK3 X11 client version 4.0.2-r26625 64-bit
2020-08-24 15:33:39,688  running on Linux Ubuntu 20.04 focal
2020-08-24 15:33:39,693  window manager is 'GNOME Shell'
2020-08-24 15:33:39,728 Warning: failed to import opencv:
2020-08-24 15:33:39,728  No module named 'cv2'
2020-08-24 15:33:39,728  webcam forwarding is disabled
2020-08-24 15:33:40,171 GStreamer version 1.16.2 for Python 3.8.2 64-bit
2020-08-24 15:33:40,173 Warning: no pulseaudio information available
2020-08-24 15:33:40,173  No module named 'distutils.spawn'
2020-08-24 15:33:40,394 No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
2020-08-24 15:33:40,777 Warning: vendor 'Intel Open Source Technology Center' is greylisted,
2020-08-24 15:33:40,778  you may want to turn off OpenGL if you encounter bugs
2020-08-24 15:33:40,926 OpenGL enabled with Mesa DRI Intel(R) HD Graphics 4000 (IVB GT2)
####full str is  ssh user@bastion ssh cluster ssh user@activejob -T 'singularity exec /home/u4/user/Documents/test5.simg bash /home/u4/user/xpra_init_commands.sh'
2020-08-24 15:33:41,124  keyboard settings: rules=evdev, model=pc105, layout=us
2020-08-24 15:33:41,129  desktop size is 1366x768 with 1 screen:
2020-08-24 15:33:41,130   :0.0 (361x203 mm - DPI: 96x96) workarea: 1294x741 at 72x27
2020-08-24 15:33:41,130     BOE LVDS-1 (309x173 mm - DPI: 112x112)
Entering daemon mode; any further errors will be reported to:
  /home/u4/user/xpra/:100.log
xpra initialization error:
 failed to identify the new server display!
2020-08-24 15:35:17,502 Error: failed to receive anything, not an xpra server?
2020-08-24 15:35:17,502   could also be the wrong protocol, username, password or port
2020-08-24 15:35:17,502   or the session was not found
2020-08-24 15:35:17,502 Connection lost

and the contents of the :100.log are

X.Org X Server 1.19.6
Release Date: 2017-12-20
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-168-generic x86_64 Ubuntu
Current Operating System: Linux activejob.cluster 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1127.13.1.el7.x86_64 root=UUID=7fceea1a-6abd-4797-b6c3-183e614a424a ro
Build Date: 14 November 2019  06:20:00PM
xorg-server 2:1.19.6-1ubuntu4.4 (For technical support please see http://www.ubuntu.com/support) 
Current version of pixman: 0.34.0
	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(++) Log file: "/home/u4/user/xpra/Xorg.:100.log", Time: Mon Aug 24 15:33:47 2020
(++) Using config file: "/etc/xpra/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
2020-08-24 15:33:50,217 Warning: failed to create socket directory '/run/user/25391/xpra'
2020-08-24 15:33:50,218  [Errno 2] No such file or directory: '/run/user/25391/xpra'
2020-08-24 15:33:50,220 Warning: cannot create socket '/run/user/25391/xpra/activejob.cluster-100':
2020-08-24 15:33:50,220  [Errno 2] No such file or directory
2020-08-24 15:33:50,220  ($XDG_RUNTIME_DIR has not been created?)
2020-08-24 15:33:50,222 cannot create group socket '/run/xpra/activejob.cluster-100'
2020-08-24 15:33:50,222  [Errno 30] Read-only file system
2020-08-24 15:33:50,222 Warning: no local server sockets,
2020-08-24 15:33:50,222  forward-xdg-open cannot be enabled
2020-08-24 15:33:50,222  non-embedded ssh connections will not be available

(xpra:25044): Gtk-WARNING **: 15:33:50.293: Locale not supported by C library.
	Using the fallback 'C' locale.
2020-08-24 15:33:50,366 pointer device emulation using XTest
2020-08-24 15:33:50,461 serving html content from '/usr/share/xpra/www'
2020-08-24 15:33:50,473 Warning: no local sockets defined,
2020-08-24 15:33:50,473  disabling printer forwarding
2020-08-24 15:33:50,532 D-Bus notification forwarding is available
2020-08-24 15:33:50,548 pulseaudio not started: 'pulseaudio' command not found

(Xpra-Audio-query:25089): Gtk-WARNING **: 15:33:51.377: Locale not supported by C library.
	Using the fallback 'C' locale.
2020-08-24 15:33:51,420 GStreamer version 1.14.5 for Python 3.6.9 64-bit

(Xpra:25044): Gtk-CRITICAL **: 15:33:51.485: gtk_widget_realize: assertion 'widget->priv->anchored || GTK_IS_INVISIBLE (widget)' failed
2020-08-24 15:33:51,492 Warning: webcam forwarding is disabled
2020-08-24 15:33:51,492  the virtual video directory '/sys/devices/virtual/video4linux' was not found
2020-08-24 15:33:51,493  make sure that the 'v4l2loopback' kernel module is installed and loaded
2020-08-24 15:33:51,493 found 0 virtual video devices for webcam forwarding
2020-08-24 15:33:51,503 Warning: cannot watch for application menu changes without pyinotify:
2020-08-24 15:33:51,503  No module named 'pyinotify'
2020-08-24 15:33:51,536 xpra is ready.
2020-08-24 15:33:51,537 xpra GTK3 X11 version 4.0.3-r27082 64-bit
2020-08-24 15:33:51,562 Error parsing xdg menu data:
2020-08-24 15:33:51,562  ParsingError in file '/etc/xdg/menus/kde-debian-menu.menu', File not found
2020-08-24 15:33:51,562  this is either a bug in python-xdg,
2020-08-24 15:33:51,562  or an invalid system menu configuration
2020-08-24 15:33:51,945  uid=25391 (user), gid=339 (someone)
2020-08-24 15:33:51,945  running with pid 25044 on Linux Ubuntu 18.04 bionic
2020-08-24 15:33:51,948  connected to X11 display :100 with 24 bit colors
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5007:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM default
AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': No such file or directory
2020-08-24 15:33:52,186 503.8GB of system memory
2020-08-24 15:33:52,264 Error parsing xdg menu data:
2020-08-24 15:33:52,264  ParsingError in file '/etc/xdg/menus/kde-debian-menu.menu', File not found
2020-08-24 15:33:52,265  this is either a bug in python-xdg,
2020-08-24 15:33:52,265  or an invalid system menu configuration
2020-08-24 15:33:54,042 OpenGL is supported on display ':100'
2020-08-24 15:33:54,043  using 'llvmpipe (LLVM 9.0, 128 bits)' renderer

2020-08-24 15:39:47,335 GTK3 X11 Server got signal SIGTERM
2020-08-24 15:39:47,336 xpra GTK3 X11 server is terminating
2020-08-24 15:39:47,373 killing xvfb with pid 25046
(II) Server terminated successfully (0). Closing log file.
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":100"
      after 331 requests (331 known processed) with 3 events remaining.
Warning: Falling back to the standard locale ("C")
/home/u4/user/gvfs/ non-existent directory
found bundled python: /home/u4/user/Documents/blender-2.83.5-linux64/2.83/python

I think in this particular case I used CTRL-C to close the client after it hanged for a while, but othertimes it has closed on its own and the log is basically the same. I don't know what else to try.

Any thoughts? The fact I was able to start up the xpra server after connecting over ssh and executing the sigularity file makes me feel like there's maybe only a few more things I have to trouble shoot

[Antoine Martin]

I think it would help to completely separate the two problems:

• create a script command that starts xpra via singularity and make sure that works reliably (see below)
• find an --ssh=... command line that runs yourscript when invoked (maybe that's working already?)

Warning: no local server sockets,
forward-xdg-open cannot be enabled

That's likely to be the real problem here. That's a very common issue with containers.
The server can start, but the lack of local server sockets will prevent the proxy command from connecting to it.

Change your server config with something like:

echo "socket-dirs=/tmp" >> /etc/xpra/conf.d/90_socket_dirs_override.conf

I've modified the net/ssh.py line 876 to be the complete ssh command I need

I don't think you need to do that.

full str is ssh user@bastion ssh cluster ssh user@activejob -T 'singularity exec /home/u4/user/Documents/test5.simg bash /home/u4/user/xpra_init_commands.sh'

OTOH, something like this should do what you want:

xpra start --ssh='ssh user@bastion ssh cluster -T' ssh://user@activejob/ --run-xpra=/path/to/xpra-singularity-script

Where xpra-singularity-script should look something like this (untested):

singularity exec /home/u4/user/Documents/test5.simg xpra --daemon=no --env=XDG_RUNTIME_DIR=$HOME/user --env=PATH=$PATH:$HOME/user

ie:

xpra-singularity-script start --bind-tcp=0.0.0.0:10000 --start=xterm

Should start an xpra server with an xterm and make it available on port 10000 (wherever that's mapped by singularity). If that works, then the _proxy_start command should also work.

[devin]

I really like the idea of separating out the things that need to be figured out. I'm also going to install singularity outside the HPC on remote vm so that testing things out can go a little faster.

I'll try out the changes you've suggested. Have I mentioned how grateful I am that you're this responsive? Also I'm not kidding this could really change up the way people get to use our HPC system. Thanks for all your hard work, and assistance!

[devin]

is the --run-xpra supposed to be --remote-xpra ?

I'm still pretty sure something isn't happening correctly with the xpra initenv.

Here's the output of your suggestions so far, I haven't modified the server config for the port point you made, and I'm pretty sure this is breaking before that becomes relevant

xpra start --ssh="ssh -v" ssh://baylyd@129.114.17.40 --remote-xpra=/vol_b/documents/xpra_singularity_script --start=xeyes
Warning: vendor 'Intel Open Source Technology Center' is greylisted,
 you may want to turn off OpenGL if you encounter bugs
2020-08-25 11:13:54,985 Xpra GTK3 X11 client version 4.0.2-r26625 64-bit
2020-08-25 11:13:55,136  running on Linux Ubuntu 20.04 focal
2020-08-25 11:13:55,139  window manager is 'GNOME Shell'
2020-08-25 11:13:55,167 Warning: failed to import opencv:
2020-08-25 11:13:55,167  No module named 'cv2'
2020-08-25 11:13:55,167  webcam forwarding is disabled
2020-08-25 11:13:55,601 GStreamer version 1.16.2 for Python 3.8.2 64-bit
2020-08-25 11:13:55,603 Warning: no pulseaudio information available
2020-08-25 11:13:55,603  No module named 'distutils.spawn'
2020-08-25 11:13:55,817 No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
2020-08-25 11:13:56,192 Warning: vendor 'Intel Open Source Technology Center' is greylisted,
2020-08-25 11:13:56,193  you may want to turn off OpenGL if you encounter bugs
2020-08-25 11:13:56,341 OpenGL enabled with Mesa DRI Intel(R) HD Graphics 4000 (IVB GT2)
inserted, above executing the command, did it still use start?
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/lil/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 129.114.17.40 [129.114.17.40] port 22.
debug1: Connection established.
debug1: identity file /home/lil/.ssh/id_rsa type 0
debug1: identity file /home/lil/.ssh/id_rsa-cert type -1
debug1: identity file /home/lil/.ssh/id_dsa type -1
debug1: identity file /home/lil/.ssh/id_dsa-cert type -1
debug1: identity file /home/lil/.ssh/id_ecdsa type -1
debug1: identity file /home/lil/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/lil/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/lil/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/lil/.ssh/id_ed25519 type -1
debug1: identity file /home/lil/.ssh/id_ed25519-cert type -1
debug1: identity file /home/lil/.ssh/id_ed25519_sk type -1
debug1: identity file /home/lil/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/lil/.ssh/id_xmss type -1
debug1: identity file /home/lil/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to 129.114.17.40:22 as 'baylyd'
debug1: SSH2_MSG_KEXINIT sent
2020-08-25 11:13:56,540  keyboard settings: rules=evdev, model=pc105, layout=us
2020-08-25 11:13:56,544  desktop size is 1366x768 with 1 screen:
2020-08-25 11:13:56,544   :0.0 (361x203 mm - DPI: 96x96) workarea: 1294x741 at 72x27
2020-08-25 11:13:56,544     BOE LVDS-1 (309x173 mm - DPI: 112x112)
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:TIjS3XHcP8LRXE80GRrz//Cu053ZG14TUBDPo+Za5B4
debug1: Host '129.114.17.40' is known and matches the ECDSA host key.
debug1: Found key in /home/lil/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/lil/.ssh/id_rsa RSA SHA256:q1DpJgIbIPOYOlqc/fczfFuq1Rw6Cn28wY8TzTcuN8U agent
debug1: Will attempt key: /home/lil/.ssh/id_dsa 
debug1: Will attempt key: /home/lil/.ssh/id_ecdsa 
debug1: Will attempt key: /home/lil/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/lil/.ssh/id_ed25519 
debug1: Will attempt key: /home/lil/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/lil/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/lil/.ssh/id_rsa RSA SHA256:q1DpJgIbIPOYOlqc/fczfFuq1Rw6Cn28wY8TzTcuN8U agent
debug1: Server accepts key: /home/lil/.ssh/id_rsa RSA SHA256:q1DpJgIbIPOYOlqc/fczfFuq1Rw6Cn28wY8TzTcuN8U agent
debug1: Authentication succeeded (publickey).
Authenticated to 129.114.17.40 ([129.114.17.40]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: sh -c 'xpra initenv;if [ -x /vol_b/documents/xpra_singularity_script ]; then /vol_b/documents/xpra_singularity_script _proxy_start "--ssh=ssh -v" "--remote-xpra=/vol_b/documents/xpra_singularity_script" "--start=xeyes";else echo "no run-xpra command found"; exit 1; fi'
sh: 1: xpra: not found
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Transferred: sent 52020, received 2796 bytes, in 0.8 seconds
Bytes per second: sent 62530.2, received 3360.9
debug1: Exit status 1
2020-08-25 11:13:57,833 no run-xpra command found

where xpra_singularity_script contains

singularity exec /vol_b/documents/test6.simg xpra --daemon=no --env=XDG_RUNTIME_DIR=/vol_b/ --env=PATH=$PATH:/vol_b/

/vol_b/ is just the place I have the most storage so its kind of my home directory on this vm.

xpra-singularity-script start --bind-tcp=0.0.0.0:10000 --start=xterm
Should start an xpra server with an xterm and make it available on port 10000 (wherever that's mapped by singularity). If that works, then the _proxy_start command should also work.

baylyd@js-17-40:/vol_b/documents$ ./xpra_singularity_script start --bind-tcp=0.0.0.0:10000 --start=xeyes -d all

outputs

xpra: need a mode

I'm sort of puzzled about how the last command was supposed take the additional arguments and use them.

[Antoine Martin]

is the --run-xpra supposed to be --remote-xpra ?

Yes, sorry.

I'm still pretty sure something isn't happening correctly with the xpra initenv.

That's not important. xpra initenv is allowed to fail.

where xpra_singularity_script contains
(..)
xpra: need a mode

Opps: just add "$@" at the end of your script command:

singularity exec /vol_b/documents/test6.simg xpra --env=XDG_RUNTIME_DIR=/vol_b/ --env=PATH=$PATH:/vol_b/ "$@"

So that the arguments given to the script will be passed on to xpra. (forgot that when I gave you the example script)

[devin]

!!! ok!! the "$@" and some little fixes such as removing the non root user from the singularity container got me to the point where the bind-tcp command worked from within singularity!

./xpra_singularity_script start --bind-tcp=0.0.0.0:10000 --start="blender_folder/blender"

and then going to my machine's ip:10000 brought up a blender window in the browser. Good stuff!

now I'm going to try the ssh version of this.

[devin]

ok, so I still get the same issue that I saw on the HPC when I try to use ssh instead

this command, and no more inline src code changes =D
xpra start --start="blender_folder/blender" --ssh="ssh -v" ssh://baylyd@129.114.17.40 --remote-xpra=/home/baylyd/xpra_singularity_script
leads to this output (should I attach as file instead?)

Warning: vendor 'Intel Open Source Technology Center' is greylisted,
 you may want to turn off OpenGL if you encounter bugs
2020-08-26 12:23:15,794 Xpra GTK3 X11 client version 4.0.2-r26625 64-bit
2020-08-26 12:23:15,901  running on Linux Ubuntu 20.04 focal
2020-08-26 12:23:15,903  window manager is 'GNOME Shell'
2020-08-26 12:23:15,927 Warning: failed to import opencv:
2020-08-26 12:23:15,928  No module named 'cv2'
2020-08-26 12:23:15,928  webcam forwarding is disabled
2020-08-26 12:23:16,388 GStreamer version 1.16.2 for Python 3.8.2 64-bit
2020-08-26 12:23:16,389 Warning: no pulseaudio information available
2020-08-26 12:23:16,389  No module named 'distutils.spawn'
2020-08-26 12:23:16,555 No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
2020-08-26 12:23:16,849 Warning: vendor 'Intel Open Source Technology Center' is greylisted,
2020-08-26 12:23:16,849  you may want to turn off OpenGL if you encounter bugs
2020-08-26 12:23:16,961 OpenGL enabled with Mesa DRI Intel(R) HD Graphics 4000 (IVB GT2)
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/lil/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 129.114.17.40 [129.114.17.40] port 22.
debug1: Connection established.
debug1: identity file /home/lil/.ssh/id_rsa type 0
debug1: identity file /home/lil/.ssh/id_rsa-cert type -1
debug1: identity file /home/lil/.ssh/id_dsa type -1
debug1: identity file /home/lil/.ssh/id_dsa-cert type -1
debug1: identity file /home/lil/.ssh/id_ecdsa type -1
debug1: identity file /home/lil/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/lil/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/lil/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/lil/.ssh/id_ed25519 type -1
debug1: identity file /home/lil/.ssh/id_ed25519-cert type -1
debug1: identity file /home/lil/.ssh/id_ed25519_sk type -1
debug1: identity file /home/lil/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/lil/.ssh/id_xmss type -1
debug1: identity file /home/lil/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
2020-08-26 12:23:17,103  keyboard settings: rules=evdev, model=pc105, layout=us
2020-08-26 12:23:17,106  desktop size is 1366x768 with 1 screen:
2020-08-26 12:23:17,106   :0.0 (361x203 mm - DPI: 96x96) workarea: 1294x741 at 72x27
2020-08-26 12:23:17,106     BOE LVDS-1 (309x173 mm - DPI: 112x112)
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to 129.114.17.40:22 as 'baylyd'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:TIjS3XHcP8LRXE80GRrz//Cu053ZG14TUBDPo+Za5B4
debug1: Host '129.114.17.40' is known and matches the ECDSA host key.
debug1: Found key in /home/lil/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/lil/.ssh/id_rsa RSA SHA256:q1DpJgIbIPOYOlqc/fczfFuq1Rw6Cn28wY8TzTcuN8U agent
debug1: Will attempt key: /home/lil/.ssh/id_dsa 
debug1: Will attempt key: /home/lil/.ssh/id_ecdsa 
debug1: Will attempt key: /home/lil/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/lil/.ssh/id_ed25519 
debug1: Will attempt key: /home/lil/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/lil/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/lil/.ssh/id_rsa RSA SHA256:q1DpJgIbIPOYOlqc/fczfFuq1Rw6Cn28wY8TzTcuN8U agent
debug1: Server accepts key: /home/lil/.ssh/id_rsa RSA SHA256:q1DpJgIbIPOYOlqc/fczfFuq1Rw6Cn28wY8TzTcuN8U agent
debug1: Authentication succeeded (publickey).
Authenticated to 129.114.17.40 ([129.114.17.40]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: sh -c 'xpra initenv;if [ -x /home/baylyd/xpra_singularity_script ]; then /home/baylyd/xpra_singularity_script _proxy_start "--ssh=ssh -v" "--remote-xpra=/home/baylyd/xpra_singularity_script" "--start=blender_folder/blender";else echo "no run-xpra command found"; exit 1; fi'
sh: 1: xpra: not found
Entering daemon mode; any further errors will be reported to:
  /home/baylyd/xpra/S26060.log
Actual display used: :1
Actual log file name is now: /home/baylyd/xpra/:1.log
xpra initialization error:
 failed to identify the new server display!
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Transferred: sent 52108, received 3068 bytes, in 98.1 seconds
Bytes per second: sent 531.1, received 31.3
debug1: Exit status 1
2020-08-26 12:24:56,031 Error: failed to receive anything, not an xpra server?
2020-08-26 12:24:56,031   could also be the wrong protocol, username, password or port
2020-08-26 12:24:56,031   or the session was not found
2020-08-26 12:24:56,032 Connection lost

You're totally right, the rest of the script continued after the initial xpra initenv didn't find xpra

sh: 1: xpra: not found

then it went on to

Entering daemon mode; any further errors will be reported to: /home/baylyd/xpra/S26060.log

I'm glad you didn't get distracted by the error line.

I'm going to try your socket suggestion now and hopefully it is the last step that's needed.

[devin]

omg it worked!! Now I have to try it on the hpc!

[devin]

so close!

I still ran into some malformed ssh with xpra issue where the final jump was taken as a command to run after the -T, so I modified the net/ssh.py to put in the exact correct ssh command. This isn't a permanent fix, and I look forward to trying to trouble shoot it with you in time.

However, the socket modification that you suggested doesn't quite work yet. in the log created in the active job a long string of this spams the file

2020-08-26 13:38:40,096 New unix-domain connection received
2020-08-26 13:38:40,096  on '/tmp/activejob-0'
2020-08-26 13:38:40,220 New unix-domain connection received
2020-08-26 13:38:40,220  on '/tmp/activejob-0'
2020-08-26 13:38:40,343 New unix-domain connection received
2020-08-26 13:38:40,343  on '/tmp/activejob-0'
2020-08-26 13:38:40,467 New unix-domain connection received

So it seems like the server on the active job is now detected, but something else is going wrong.

[devin]

seems pretty similar to #1447, perhaps I let things get out of date with some of the other changes I was making.

[devin]

updating to xpra v4 has made it work! This is incredible! Thanks so much Antoine. I'll post to a write up that I'll make of this work later on.

[devin]

I'm still not sure why, but because of the way that ssh://user@activejob gets parsed, I wind up with a long ssh command that assumes activejob is the command. An example would be
xpra start --ssh="ssh firsthost ssh secondhost" ssh://user@thirdhost becomes something like
ssh firsthost ssh secondhost -l user -T thirdhost which isn't correct. Do I need to provide a sort of floating last ssh so that the command looks like
xpra start --ssh="ssh firsthost ssh secondhost ssh" ssh://user@thirdhost creating
ssh firsthost ssh secondhost ssh -l user -T thirdhost. I guess it just seems weird, but that might work.

[Antoine Martin]

updating to xpra v4 has made it work!

I'm not sure why, but OK.
The v3 LTS branch should work just as well.

I wind up with a long ssh command that assumes activejob is the command

Please file a separate ticket for that, and include the -d ssh output.

[devin]

OK creating new ticket. Thanks so much for all of this help!

#2867