Hello, think that this enhancement may be considered as well. The proxy listens the incoming connections and creates sessions on back-end xpra servers. Each session should has:
P.S. Waiting for the standard feature inside xpra proxy server I will try to adapt HAproxy (or other) for such function. will keep you updated on that.
Please clarify:
variable data the ip of host server, socket port(based on back end server status and user status if session already open)
What does this do? Is the connection target fixed or not?
application name and settings (based on client profile)
what does this mean?
1) users' sessions will be migrating from one backend server to another in regard of server actual load at the moment of connection. So it means that after the session termination --exit with children the next application launch for the same user might occur on another server. 2) for each user we can define which application is started by default. For first user - OpenOffice?, for the second - Firefox, etc
This looks like a lot of work, not well specced out yet, and I don't have time for this... So this is very unlikely to move forward unless someone else starts working on this. (hence the milestone "future")
Ok, sure. I'll try to assemble the solution for the test.though have not still seen how it could work in HAPROxy (or others) with dynamic websockets... But anyway to continue with this study it is better to have users list in mySQL table rather than in multifile(to be able to update the connection stings on-fly without blocking a multifile completly (as it is needed to add the line for the user server ip and port in order to give xpra proxy idea where to connect during Logon). Would it be possible to have it in the coming updates? Should I create a specific ticket for that?
.. it is better to have users list in mySQL table rather than in multifile ..
Yeah, that would be more do-able and good first step.
Hello, no results of research for now. I thought that http://zookeeper.apache.org/ could help. but no way - the worker should have been launched and listen the port in order to be used for client service.
That means that the simple broker should be written. Will try to write one.
Hello, Started to test the module with load balancing (everything seems fine,server and port choice, open ports checks, etc) but the error happens when starting the new process (tests with gedit application):
- (gedit:28350): WARNING **: Could not open display Xpra: Fafal IO error (Resource gemporary unavailable) on X server :4 (gedit:28350): Gtk-Warning **: cannot open display: :4
Test with 2 users (2 sessions with gedit lauched by 2 separate accounts on the same server). The first session :10000 starts normally but the next session on :10001 fails..
:-) Server reboot solved the issue :-) Some makeup needed for the code of module but it works. Antoine, found the code of miltifile auth. And will try to make smth similair for "sql mode" Are guid and uid are needed?
Hello, Checked once again the proxy connextion process. The script sent yesterday won't work (need to add a couple of lines but that not a problem at all) to be executed of TARGETHOST server as
And for the test purposes will try to adapt multifile auth process to sql base process at my server. Antoine, 1) will it work if i just remplace file reading to sql data reading in multifile_auth.py? As far as understood funtion get_sessions from class Authenticator is called by proxy_sever.py each time then new user connects and so if the sql tablw with session is modified before this call Proxy should establish the new connection. Am i right? 2) where it is better(which py module and which function) to place the call of Load balancing process? The script sent yesterday which create session for user on TARGETHOST? 3) what is the idea of uid and gid? They should be just unique for each user and that's all?
Thank a lot in advance for the answers!! Denis
1) correct: the proxy will call get_sessions
on each authenticator instance - one gets created for each authentication attempt, then discarded (it contains state, ie: the list of sessions)
2) not sure without looking at the script, please attach it to this ticket - I would have thought that load-balancing should be separate from authentication. You could start with just a random / round-robin dispatch initially.
3) the uid and gid can be used for two things:
Hello, In the attached files: script to start sessions on the remote target hosts and sqlite to manage connections (load balancing with round-robin). In fact it can work with multifile for auth process (in this case the call for create_session() should be added to xpra_proxy near call for get_sessions() - line 196. To prepare the session on targethost. Or for my purposes (temporary waiting for proper sql authorisation) i'll modify parse_filedata in multifile_auth to read data directly from sql table. But of cause it is not a good way of doing - multifile file should exist anyway (even with fakes data inside) in order to keep working Authenticator and FileAuthenticatorBase? classes. Tried to figue out how to modify them (for example to read if the multifile has the extention -sdb so it should be sql connection) but failed - too complex for me to understand the code. But if you could explain how to create a new class(or probably better adapt the existing ones - might be easier) for sql auth i will try to do that.
Don't try to modify the file auth modules, there is no code to re-use in there.
Just create a new auth module, see the stub attached and fill in the blanks.
Load balancing, session creation and SQL auth are all separate things. Don't put them all in the same module.
Hello, it got me absolutely crazy today. (HTML client, and Proxy and Targethost are on the same server) It the attached file the test script which prepares the connection for Proxy. If the script is started from Putty or in Terminal directly (no matter before start of Proxy or after) Proxy creates the link for the client and everything works.
2017-04-15 09:12:00,420 ProxyProcess(1003, 1003, {'EXAMPLE_ENV': 'VALUE'}, {}, '/var/run/user/$UID/xpra', [], ['swscale'], tcp websocket: 151.248.112.232:443 <- 88.105.53.196:52836, '"{\'encoder\': \'bencode\', \'max_packet_size\': 10485 .. \'cipher_out_block_size\': 0, \'cipher_in\': None}"', None, None, tcp socket: 151.248.112.232:56186 <- 151.248.112.232:10000, '<class \'xpra.util.typedict\'>: "{\'named_cursors\': 0, \'encoding.h264.deblocking- .. lipboard.selections\': [\'CLIPBOARD\', \'PRIMARY\']}"..', <multiprocessing.queues.Queue object at 0x26ed210>) 2017-04-15 09:12:00,420 starting <ProxyInstanceProcess(tcp websocket: 151.248.112.232:443 <- 88.105.53.196:52836, initial)> from pid=5221 2017-04-15 09:12:00,422 process started 2017-04-15 09:12:00,424 ProxyProcess.run() pid=5405, uid=0, gid=0 2017-04-15 09:12:00,479 Warning: libvpx ABI version 5 is too old: 2017-04-15 09:12:00,480 disabling YUV444P support with VP9
But if this script is called from Proxy module or Auth module - proxy returns the error:
2017-04-15 09:08:50,361 start_proxy(Protocol(tcp websocket: 151.248.112.232:443 <- 88.105.53.196:52836), {..}, None) found sessions: (1003, 1003, ['tcp:151.248.112.232:10000'], {'EXAMPLE_ENV': 'VALUE'}, {}) 2017-04-15 09:08:50,364 start_proxy: displays=['tcp:151.248.112.232:10000'], start-new-session=False 2017-04-15 09:08:50,364 start_proxy: proxy-virtual-display=:100 (ignored), user specified display=None, found displays=['tcp:151.248.112.232:10000'] 2017-04-15 09:08:50,364 start_proxy(Protocol(tcp websocket: 151.248.112.232:443 <- 88.105.53.196:52836), {..}, None) using server display at: tcp:151.248.112.232:10000 2017-04-15 09:08:50,364 display description(tcp:151.248.112.232:10000) = {'display_name': 'tcp:151.248.112.232:10000', 'uid': 1003, 'local': False, 'host': '151.248.112.232', 'gid': 1003, 'type': 'tcp', 'port': 10000} 2017-04-15 09:08:50,364 cannot connect Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/xpra/server/proxy/proxy_server.py", line 310, in start_proxy server_conn = connect_to(disp_desc, opts) File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 1854, in connect_to conn = _socket_connect(sock, tcp_endpoint, display_name, dtype) File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 1622, in _socket_connect raise InitException("failed to connect to '%s':\n %s" % (pretty_socket(endpoint), e)) InitException: failed to connect to '151.248.112.232:10000': [Errno 111] Connection refused 2017-04-15 09:08:50,365 Error: cannot start proxy connection: 2017-04-15 09:08:50,365 failed to connect to '151.248.112.232:10000': [Errno 111] Connection refused 2017-04-15 09:08:50,365 connection definition: 2017-04-15 09:08:50,366 * display_name : tcp:151.248.112.232:10000 2017-04-15 09:08:50,366 * gid : 1003 2017-04-15 09:08:50,366 * host : 151.248.112.232 2017-04-15 09:08:50,366 * local : False 2017-04-15 09:08:50,366 * port : 10000 2017-04-15 09:08:50,366 * type : tcp 2017-04-15 09:08:50,366 * uid : 1003 2017-04-15 09:08:50,367 disconnect(session not found error, ('failed to connect to display',))
Tried manay things but nothing worked....
I have no idea how and when you're running that script. But generally speaking, SSH failures like this one are likely to be one of 2 things:
From multifile_auth.py tried 2 places in authenticate_hmac and in get_sessions(self) by calling proxy_sessions.sessions_create() And in proxy_server.py in proxy_start. The same result.
uid I checked. And auth modules and proxy_server are under root. And I started this script under root as well.
And to connect to "remote" targethost the connection string for SSH command "xpra start..." is defined inside the script. So technically ssh command is launched by this user with the same rights.
From the same terminal "xpra proxy..." with script inside - doesn't work. And "Xpra proxy...." and then "python proxy_sessions.py" - works.
Just as a temporary solution (apache +php install is needed). Added in connect.html $ajax call to 1 page php (with 2 arguments:login and password) in doConnect function. Php page verifies login and pass in the same sqlite base. If OK starts python script to create a session. Seems to work properly. Tomorrow will send the modifs for connect.html, php page and new py load balancing script.
Hello, Tuning the chain html->php->python found the point that sometimes Proxy can't find the session because it is simply hasn't been created because of the latency of script (connection time of ssh etc) so getting back to start directly from python coult it be because of some asynchronius processus inside proxy or auth modules? And in this case those modules just don't see the session started on targethost and kill everythig without waiting? If so from where it is beffer to start session_creation module?
So definely it is not a latency. Proxy opens the connection strictly after second user's login. At first login (even if a session is correctly opened at targetost proxy_server.start_proxy() fails to connect to the socket(command server_conn=connect_to(...)) of targethost (error in the scripts/main.py in the block linked with python socket module). But on the second login (without the re-creation of session on targethost) it works. The only annoyning thing is to type twice the password in html client.... And php, etc is not needed as it works in the same way.... Is it possible at least to restart attempt to connect without asking a user for login at this point?
from where it is better to start session_creation module?
There is already the ability to start new sessions in the proxy server, look for the lines:
if self._start_sessions: #start a new session
This could be modularized and is the right place to hook code to start remote servers. There is already a run_remote_server
helper method which could be re-used for that.
OTOH: we can add a method to the auth modules that returns the servers to start a session on.
The default would be to return "None" and keep using the existing start_server_subprocess
code. The sqlauth module could have an optional query attribute that returns the servers to try to use. (initially a single server would be easier to implement)
But like I said before, do one thing at a time. Get the sqlauth module in shape first, then we can merge it. Only then can we look at changing the proxy server code to make it do more.
good morning. yes, you are absolutely right - one thing at a time. But having collected everything for sql turned back to loadbalancing and faced prbms, then had to test with apache and modified sql base in order to have some setting tables for php, apache connection etc..... And latest info on this issue :-) So for the test I included the start of session creation in auth file (exactly in auth_hmac). It is executed even before "self._start_sessions" in proxy. Auth is not a good place for that but in case of failure of script it just restarts auth process and doesn't block the whole Proxy connection process sending "server not found" to client (occurs then session is launched by the call directly from proxy_start). As the blocking error is socket prb - [Errno 111] Connection.refused I ve just added the call for socket.connect at the end of Load_balancing script(to emulate the first socket connection). At this first connection error the sestem is forced to reboot auth process, the second connect to socket at targethost is accepted and then Proxy easily connects the user without asking to retype the password!!!! So this error is not visible for the user at all. Tried that with multifile_auth hope that with sql that will work as well. So it seems that links with some init sockets process (socket init, opt init etc) later needs to see that more in details. And it might work differently on the really remote servers and not on the same one. But we will see later. So started to assemble everything together.
Note: the sql auth stub patch has been moved to #1488. This is where the work should start. I'm going to wait until something happens there before taking another look at this ticket.
Also, the comments are very confusing: apache? php? undocumented code changes, "reboot auth" (whatever that means). And generally feels like the wrong approach - difficult to say without seeing the code.
So new scripts: 1) sql_auth.py
Command used to start sql
Xpra proxy .... auth=sql..
2) xpra.sdb 3) xpra_lb.py (port search and session creation)
etc..
Hello, 1) ok, i'll put it in #1488 but after some refactoring as you mentioned here. 2) what do you mean by "creating the files instead"? Sqlite after "create" really creates a new empty DBase with such name. Do you mean create the complete copy(all the tables) of original DB and to use this new DB as temporary one and drop it then session is over? 3) ok. After some tests - HMAC is absolutly useless and can't solve the prb with sockets 4) hard-code SQL - Ok. I'll fix that.
Antoine.... i'm absolutly desperate with sessions creation on the target_host... tried a lot of thing but seems that only after xpra server reboots (after connection error) all the connections port becomes open As for example I tried to launch little py script
import sockets s=socket.socket() s.connect(host, port)
via SSH command line under the user (for whom we are preparion session) but all other open sessions the users were rebooted..... there are some infos on python socket prb....
http://stackoverflow.com/questions/16130786/why-am-i-getting-the-error-connection-refused-in-python-sockets
Probably there is a xpra's command to force session open the port without reinitialisation of xpra session? As after ssh command of "xpra start..." the unix process is well open (i mean it appears in "ps ax | grep" Could you please help.... i'm really upset after all those tests which have given nothing as result...
Do you mean create the complete copy(all the tables)
Yes. Create the schema.
HMAC is absolutly useless and can't solve the prb with sockets
No idea what you mean here. HMAC has nothing whatsoever to do with sockets.
i'm absolutly desperate with sessions creation on the target_host tried a lot of thing but seems that only after xpra server reboots
"server reboots" - again, as I said before, I have absolutely no idea what "reboot" means for xpra.
but all other open sessions the users were rebooted.....
You keep using that term: "reboot". You really need to explain what you mean here.
(..)
You have not explained (or not very well) what it is that you are trying to do here. The load-balancing is one thing, but you seem to be having basic problems with starting sessions over ssh. This has nothing to do with load balancing, so try to diagnose and report this problem separately before trying more complicated things. A shot in the dark: maybe your init system (systemd?) is killing the commands after your ssh session closes?
So as far as i understand r15693 allows to pass all the possible commands while creating of session by Proxy. Command of Proxy to create a session:
start_server_subprocess
and this command is called then displays is not defined. 1) is it possible to pass targethost IP and port to this command? 2) What is "_start_sessions = opts.proxy_start_sessions" for?
P.S. Sorry to ask probably very general questions but the code is quite complex for me
So as far as i understand r15693 allows to pass all the possible commands while creating of session by Proxy.
Not all, but most.
Command of Proxy to create a session:
start_server_subprocess
No. The function you want to call from the proxy is run_remote_server
.
and this command is called then displays is not defined. 1) is it possible to pass targethost IP and port to this command?
Yes, using run_remote_server
.
2) What is "_start_sessions = opts.proxy_start_sessions" for?
Configure whether the proxy server is allowed to start new sessions or not. Look for "proxy-start-sessions" in the config object, or in "65_proxy.conf"
Hello, tried to understand... several attempts... and just can't consume your time answering a lot of my stupid questions. Could you please help to code the request to create remote session? And i will prepare the script with a function which returns server and port address (as a session.displays data like tcp:XX.XX.XX.XX:YY or the return ip and port data as 2 separate values...) according all your previous recommendations.
And tried with "xpra start ssh" as well
comm_string = "xpra start ssh/user:pass@XXX.XXX.XXX.XXX:22 --bind-tcp=XXX.XXX.XXX.XXX:10000 --start-child=/bin/gedit --attach=no" subprocess.call(comm_string, shell=True)
Probably doing smth wrong but didn't succeed to start Without "shell" - doesn't work. Can't pass authorization of SSH. When try with "shell=True" Terminal appears asking for password. Auth passes well. And session is started. But after attempt to connect with HTML client the session gives an error -
Actual display used:3 Actual log file name is: /run/user/1000/xpra/:3.log xpra initialisation error: failed to identify the new server display!
Version is 2.1-r15785 64-bit Centos - 7.3.1611
Help :-(
The right way to start a remote xpra server is to use run_remote_server
(as I mentioned already in comment:21, comment:29), not by starting a new xpra process.
Can't pass authorization of SSH.
Details? Errors? Logs?
Terminal appears asking for password.
xpra should be able to use "sshpass" if it is installed to feed the password to ssh.
failed to identify the new server display!
Lots of steps and details are missing, I cannot attempt to reproduce what you did if you don't document things.
Hello, was absent as had to switch from VPS to dedicated server. As provider sometimes stopped the sessions because of memory or smth. So to exclude any impact on port availability decided to switch to dedicated... So with sspass performs well. Fast and guaranteed ports' opening. But 2 problems: 1) works from Terminal but doesn't work from the code (then function is called directly by application to start the session) neither with the command
os.system("script")
no with
subprocess.call("script", shell=True)
Probably the explanation is if we set in subprocess,call "shell=False" - error:
File "/usr/lib64/python2.7/subprocess.py", line 524, in call return Popen(*popenargs, **kwargs).wait() File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory
2) doesn't take some arguments. For example "--exit-with-children"
--exit-with-children option does not take a value
Concerning
run_remote_server()
found such function in /scripts/main.py. Seems to be the core function in fact for "start xpra ssh/..." command and can be called from start_proxy() directly in the case of need of remote session. But the function needs to have the following arguments as:
(error_cb, opts, args, mode, defaults)
some of them seem to be the global variables and so don't need to be defined. But can't figure out how to pass the needed options as IP, Port, path to application to be started, etc) to start the session... By setting of some parameters of global variables?
:-) Please don't take in account the questions on
run_remote_server()
I will play with that and will be back with the precise questions on "opts" and "defaults" settings
And a new update :-) 1) could you please add the option "--exit-with-children" to run_remote_server(). I tried to set up directly in "opts" attribute (to True) but the same error -
2017-05-06 06:23:45,832 xpra initialization error: 2017-05-06 06:23:45,832 --exit-with-children option does not take a value
2) then starting from Terminal by command line:
xpra start ssh/XX:YY@XXX.XXX.XXX.XXX:22 .......
sometimes happens that 2 similar sessions are created at the same time
/usr/bin/python2 /usr/bin/xpra start --bind-tcp=XXX.XXX.XXX.XXX:10000 --start-child=/bin/gedit --env=XPRA_PROXY_START_UUID=d94eadd638314779b4c36c79a278e2a3--systemd-run=no /usr/bin/python2 /usr/bin/xpra start --bind-tcp=XXX.XXX.XXX.XXX:10000 --start-child=/bin/gedit --env=XPRA_PROXY_START_UUID=20594515b877419bb8f509a0fb5068a2 --systemd-run=no
2 sessions happens when server says:
Entering daemon mode; any further errors will be reported to: /run/user/1000/xpra/S5757.log Actual display used: :0 Actual log file name is now: /run/user/1000/xpra/:0.log 2017-05-06 06:27:19,069 xpra initialization error: 2017-05-06 06:27:19,069 failed to identify the new server display! 2017-05-06 06:27:19,069 Killed by signal 1.
and in user's folder "/run/user/1000/xpra/" 2 sessions (files) are created. Both of them in the attached files.
And when system says:
Actual display used: :0 Actual log file name is now: /run/user/1000/xpra/:0.log Killed by signal 1.
Everything is OK. Only 1 session is created.
And add as a separate comment. so finally figure out how to set up all the arguments and having started from "start_proxy()".
run_remote_server(err, opt1, args, mode, defaults)
system gives the following error
2017-05-06 09:19:33,695 server error processing new connection from Protocol(tcp websocket: XXX.XXX.XXX.XXX:443 <- YYY.YYY.YYY.YYY:58328): signal only works in main thread Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/xpra/server/server_core.py", line 951, in _process_hello self.hello_oked(proto, packet, c, auth_caps) File "/usr/lib64/python2.7/site-packages/xpra/server/proxy/proxy_server.py", line 178, in hello_oked self.start_proxy(proto, c, auth_caps) File "/usr/lib64/python2.7/site-packages/xpra/server/proxy/proxy_server.py", line 253, in start_proxy run_remote_server(err, opt1, args, mode, defaults) File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 2276, in run_remote_server app = ConnectTestXpraClient((connect(), params), opts) File "/usr/lib64/python2.7/site-packages/xpra/client/gobject_client_base.py", line 109, in __init__ GObjectXpraClient.init(self, opts) File "/usr/lib64/python2.7/site-packages/xpra/client/gobject_client_base.py", line 34, in init self.install_signal_handlers() File "/usr/lib64/python2.7/site-packages/xpra/client/client_base.py", line 179, in install_signal_handlers signal.signal(signal.SIGINT, app_signal) ValueError: signal only works in main thread 2017-05-06 09:19:33,697 Disconnecting client YYY.YYY.YYY.YYY:58328: 2017-05-06 09:19:33,697 server error (error accepting new connection)
but requested session is started and port is open.
How it can be fixed?
so finally figure out how to set up all the arguments and having started from "start_proxy()".
Then you should post it.
How it can be fixed?
No idea what you changed, so I cannot help you.
For the test just put some hard code lines but the following modifs were done: line 30:
from xpra.scripts.main import parse_display_name, connect_to, start_server_subprocess, run_remote_server, fixup_defaults, do_parse_cmdline
Then after the line 228
opts = make_defaults_struct()
have put the lines. IP address should be typed instead of XXX, and application used here is /gedit and instead of "log" and "pass" real data should be used to connect by ssh
defaults = make_defaults_struct() fixup_defaults(defaults) aa = [] opt1, args1 = do_parse_cmdline(aa, defaults) opt1.bind_tcp = ['XXX.XXX.XXX.XXX:10000'] opt1.start_child = ['/bin/gedit'] opt1.attach = False mode="start" args=['ssh/log:pass@XXX.XXX.XXX.XXX:22'] def err(*args): raise InitException(*args) run_remote_server(err, opt1, args, mode, defaults)
Please use the patch format.
hm, patch file. See what you mean but don't know how to do it easily. Is there any special tool for that (just can't see how it can to done in Geany...)?
To generate a patch:
svn diff
or:
diff -u oldfile newfile
proxy_server.patch
is that correct?
This code to start a new session belongs with the existing code to start new sessions, near:
if self._start_sessions:
Maybe it just needs an extra switch, which you can hardcode to True for now. (this switch may be added to auth modules or whatever - not a big deal)
display
", otherwise the proxy server won't know where to connect to. At the moment, your changes go through:
display = None
So this cannot work.
run_remote_server
:
GObjectXpraClient.INSTALL_SIGNAL_HANDLERS = False
Ideally, the ssh subprocess would be tied to the proxy thread so we would make sure we kill it if the proxy fails to start. I can't see how OTOH.
Hello, 1) Thanks!
INSTALL_SIGNAL_HANDLERS = False.
resolved the error. But anyway it seems that the system stays in the function "run_remote_server()" and does not return to the "start_proxy()" to continue the execution of code.... So log file just stays at:
Entering daemon mode; any further errors will be reported to: /run/user/1000/xpra/S1103.log Actual display used: :1 Actual log file name is now: /run/user/1000/xpra/:1.log Killed by signal 1.
And session on the remote server is created but then - neither action nor error... Tried to put the code "print("the sessions is created" after the call for "run_remote_server" - no print event occurs. 2) And if we get back to the comment 35, it happens really often then 2 sessions are created. So if the creation time is more then 3-4 seconds for sure i can say that there will be 2 sessions created instead of 1.
Hello, 1) concerning config object -"opts". "run-remote_server()" takes both arguments and "opts" (as instance) and "defaults" (as class). The only way i have found to create "instance" is parse_command_line. Probably it is the best solution for now but for the test reasons it works. And anyway it will be used later as the command line (in the same format as used everywhere now) is needed to request proxy to start the remote session.
2) Use "display" to connect Proxy to session. Ok, sure. Will test that. The point in this case which needs to be thought out is how to link "display" and "port" as port is used to balancing (open/close...) now. On the other hand (from previous tests) normally Proxy can connect to open session by using the "sessions" data with uid, guid, tcp etc. Probably i just don't understand the logic of SSH and proxy.... but, ok. will study it in details then get thought the stopping point from previous comment.
3) move "ru_remote" call closer to "if self._start_sessions" - ok.
Not heard back, closing.
If we do implement load-balancing, a clean new ticket would be better.
See also #675.
this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/1486