xpra icon
Bug tracker and wiki

Opened 4 weeks ago

Last modified 19 hours ago

#1486 new enhancement

Proxy load balancing and

Reported by: Denis01 Owned by: Denis01
Priority: major Milestone: future
Component: server Version: trunk
Keywords: proxy Cc:

Description

Hello,
think that this enhancement may be considered as well.
The proxy listens the incoming connections and creates sessions on back-end xpra servers.
Each session should has:

  • variable data the ip of host server, socket port(based on back end server status and user status if session already open)
  • application name and settings (based on client profile).

P.S.
Waiting for the standard feature inside xpra proxy server I will try to adapt HAproxy (or other) for such function.
will keep you updated on that.

Attachments (6)

xpra_bal.py (6.1 KB) - added by Denis01 2 weeks ago.
xpra(1).sdb (28.0 KB) - added by Denis01 2 weeks ago.
proxy_sessions.py (790 bytes) - added by Denis01 2 weeks ago.
xpra_lb.py (7.3 KB) - added by Denis01 11 days ago.
sql_auth.py (5.1 KB) - added by Denis01 11 days ago.
xpra.sdb (28.0 KB) - added by Denis01 11 days ago.

Download all attachments as: .zip

Change History (35)

comment:1 Changed 4 weeks ago by Antoine Martin

Milestone: future
Owner: changed from Antoine Martin to Denis01

Please clarify:

variable data the ip of host server, socket port(based on back end server status and user status if session already open)

What does this do?
Is the connection target fixed or not?

application name and settings (based on client profile)

what does this mean?

comment:2 Changed 4 weeks ago by Denis01

1) users' sessions will be migrating from one backend server to another in regard of server actual load at the moment of connection. So it means that after the session termination --exit with children the next application launch for the same user might occur on another server.
2) for each user we can define which application is started by default. For first user - OpenOffice?, for the second - Firefox, etc

comment:3 Changed 4 weeks ago by Antoine Martin

This looks like a lot of work, not well specced out yet, and I don't have time for this...
So this is very unlikely to move forward unless someone else starts working on this. (hence the milestone "future")

comment:4 Changed 4 weeks ago by Denis01

Ok, sure.
I'll try to assemble the solution for the test.though have not still seen how it could work in HAPROxy (or others) with dynamic websockets...
But anyway to continue with this study it is better to have users list in mySQL table rather than in multifile(to be able to update the connection stings on-fly without blocking a multifile completly (as it is needed to add the line for the user server ip and port in order to give xpra proxy idea where to connect during Logon). Would it be possible to have it in the coming updates? Should I create a specific ticket for that?

comment:5 Changed 4 weeks ago by Antoine Martin

.. it is better to have users list in mySQL table rather than in multifile ..

Yeah, that would be more do-able and good first step.

comment:6 Changed 4 weeks ago by Denis01

Hello,
no results of research for now. I thought that http://zookeeper.apache.org/ could help. but no way - the worker should have been launched and listen the port in order to be used for client service.

That means that the simple broker should be written.
Will try to write one.

comment:7 Changed 3 weeks ago by Denis01

Hello,
Started to test the module with load balancing (everything seems fine,server and port choice, open ports checks, etc) but the error happens when starting the new process (tests with gedit application):

- (gedit:28350): WARNING **: Could not open display 
Xpra: Fafal IO error (Resource gemporary unavailable) on X server :4
(gedit:28350): Gtk-Warning **: cannot open display: :4

Test with 2 users (2 sessions with gedit lauched by 2 separate accounts on the same server). The first session :10000 starts normally but the next session on :10001 fails..

Last edited 3 weeks ago by Antoine Martin (previous) (diff)

comment:8 Changed 3 weeks ago by Denis01

:-)
Server reboot solved the issue :-)
Some makeup needed for the code of module but it works.
Antoine, found the code of miltifile auth. And will try to make smth similair for "sql mode"
Are guid and uid are needed?

comment:9 Changed 3 weeks ago by Denis01

Hello,
Checked once again the proxy connextion process. The script sent yesterday won't work (need to add a couple of lines but that not a problem at all) to be executed of TARGETHOST server as

  • password file for user being connected
  • xpra attach call

And for the test purposes will try to adapt multifile auth process to sql base process at my server. Antoine,
1) will it work if i just remplace file reading to sql data reading in multifile_auth.py?
As far as understood funtion get_sessions from class Authenticator is called by proxy_sever.py each time then new user connects and so if the sql tablw with session is modified before this call Proxy should establish the new connection. Am i right?
2) where it is better(which py module and which function) to place the call of Load balancing process? The script sent yesterday which create session for user on TARGETHOST?
3) what is the idea of uid and gid? They should be just unique for each user and that's all?

Thank a lot in advance for the answers!!
Denis

comment:10 Changed 3 weeks ago by Antoine Martin

1) correct: the proxy will call get_sessions on each authenticator instance - one gets created for each authentication attempt, then discarded (it contains state, ie: the list of sessions)
2) not sure without looking at the script, please attach it to this ticket - I would have thought that load-balancing should be separate from authentication. You could start with just a random / round-robin dispatch initially.
3) the uid and gid can be used for two things:

  • security: changing the proxy instance process owner when the master proxy process runs as root. ie: so the proxy server can listen on a privileged port (<1024), and each proxy instance won't run as root
  • locating local sessions: some authentication modules will find local sessions for local users, by specifying those uid / gid, the proxy will be able to locate them

comment:11 Changed 2 weeks ago by Denis01

Hello,
In the attached files: script to start sessions on the remote target hosts and sqlite to manage connections (load balancing with round-robin).
In fact it can work with multifile for auth process (in this case the call for create_session() should be added to xpra_proxy near call for get_sessions() - line 196. To prepare the session on targethost.
Or for my purposes (temporary waiting for proper sql authorisation) i'll modify parse_filedata in multifile_auth to read data directly from sql table.
But of cause it is not a good way of doing - multifile file should exist anyway (even with fakes data inside) in order to keep working Authenticator and FileAuthenticatorBase? classes.
Tried to figue out how to modify them (for example to read if the multifile has the extention -sdb so it should be sql connection) but failed - too complex for me to understand the code.
But if you could explain how to create a new class(or probably better adapt the existing ones - might be easier) for sql auth i will try to do that.

Changed 2 weeks ago by Denis01

Attachment: xpra_bal.py added

Changed 2 weeks ago by Denis01

Attachment: xpra(1).sdb added

comment:12 Changed 2 weeks ago by Antoine Martin

Don't try to modify the file auth modules, there is no code to re-use in there.

Just create a new auth module, see the stub attached and fill in the blanks.

Load balancing, session creation and SQL auth are all separate things.
Don't put them all in the same module.

comment:13 Changed 2 weeks ago by Denis01

Hello,
it got me absolutely crazy today. (HTML client, and Proxy and Targethost are on the same server)
It the attached file the test script which prepares the connection for Proxy.
If the script is started from Putty or in Terminal directly (no matter before start of Proxy or after) Proxy creates the link for the client and everything works.

2017-04-15 09:12:00,420 ProxyProcess(1003, 1003, {'EXAMPLE_ENV': 'VALUE'}, {}, '/var/run/user/$UID/xpra', [], ['swscale'], tcp websocket: 151.248.112.232:443 <- 88.105.53.196:52836, '"{\'encoder\': \'bencode\', \'max_packet_size\': 10485 ..  \'cipher_out_block_size\': 0, \'cipher_in\': None}"', None, None, tcp socket: 151.248.112.232:56186 <- 151.248.112.232:10000, '<class \'xpra.util.typedict\'>: "{\'named_cursors\': 0, \'encoding.h264.deblocking- .. lipboard.selections\': [\'CLIPBOARD\', \'PRIMARY\']}"..', <multiprocessing.queues.Queue object at 0x26ed210>)
2017-04-15 09:12:00,420 starting <ProxyInstanceProcess(tcp websocket: 151.248.112.232:443 <- 88.105.53.196:52836, initial)> from pid=5221
2017-04-15 09:12:00,422 process started
2017-04-15 09:12:00,424 ProxyProcess.run() pid=5405, uid=0, gid=0
2017-04-15 09:12:00,479 Warning: libvpx ABI version 5 is too old:
2017-04-15 09:12:00,480  disabling YUV444P support with VP9

But if this script is called from Proxy module or Auth module - proxy returns the error:

2017-04-15 09:08:50,361 start_proxy(Protocol(tcp websocket: 151.248.112.232:443 <- 88.105.53.196:52836), {..}, None) found sessions: (1003, 1003, ['tcp:151.248.112.232:10000'], {'EXAMPLE_ENV': 'VALUE'}, {})
2017-04-15 09:08:50,364 start_proxy: displays=['tcp:151.248.112.232:10000'], start-new-session=False
2017-04-15 09:08:50,364 start_proxy: proxy-virtual-display=:100 (ignored), user specified display=None, found displays=['tcp:151.248.112.232:10000']
2017-04-15 09:08:50,364 start_proxy(Protocol(tcp websocket: 151.248.112.232:443 <- 88.105.53.196:52836), {..}, None) using server display at: tcp:151.248.112.232:10000
2017-04-15 09:08:50,364 display description(tcp:151.248.112.232:10000) = {'display_name': 'tcp:151.248.112.232:10000', 'uid': 1003, 'local': False, 'host': '151.248.112.232', 'gid': 1003, 'type': 'tcp', 'port': 10000}
2017-04-15 09:08:50,364 cannot connect
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xpra/server/proxy/proxy_server.py", line 310, in start_proxy
    server_conn = connect_to(disp_desc, opts)
  File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 1854, in connect_to
    conn = _socket_connect(sock, tcp_endpoint, display_name, dtype)
  File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 1622, in _socket_connect
    raise InitException("failed to connect to '%s':\n %s" % (pretty_socket(endpoint), e))
InitException: failed to connect to '151.248.112.232:10000':
 [Errno 111] Connection refused
2017-04-15 09:08:50,365 Error: cannot start proxy connection:
2017-04-15 09:08:50,365  failed to connect to '151.248.112.232:10000':
 [Errno 111] Connection refused
2017-04-15 09:08:50,365  connection definition:
2017-04-15 09:08:50,366  * display_name       : tcp:151.248.112.232:10000
2017-04-15 09:08:50,366  * gid                : 1003
2017-04-15 09:08:50,366  * host               : 151.248.112.232
2017-04-15 09:08:50,366  * local              : False
2017-04-15 09:08:50,366  * port               : 10000
2017-04-15 09:08:50,366  * type               : tcp
2017-04-15 09:08:50,366  * uid                : 1003
2017-04-15 09:08:50,367 disconnect(session not found error, ('failed to connect to display',))

Changed 2 weeks ago by Denis01

Attachment: proxy_sessions.py added

comment:14 Changed 2 weeks ago by Denis01

Tried manay things but nothing worked....

comment:15 Changed 2 weeks ago by Antoine Martin

I have no idea how and when you're running that script.
But generally speaking, SSH failures like this one are likely to be one of 2 things:

  • wrong uid, environment or file permissions when running from a different context
  • missing tty when running deamonized

comment:16 Changed 2 weeks ago by Denis01

From multifile_auth.py
tried 2 places in authenticate_hmac and in get_sessions(self)
by calling proxy_sessions.sessions_create()
And in proxy_server.py in proxy_start.
The same result.

uid I checked. And auth modules and proxy_server are under root. And I started this script under root as well.

And to connect to "remote" targethost the connection string for SSH command "xpra start..." is defined inside the script. So technically ssh command is launched by this user with the same rights.

Last edited 2 weeks ago by Denis01 (previous) (diff)

comment:17 Changed 2 weeks ago by Denis01

From the same terminal "xpra proxy..." with script inside - doesn't work. And
"Xpra proxy...." and then "python proxy_sessions.py" - works.

comment:18 Changed 2 weeks ago by Denis01

Just as a temporary solution (apache +php install is needed).
Added in connect.html $ajax call to 1 page php (with 2 arguments:login and password) in doConnect function.
Php page verifies login and pass in the same sqlite base. If OK starts python script to create a session.
Seems to work properly.
Tomorrow will send the modifs for connect.html, php page and new py load balancing script.

comment:19 Changed 13 days ago by Denis01

Hello,
Tuning the chain html->php->python found the point that sometimes Proxy can't find the session because it is simply hasn't been created because of the latency of script (connection time of ssh etc) so getting back to start directly from python coult it be because of some asynchronius processus inside proxy or auth modules? And in this case those modules just don't see the session started on targethost and kill everythig without waiting? If so from where it is beffer to start session_creation module?

Last edited 13 days ago by Denis01 (previous) (diff)

comment:20 Changed 13 days ago by Denis01

So definely it is not a latency.
Proxy opens the connection strictly after second user's login.
At first login (even if a session is correctly opened at targetost proxy_server.start_proxy() fails to connect to the socket(command server_conn=connect_to(...)) of targethost (error in the scripts/main.py in the block linked with python socket module).
But on the second login (without the re-creation of session on targethost) it works.
The only annoyning thing is to type twice the password in html client....
And php, etc is not needed as it works in the same way....
Is it possible at least to restart attempt to connect without asking a user for login at this point?

comment:21 Changed 13 days ago by Antoine Martin

from where it is better to start session_creation module?

There is already the ability to start new sessions in the proxy server, look for the lines:

            if self._start_sessions:
                #start a new session

This could be modularized and is the right place to hook code to start remote servers. There is already a run_remote_server helper method which could be re-used for that.

OTOH: we can add a method to the auth modules that returns the servers to start a session on.
The default would be to return "None" and keep using the existing start_server_subprocess code. The sqlauth module could have an optional query attribute that returns the servers to try to use. (initially a single server would be easier to implement)

But like I said before, do one thing at a time.
Get the sqlauth module in shape first, then we can merge it. Only then can we look at changing the proxy server code to make it do more.

comment:22 Changed 12 days ago by Denis01

good morning.
yes, you are absolutely right - one thing at a time.
But having collected everything for sql turned back to loadbalancing and faced prbms, then had to test with apache and modified sql base in order to have some setting tables for php, apache connection etc.....
And latest info on this issue :-)
So for the test I included the start of session creation in auth file (exactly in auth_hmac). It is executed even before "self._start_sessions" in proxy. Auth is not a good place for that but in case of failure of script it just restarts auth process and doesn't block the whole Proxy connection process sending "server not found" to client (occurs then session is launched by the call directly from proxy_start).
As the blocking error is socket prb - [Errno 111] Connection.refused
I ve just added the call for socket.connect at the end of Load_balancing script(to emulate the first socket connection). At this first connection error the sestem is forced to reboot auth process, the second connect to socket at targethost is accepted and then Proxy easily connects the user without asking to retype the password!!!! So this error is not visible for the user at all.
Tried that with multifile_auth hope that with sql that will work as well.
So it seems that links with some init sockets process (socket init, opt init etc) later needs to see that more in details. And it might work differently on the really remote servers and not on the same one. But we will see later.
So started to assemble everything together.

Last edited 12 days ago by Denis01 (previous) (diff)

comment:23 Changed 11 days ago by Antoine Martin

Note: the sql auth stub patch has been moved to #1488. This is where the work should start.
I'm going to wait until something happens there before taking another look at this ticket.

Also, the comments are very confusing: apache? php? undocumented code changes, "reboot auth" (whatever that means). And generally feels like the wrong approach - difficult to say without seeing the code.

comment:24 Changed 11 days ago by Denis01

So new scripts:
1) sql_auth.py

  • have not figued out how to pass the directory location for sql db file. So used the fix location in init

Command used to start sql

Xpra proxy .... auth=sql..
  • had to keep hmac definition and place to start port search and session activation (as explained before)

2) xpra.sdb
3) xpra_lb.py (port search and session creation)

Changed 11 days ago by Denis01

Attachment: xpra_lb.py added

Changed 11 days ago by Denis01

Attachment: sql_auth.py added

Changed 11 days ago by Denis01

Attachment: xpra.sdb added

comment:25 Changed 11 days ago by Antoine Martin

  • as per comment:23, the correct ticket for sql auth is #1488. Please post your code there.
  • do not attach sqlite database files, post instructions for creating the files instead. Or better yet, add the creation commands to the sql auth script - the template is already there
  • do not duplicate the authenticate method
  • the indentation is wrong in some places
  • do not use hard-coded SQL queries, do not use "select *"

etc..

comment:26 Changed 10 days ago by Denis01

Hello,
1) ok, i'll put it in #1488 but after some refactoring as you mentioned here.
2) what do you mean by "creating the files instead"? Sqlite after "create" really creates a new empty DBase with such name. Do you mean create the complete copy(all the tables) of original DB and to use this new DB as temporary one and drop it then session is over?
3) ok. After some tests - HMAC is absolutly useless and can't solve the prb with sockets
4) hard-code SQL - Ok. I'll fix that.

Antoine....
i'm absolutly desperate with sessions creation on the target_host...
tried a lot of thing but seems that only after xpra server reboots (after connection error) all the connections port becomes open
As for example I tried to launch little py script

import sockets
s=socket.socket()
s.connect(host, port)

via SSH command line under the user (for whom we are preparion session) but all other open sessions the users were rebooted.....
there are some infos on python socket prb....

http://stackoverflow.com/questions/16130786/why-am-i-getting-the-error-connection-refused-in-python-sockets

Probably there is a xpra's command to force session open the port without reinitialisation of xpra session?
As after ssh command of "xpra start..." the unix process is well open (i mean it appears in "ps ax | grep"
Could you please help.... i'm really upset after all those tests which have given nothing as result...

comment:27 Changed 10 days ago by Antoine Martin

Do you mean create the complete copy(all the tables)

Yes. Create the schema.

HMAC is absolutly useless and can't solve the prb with sockets

No idea what you mean here. HMAC has nothing whatsoever to do with sockets.

i'm absolutly desperate with sessions creation on the target_host
tried a lot of thing but seems that only after xpra server reboots

"server reboots" - again, as I said before, I have absolutely no idea what "reboot" means for xpra.

but all other open sessions the users were rebooted.....

You keep using that term: "reboot". You really need to explain what you mean here.

(..)

You have not explained (or not very well) what it is that you are trying to do here.
The load-balancing is one thing, but you seem to be having basic problems with starting sessions over ssh. This has nothing to do with load balancing, so try to diagnose and report this problem separately before trying more complicated things.
A shot in the dark: maybe your init system (systemd?) is killing the commands after your ssh session closes?

comment:28 Changed 23 hours ago by Denis01

So as far as i understand r15693 allows to pass all the possible commands while creating of session by Proxy.
Command of Proxy to create a session:

start_server_subprocess

and this command is called then displays is not defined.
1) is it possible to pass targethost IP and port to this command?
2) What is "_start_sessions = opts.proxy_start_sessions" for?

P.S. Sorry to ask probably very general questions but the code is quite complex for me

Last edited 20 hours ago by Antoine Martin (previous) (diff)

comment:29 in reply to:  28 Changed 19 hours ago by Antoine Martin

So as far as i understand r15693 allows to pass all the possible commands while creating of session by Proxy.

Not all, but most.

Command of Proxy to create a session:

start_server_subprocess

No. The function you want to call from the proxy is run_remote_server.

and this command is called then displays is not defined.
1) is it possible to pass targethost IP and port to this command?

Yes, using run_remote_server.

2) What is "_start_sessions = opts.proxy_start_sessions" for?

Configure whether the proxy server is allowed to start new sessions or not.
Look for "proxy-start-sessions" in the config object, or in "65_proxy.conf"

Note: See TracTickets for help on using tickets.