OK - so the results are in. This made a MAJOR difference in usability, if an app is likely at all to get temporarily locked up (processing a large file or something). The fix was what @Travis Hill had suggested - thanks Travis. In the end, this works pretty much as I hoped and allows you to run separate instances of standalone apps across different processors and still keep the connections xojo needs to maintain sessions.
Nginx Conf:
upstream myproject {
ip_hash;
server 127.0.0.1:8080;
server 127.0.0.1:8081;
server 127.0.0.1:8082;
#add a listing for any port you are running an instance on
}
server {
listen 80;
location / {
proxy_buffering off;
proxy_read_timeout 5m;
proxy_pass http://myproject;
}
}
ip_hash;
is required to maintain session, app fails to launch without it.
proxy_buffering off;
needed to keep data flowing from app to client through proxy
proxy_read_timeout 5m;
keeps connection open for app “pings”. If left at default the app will disconnect after 60 seconds. I tried several settings between 1-10 minutes and this was the lowest setting that worked consistently. The app remains open.
In order to start the app instances, I used upstart. I am on CentOS but this should also work on RHEL and possibly Fedora and many others. This is my method of choice because it will automatically respawn instances when they fail, allows easy starting and stopping of the whole group and can exec your app as a non-root user. If you are not familiar with upstart, here are my app.conf files for your reference:
Upstart, first app instance:
[code]# /etc/init/myapp.conf
description ‘First instance of app’
author ‘john-joyce.com’
env LOG_FILE=/path/to/logfiles/myapp.log
env USER=
start on runlevel [2345]
stop on runlevel [016]
respawn
script
touch $LOG_FILE
chown $USER:$USER $LOG_FILE
exec su -s /bin/sh -c ‘exec “$0” “$@”’ $USER – /path/to/myapp --port=8080 >> $LOG_FILE 2>&1
end script
[/code]
Upstart, additional instances:
[code]# /etc/init/myapp1.conf
description ‘Additional instance of app’
author ‘john-joyce.com’
env LOG_FILE=/path/to/logfiles/myapp.log
env USER=
start on starting myapp
stop on stopping myapp
respawn
script
exec su -s /bin/sh -c ‘exec “$0” “$@”’ $USER – /path/to/myapp --port=8081 >> $LOG_FILE 2>&1
end script
[/code]
This is just how I did it, if you are launching 20 instances you might want to iterate through it in the script rather than have a file for each instance, but this was easy for just a few instances. You just duplicate the second conf file for as many instances as you want and change the port.
Then when you do ‘initctl myapp’ all of the instances will start, stop, restart.
Observations:
This worked surprisingly well and I would recommend this kind of setup for any app that is getting decent usage. In opening up and rendering an image file I was easily able to get the CPU up to 100% with just one browser, and as most of you know, xojo doesn’t spawn it’s own child process or anything so it is effectively locked up for anyone else trying to use it - even if you are running it on a 24 core server with plenty of memory. Just creating 2 additional instances makes a remarkable difference in performance in a multi-user situation.
Above you can see 1 instance of the app maxed out and unavailable (CPU 11) while the others are on CPUs 18,19. Instance 1 is totally unresponsive even though there are still 20+ idle cores. But the instances on 18 and 19 are unaffected and running fine. I have to say I was surprised how easy it was to overload a single instance of the app.
It should be noted that because the processes are grouped and routed by IP (same process receives all requests for any one IP) that users in an office using NAT might all appear as one IP to the server and all be routed to the same instance, bypassing the load-balancing for that office. I haven’t tested that though.
Let me know if this helps anyone out there - hope it does.
John