I have several Web Apps running on a Windows Server 2008 R2. It is a VMware virtual machine but it has 8GB of RAM and 4 virtual processors. There are two similar VM’s on one hardware box with 4 processors and 32GB of RAM.
When I get about 10 users the performance from the browser perspective can get almost un-usable.
If I run SpeedTest.net it shows OVER 350 Megs for download and upload on a university network. When I check the performance with TaskManager during a slow period it shows CPU usage typically at 25% or less with a few spikes and about 4GB of RAM used (50% of the configured RAM).
I am running the Abyss Web Server with all of the apps as CGI apps all SSL protected.
What is the best way to figure out what might be happening?
Is there a way in Xojo to know how many bytes are pushed to the browser? I know I have some complex pages with lots of interaction at the control level for such things as check box processing in a testing application.
I did recently bump the number of concurrent connections config setting in the Web server from 20 to 200 and it seems to have a positive effect but I have not seen any high loads yet. I do know that Xojo creates lots of loopback connections when looking at the Resource Monitor in Windows. They seem to spike at connection time for a new client then drop off. Could they tie up the Web Server connections?
Some users are on the LAN and some outside of the firewall. I don’t have any statistics or information where the users are when there is a problem. In general the Internet at the University has huge bandwidth capability. The IP of the server is exposed via the firewall to the Internet. There is no NAT so the IP of the server is a fully routable IP address.
I just checked a few minutes ago and there were 5-6 users and the performance looked OK.
Yes it does seem like a ghost that comes and goes.
If you are, don’t over allocate your cpu or memory resources between running vms.
For example, if you have a 2 core hyperthreading CPU you have a total of 4 logical processors. If you then allocate those 4 logical processors to 3 different VMs you’re essentially trying to run 3 threads on the same logical processor at the same time, which will cause slow downs as clashes occur.
Does your vm remain responsive during the “slow period” of web browsing? If it stays nice and responsive then it’s a problem inside the vm and not the vm configuration so you can ignore the last two posts.
If that doesnt shed any light on the problem, I would try it on IIS to rule out any conflicts with Abyss Web Server. If you can replicate the problem say on another machine using IIS then its the web app thats having the problem.
I have seen issues with SSL webApps https://forum.xojo.com/30191-webapp-struggling-under-load which were made much worse by a malfunctioning firewall. I think that when webApps are running SSL, if a connection ever times out then the app gets in a bad state. I think this is probably a bug in the framework, but I’ve not had time to build a test case.
The test case would be something like this
Xojo standalone webApp
running SSL
a few dozen clients
some sort of network degredation (artificial or natural) that causes lots of connections to time out.
Absolutely. The loopback connections you are seeing are the cgi script communicating with the app itself. On top of that, when a browser starts a connection to a new server or page, there’s typically a momentary surge in connections so the browser can get the initial page as quickly as possible. This number varies from browser to browser, but typically what I’ve seen is 4 to 6.
Keep in mind that a web app only actually runs on a single core (although it’s likely that the cgi script runs on a separate core from the app anyway). I would suggest that you not run as many apps as you have cores, otherwise the underlying OS may not have enough CPU to operate.
Now you talked about CPU and network speeds, but I didn’t see where you tested them from… the server itself or the virtual machine.
The CPU testing was from the VM not the hardware host. The hardware host has 32GB of RAM and only 16GB is allocated for two VM’s at 8GB each (both Windows) on the same box. One VM is the Xojo Web App machine and the other runs the open source PHPbb forum. The Xojo VM gets much more use than the other VM.
Today with the number of concurrent connections bumped up to 200 things are MUCH better. The loopback connections must have “stacked up” and even though they are short lived they had to disappear before the actual session doing the “real work” got any CPU. Essentially they were just blocking the “real” sessions until sufficient loopback connections disappeared.
The CPU usage at the VM is up some but averaging only 30%-40%. There is a nice “ebb and flow” of CPU use across the graph and it is fairly well distributed across all 4 virtual cores. I am measuring all of this with the Resource Monitor in Windows on the VM.
It probably makes sense that the CPU use is up because the connections doing the “real work” are now getting more CPU and not just blocked until the loopback connections die.
All of this seems to make sense because with only a few users the app is pretty responsive but seemed to “hit the wall” as more users logged on. There was not a linear slow down with more users.
I am still seeing 200-600 TCP connections “ebb and flow” with 25 or so users across two different Web apps. The network graph shows consistently about 1 Mbps with some occasional random spikes between 5-10 Mbps. Since I proved with SpeedTest.net the VM can do over 300 Mbps up and down that seems to be very low and the network is not a bottleneck. The hardware network card is a gigabit card but shared between two VM’s.
The Resource Monitor shows disc queue length to be typically about 0.001 seconds and overall disk bandwidth to be typically about 10 KB / sec. Over all memory use is up some but just slightly over 4GB which is only 50% of the allocated VM 8GB.
A few months ago “adjusted” the loopback connection timeout with a registry change mentioned by others here on this forum. That change maybe had the loopback connections die quick enough to just barely be able to limp along and explains why sometimes people just got kicked out with the “Gone Off Line” message when the session would just die.
All of this has come about because of the launch of the two new apps that are now generating more activity from more users … and sadly by me … no load testing.
Thanks for the help. If I find anything else interesting I will post it here for other so find.
If you open up the Abyss Console (typically port 9999) then go to Server Configuration > Parameters and you will find something called Maximum Simultaneous Requests. Set that to 200 and restart Abyss (it will prompt you).
I am not sure how I over looked this previously BUT the default was 20. I think that I also assumed that TCP Loopback Connections would NOT go through the Web server but it appears that they do.
I am considering bumping this up even higher than 200 since I am seeing more Loopback Connections but I want to let this run for a day or two first.
I will also ask Aprelium Support if a number like 400 would be reasonable and could the software handle it. The server must keep track of connections in some sort of array or table so having too many could be a bad thing. I will post here their answer.
A Xojo Web app cannot support very many active users in CGI mode. I see 10-15 as being max before all users suffer.
Why?
Xojo web framework is very chatty and all those events fire back requests to the server. Server has to respond to each one.
If (Users * NumRequestsPerMinute > MaxRequestsCpuCanHandlePerMinute) Then
WaitLonger()
End if
Things start slowing down… You probably want to start looking at load balancing your app. CGI is also additional overhead as each request requires the web server to spin up another perl interpreter. Expensive!
Has anyone tried ActivePerl on IIS (I’ve not tried xojo web so I dont know if it works) , or host header redirection to multiple stand alone instances on different ports (if more than one app is required per windows machine) eg this, see “IIS Reverse Proxy Config” at the end of that page.
My VPS osting is an Windows 2003 server using IIS and ActivePerl.
My program run but failed to show value of the textbox once the click was sent to the server. I dont know whats the problem so I decided to drop it and subscribe to xojo cloud.
The Xojo Web standalone mode is much faster than CGI. On a Windows server like you have you could run several instances of the standalone app. You could use Abyss web server to reverse proxy to the backend processes and thus you have load balancing.
My VPS osting is an Windows 2003 server using IIS and ActivePerl.
My program run but failed to show value of the textbox once the click was sent to the server. I dont know whats the problem so I decided to drop it and subscribe to xojo cloud.[/quote]
That would be quite difficult to troubleshoot if you dont have root/desktop access with the VPS, best stick with Xojo Cloud if you like things working out of the box so to speak.
I confirmed with Aprelium that setting the number of Maximum Simultaneous Request to a high value is OK so long as you have sufficient RAM. I now have one server set to 400 and the response time is MUCH better (actually 200 was much better than the default of 20). I am only using about 4GB of the configured 8GB. It still takes a few seconds to initially launch a larger app but I can now handle what is my typical max of concurrent users of 15-20 with good response time.
NOTE TO PHILLIP or anybody else that would like to help …
Can we get some notes or suggestions on how to configure Abyss as a reverse proxy and use stand alone apps?
