I have a WebApp which I use for collecting survey data, which has worked fine for a few years now, handling moderate loads of between 1 and 50 simultaneous connections).
Feeling pretty comfortable with it, I decided to try updating it to 2016R1, and it seemed OK in debug tests. Foolishly (in hindsight) I put this into production, and it immediately fell apart : CPU spiking to 100% and barely handling a dozen connections at once.
I took samples of the app under load and killed it. I recompiled in an older version of Xojo (2015 R4.1) but saw basically the same issues. I need to check my notes, but I think the earlier versions that handled high load better was possibly made with 2014 R2.1?
Anyone ran into similar issues: recompiling a web app with newer version of the IDE is giving much worse load-handling ability?
I ran my “standardized*” test against the 2015R3.1 build, and although it appeared to do a little better, it still fell apart, and is now sitting with zero live connections but using 70% cpu and a very similar stack trace.
I’m pretty sure this used to work just fine, but then something changed. Of course, I changed a few things at once:
New version of Xojo
Using SSL rather than not
New version of OS X (10.9 -> 10.10)
Your comment about SecureSockets is interesting. I am not using any of them in the app, but the whole web app is invoked with the “–secureport=xxxx” option.
So maybe the problem is not with the Xojo IDE version but running using SSL? I will do more tests…
As for VirtualVolume: I’m not using them in the app at all, so not clear why that ends up in the stack trace. Maybe WebFile uses them internally?
Where I ask a classroom of 45 people on computers to all hit the app at once.
I like using haproxy to handle SSL, and it’s also very easy to setup load balancing with it. I can’t remember who it was, but someone on the forum wrote up how to set it up. A search will probably turn it up.
[quote]Your comment about SecureSockets is interesting. I am not using any of them in the app, but the whole web app is invoked with the “–secureport=xxxx” option.
So maybe the problem is not with the Xojo IDE version but running using SSL? I will do more tests…[/quote]
All sockets in web apps are derived from SSLSocket, some with Secure=True, some with False.
Nope. My guess is this is the closest offset it was able to find in Folderitem.
That’s asking for trouble without a load balancer of some kind. When we did the first XDC app, we got hit by ~1000 users in the first 3 minutes and it brought the whole app down. Last year we used a load balancer to several instances on the same server and the problem was abated.
Offloading SSL makes a big difference and balancing multiple instances makes an even bigger one. HAproxy is free and easy to set up and you don’t need any additional hardware - just maximizing the hardware you have.
Well, the thing is I’ve used this same app for a couple years and it’s always done OK with handling 40-50 users - the app itself degraded gracefully, just becoming slower, but speeding up once the load dropped off.
Behavior now is not the same : the app actually stops serving content and seems to get stuck with running threads even after all connections are terminated.
I’ll bet you are surpassing the maximum of what you can handle now that you turned SSL on. I’ll bet turning SSL back off (or handing it off to HAProxy) returns you to your prior performance. If there is absolutely no free overhead, the actual process of winding down dead sessions and threads itself could be impacted.
I’ve just discovered something really odd. Testing the app using only a few (from 1 to 4) simultaneous connections and watching Activity monitor, every time there is data being transmitted, mds_stores starts burning cpu, spiking to 50% or higher. That doesn’t seem right to me, and I wonder if that could be affecting performance.
I do have spotlight indexing turned on, and I am using two kinds of logging in the web app:
the app is launched with the --logging flag
I’m using a simple logging system in the app:
// does rudimentary logging
static logFile as FolderItem
static tos as TextOutputStream
if logFile = nil then
logFile = GetSpecialFolder("logs", "events.log")
tos = TextOutputStream.Append(logFile)
dim d as new date
tos.WriteLine("[" + d.SQLDateTime + "]" + " " + msg)
tos.Flush // so log updates in real time
I wonder if either of these logging systems is getting caught up with Spotlight indexing to cause a big slowdown? I’ll try some tests to narrow it down…
Update: though I’m not able to do the full load test with 45 people, I was able to do a small test running 4 simultaneous sessions (with 3 of them over a slow Tor network link to simulate low bandwidth / high latency connections). Even with this small test it’s apparent that performance is much better - the web apps load much more quickly in the browser AND the cpu usage of the web app on the server is extremely low (hovering below 5% even when serving 4 sessions).
So this is very promising, and suggests that the combination of SpotlightIndexing and SSL was causing a big performance hit.
Travis, I discovered a third possible problem: My server is behind a Dual WAN configuration, and I had a mistake in the firewall settings, which meant that if the server tried to make a socket to contact the client, sometimes it would go out over WAN IP #1, sometimes over WAN IP #2. I wonder if this behavior might have caused an unusually high # of sockets to get stuck and fail to connect, which could have led to additional performance problems?