I have a web app hosted on Xojo Cloud. Currently, it is deployed with 2024 R3. I’ve been working daily to try and help with the app restarts. It occurred to me today that maybe the issue isn’t so much with my processes as it is with just landing on the login page (the initial app page). Here’s what I did:
So I didn’t disrupt the users of my app, I created an exact replica, just changing the URL, and deployed to XC
After launch, I opened my page’s memory monitor to see what’s happening. Initial memory 8MB
I quickly opened 4 new tabs and navigated to the replica’s URL. By the time I got to the 4th tab, that one was very slow to reach the login page
Viewed the page monitoring after the 4 new tabs, and memory shot up to 72MB
I kept the 4 tabs open, not interacting with any of them) and checked the memory every minute
Minutes 1 and 2, minimal change to memory: 70-74MB
Minute 3, memory significantly dropped to 20MB. And 30 minutes later, 19MB
Of note, the only thing I have in Session.Opening is a total sessions count for the memory page and a determination of if the session should open the memory page or not. My Session does contain a lot of properties (mostly strings/integers/booleans/arrays/colors but also a couple MySQLCommunityServer dbs), but none of these are filled or connected to in any Session.Opening/Shown events. The login page has no Opening or Shown event, so nothing is instantiated. This page just has textboxes, labels, and buttons. No db calls until the user interacts with a button (login, forgot password, etc)
This tells me:
Each new session needs memory to load/show stuff. Based on my replica app, each new session needs about 18MB to start it off, and after a few minutes, whatever resources it needed are garbaged/recycled and memory decreases
If there is a surge of folks reaching the initial page, memory will spike, watchdog says I see some unresponsive apps, so I will kill the app and make it restart. All users who were previously logged in and those who were just on the login page now get back to the login page, initial memory needs of 18MB per session happens, memory surges, watchdog barks, users kicked off, and so on
So my questions are:
Is this normal behavior of web apps? High initial memory upon session’s start but comes down after a couple minutes
How do you handle high traffic volumes? Do you use a stagger of some sort? Like if there are X number of sessions open/running/live, then you halt any new traffic and redirect to a page saying we will try to reconnect you to the login page in X number of seconds. Try to establish a session connection, and if still busy, go back to the redirect page? This seems like bad UX but was just a thought
I actually think some of the opened sessions (on other tabs) timed-out, Xojo closed the session, so that triggered memory release after some time.
I use a load balancer on Digitalocean.
One of my web apps has several thousand connections per day in December. I think the maximum is around 20.000 connections in a single day.
I use two 4GB 4cpu servers running my app behind a load balancer.
And I could easily add more servers in less than 5 minutes thanks to Lifeboat.
EDIT: It is actually 70.000 connections in a single day.
That makes sense. I remember noticing the hike in memory after I refreshed each tab (created a new session), and the number of current sessions was increased to 8 or 9, and then after 3 minutes (probably the timeout to end the previous sessions), the memory came back down as did the session count
Correct me if I’m wrong, but is DigitalOcean your app’s host? Which would mean, for me, switching from Xojo Cloud hosting to DO? If that’s the case, I’m not sure that’s for me at this time. I do like the ease with deploying to XC and maybe not wanting to bring in extra challenging steps as I’m still learning more about web apps in general. I always appreciate your insights on things! If DO is a different host, I will keep that idea in my back pocket for future me
XC doesn’t support load balancers, right? Are there any XC-friendly options to help with many app hits in a short period?
Xojo cloud has a load balancer built in. Because Xojo apps are single thread, the load balancer spins up the same number of apps as the number of vCPUs that you have on your server.
Good to know. So, I am on the Small Xojo Cloud server. Do you think upgrading to the medium server will help with the sluggishness/app restarts when a lot of users go to log in at nearly the same time? I see the medium server has 4 vCPUs compared to 2 in small
It might but I think it’s more important that you define what “a lot” means…
And you should also think about your app and what it means if more than one instance of your app is running at the same time. While a user’s session will always go to the same instance, if the instance is crashing, users will effectively get logged out. Personally I’d focus on figuring out why that’s happening first.
Don’t laugh, but “a lot” from what I noticed was 22 logins at nearly the same time. The other night, a class of students all had to do an assignment in the app while in class. They all went to the app at the same time, and the app restarted. I don’t know if all actually logged in. Some might’ve just gotten to the login page when it happened, which got me thinking memory might be spiking simply when the session is created
I’ve been trimming things down a lot lately. With the other night’s scenario, I thought it was something on the home page, right after logging in. I cut down the number of db calls on the initial show of the home page from 6 to 3. The remaining db calls aren’t hefty either (one query each with the “heftiest” pulling in a limit of 10 rows, also the tables are indexed with only the intended columns of interest), and they’re now staggered. They’re even staggered using a random jitter so even if 2 log in at the exact same time, their db calls will be a couple ms apart. I’ve even moved things that were originally created (in-mem db) from the session opening to later in the app when that db is needed to try and reduce unnecessary overhead. I still have other things on my list to check and make sure heavy objects aren’t getting initialized too prematurely, but my findings seem to be the culprit is hiding when the session is created and/or on the home page