Web app constantly restarting throughout the day by itself

Targets Web
1

Guys, I need help (this post is long… sorry, just want to be very concise). I’ve been trying to track down and solve this problem. Have been working on it daily, pushing through new updates with the changes. Everything seems to be good to go for a couple of hours, then BOOM! Users are getting kicked out of the app. And when I look at the memory report tool, it looks as though the app restarted on its own, making all current users to be taken back to the login page. I have the app email me when there is a user logging in when the user is already logged in. If the app crashed on them, their login details never successfully saves as “logged out”, and they go an log back in, the app sees it’s a dup, and sends me an email.

Today’s update was this afternoon. A couple of hours went by. I checked the memory report before shutting down for the day (about 70MB with 9 current sessions and 33 total sessions). The Xojo statistics window showed 45% memory. Not a ton of users at any given time, and memory per Xojo was not way up there. I go downstairs to workout and am checking my emails, and all of a sudden, I see a dozen or so emails from the app with users who had to log back in. I check the memory report page, 15MB, 1 current session, 1 total session (this was me checking the report)

I posted a similar topic last week, but I thought of making a new post with as many details as I can to see if you can help

First, this seems to be a new issue as of the past 3 weeks. I thank Greg for his explanation on circular reference in the above post, but if that were the case, would not the issue have been going on for much longer? Yes, on occasion, some users would have to re-log back in, but the app never restarted itself, kicking everyone out. In fact, weeks would go by, I would check the memory report, and once I saw that or the Xojo statistics memory getting up there, I would manually restart the server so all apps would restart, and the memory would come back down. When I would do this, the memory report would show hundreds of total users from the last time the app/server was restarted, not only 33 and then minutes later 1 without human intervention. With all that said, I am not certain this is a circular reference issue as I would have expected this to be happening over the past 2 years, but it’s only in the past 3 weeks that I notice this auto-restart

Here is a basic timeline of the changes I made over the past couple weeks. Note, after each update, the app was fine for about an hour or two, and then the auto-restarts/kickout users began:

  • April 22 (this is when the problems began): I updated Xojo to 2025 R1 (previously was using 2024 R3). Added a WebToolbar to all webpages. Added an accessibility feature that puts a small colored border around controls the mouse is hovering over
  • April 29: I removed that accessibility feature as after looking more into it, it could have been contributing to memory leaks as it was looping through all the controls on a page
  • April 30: I removed any ContainerControls I had in the app. I did this after reading Greg’s comment in the linked post above that ContainerControls can add to circular reference issues. I only had a few of these, and they were merely there to declutter the pages for myself. They did have references back to their parent pages. Taking the controls/properties/methods off of the containers and moving right onto the page wasn’t a huge task… but it made no difference with the app auto-restarts :frowning:
  • May 5: Please don’t scold me on this one… I have not been using Try…Catch or UnhandledException in my apps. The May 5 update added in some Try…Catch on some pages I thought to be problematic and UnhandledException, but only in the Session class
  • May 6 (today): The auto-restarts were still occurring. This got me thinking… duh, I never added UnhandledException to the App level. If the app crashes from some other area I missed, the app-level will catch that and prevent the crash. Pushed through the update today, feeling very confident. Two hours later, crash, auto-restart, users mad

Here is my code in the App.UnhandledException:

If error <> Nil Then
  Var type As String = Introspection.GetType(error).Name
  
  Var msg As String = "App-Level Exception at " + DateTime.Now.ToString + EndOfLine + EndOfLine
  msg = msg + "Type: " + Introspection.GetType(error).Name + EndOfLine
  msg = msg + "Message: " + error.Message + EndOfLine + EndOfLine
  msg = msg + "Stack:" + EndOfLine + String.FromArray(error.Stack, EndOfLine)
  
  EmailMeError(msg)
  
End If

Return True

My understanding after reading the docs that if Return True, the app should not crash, rather just continue uninterrupted. The code is identical in the Session.UnhandledException, except the msg begins with “Session-Level Exception”

If you notice here, the app should be sending me an email. It worked in testing, but it is not sending the emails at the app level, which I was hoping for. The stack showed me a nice path down to the method or button where the error occurred, along with the type of exception. I thought this would be the end-all to the issue. It wouldn’t crash the app for all users, and it would guide me on where to focus attention. It is sending me an email on the Session level, but this was a Nil issue in the UserDisconnected, which I corrected but have yet to push out. I also have similar EmailMe functions on specific page methods. None of these emails are being sent. That could be because I added Try…Catch in the wrong places

Should I do any of these?

  • Revert back to using 2024 R3. I hesitate ever doing this as I always worry about any new/updated features that won’t work with older versions. I usually see that message pop up when opening a project made on a newer version and get scared
  • Should I not use both App.UnhandledException and Session.UnhandledException? Should I use only one, and if so, which one? Guessing at least the App level
  • Not mentioned in my timeline, but back in December, I added Self.Quit in the Session.UserDisconnected to hopefully clean up any leftover things to free up memory. Should I omit this? The auto-restart issue didn’t occur from December to April, so I don’t think this is a problem

I also emailed @Jason_Parsley earlier tonight to ask if he can take a look at my server to see if anything looks off (using up too much disk space… I think my small server is good for 5GB, and some of my database tables are chunky… One is 1.4GB with just over 1 million rows. Other tables are 25MB or less). Should I remove old, unneeded data to free up disk space? Could that be the cause of my app woes?

2

There are differences in crashing that you need to consider.

First, a crash that’s due to an unhandled exception is a “soft” crash. That is, something caused by an issue with your code (or the Xojo framework) that resulted in an exception that wasn’t handled.

Next, there’s “hard” crashes. These are things you can’t do anything about. Framework Assertions, memory issues, segmentation faults fall in this category.

The fact that you’re running on Xojo cloud may point to another possibility however. Web apps that run on Xojo Cloud are set up to run as daemons, and within that system, they use what’s called a WatchDog. Basically, systemd sends a message to your app periodically to see if it’s still responding. If it doesn’t get a response in a certain amount of time, it will automatically kill and relaunch the app. This is meant to help with apps that get deadlocked. The flip side is that you need to be sure that you never ever run something in a tight loop without yielding. If the process you’re running is going to take a long time (more than a second or two) put it in a WebThread that periodically calls its Sleep method and send the results when that thread is done.

1 Like
3

Thanks Greg. I always appreciate your input and explanations on things. That’s good to know about the different types of crashes. Learned something new today

As for the need/use of threads. I do have a spot in the app which already uses a WebThread. It’s a section which goes through several calculations that normally would take more than a second. I added the threads about a year ago, and nothing else comes to mind that would take a considerable amount of time to process. So the watchdog ping should still be seeing a responsive app

The only main contributors that stand out to me as new changes over the past 3 weeks is the upgrade in Xojo version and the addition of the WebToolbar. I’ve actually been using the toolbar in the same app but only on the mobile pages. The new addition was just to the desktop pages. I’m sure a ton of other Xojo users already switched to 2025r1 and are not having these kinds of issues with their web apps. At least I haven’t seen any chatter about it on here

Unless anyone else chimes in with an “ah-ha”, tomorrow my plan is to build with 2024r3 and see how that goes. I’m running out of ideas

4

Both. In the session you have context like knowing which user it is.

In both cases you should at least log to a text file and later inspect the exceptions.

5

Greg, do you think WatchDog would be kicking in for the recent issues with Xojo Web apps not responding? I think you might be on to something there.

I have also been investigating this issue. When last we saw it occur, local requests to the server were failing to respond. We tried to curl http://127.0.0.1:9000 and got no response from the instance. The process was still running though, systemctl status theapp showed “active”.

I do like the idea of WatchDog, but because implementing it requires inserting code into the app itself I have been struggling with the idea of adding it to Lifeboat. I really appreciate the time you put into the example you posted …somewhere on here.

6

Ok good, I will keep both

About an hour ago, I deployed the app in 2024 R3. So far, okay, but it’s been only an hour. I am keeping the memory report screen open to monitor throughout the day.

Jason also got back to me last night and then today. He will be checking my server details to see if anything is off. I asked to also check the total disk space usage as I am a tad concerned with that one table at 1.4GB

7

Not trying to be redundant in my replies but adding Sentry to report your issues could make things easier.

There is an option in Sentry to save errors to disk before sending them. This allows to re-send errors when the app restarts if the socket was killed before it ever got a chance to send.

//Before sending an exception:
sentry.Options.save_before_sending = True

//Now send the exception
Sentry.SubmitException(error, "", "", Xojo_Sentry.errorLevel.error)

And in App.Opening event

//First initialise Sentry
Sentry = SentryController.GetInstance( "http://THE_SENTRY_DSN" )

//Then send offline exceptions
Sentry.SendOfflineExceptions

In the App.Closed event I also have this code:

//Send a warning exception to Sentry each time the app is closed
Dim error As new ClosingException // ClosingException is a class I added to my project. Its super is RuntimeException
error.Message = CurrentMethodName

sentry.Options.save_before_sending = True

try
  raise error
Catch err
  app.sentry.SubmitException(err, CurrentMethodName, "", Xojo_Sentry.errorLevel.warning) 
end try


//The following is optional. It allows sending a notification to the Pushover app each time the app closes
// https://pushover.net
#if not DebugBuild
  var http As new URLConnection
  Dim sname As String = System.EnvironmentVariable("SNAME")
  call http.SendSync("POST", "https://api.pushover.net/1/messages.json?token=" + MY_PUSHOVER_TOKEN + "&user=" + MY_PUSHOVER_USERID + "&message=WebApp%20Closed%20" + sname)
#endif
8

Sentry is also a good idea.
We have a video about that:

In general on the server your app runs quite blind. As developer, you have to log things as otherwise it just crashes and you have no clue.

See also our BugReporterKit example code coming with MBS Xojo Plugins and the classes like SignalHandlerMBS and GlobalExceptionHandlerMBS to catch more exceptions.

1 Like
9

Just as an fyi, Xojo confirmed my issue; it’s related to MySQL databases.

I made some adjustments to my database code so my running web application is much less “aggressive” in the number of calls it makes to ‘MySQLCommunityServer.Connect’:

This has (so far!) greatly improved the stability of my app.

There is a confirmed issue(s) on the Xojo framework side of things, but things are still being analyzed by Xojo team so I don’t want to speculate until I hear more.

I recently had a loss in my family, so I haven’t been online much; wanted to share that my issues seemed very much to database connections in case it was relevant to you, @Ryan_Hartz

Take care,
Anthony

3 Likes
10

My condolences to you and your family

Thank you for sharing your experience. Are you using 2025 R1+? I do make a lot of calls to MySQL, and I know they can be minimized in some spots. Perhaps this could be an issue with my app as well

11

Thanks Jeremie. Yes, I will be implementing Sentry into the web app in time

Just a mid-day update. I rebuilt two of my web apps using 2024 R3 that were having issues in 2025 R1. It’s been 4 hours, and there has been no app restarts. I am continuing to monitor the memory report pages throughout the day

Jason also got back to me via email. There is a private Issue already open that could be related to the issue I’ve been having.

2 Likes