Web1 - Can anyone tell me what this error log code means?

Tim_Seyfarth · October 27, 2023, 9:49pm

Hello All,

Does anyone know what this means? The web1 app crashed it looks like,but I do not know why.

Oct 27 14:28:13 axcys systemd[1]: axcys-embedded-viewer.service: Main process exited, code=killed, status=15/TERM

In the App.close and Unhandled Exceptions events I have code to tell me what’s going on - nothing shows up in the logs tho. I also have code in the session.close and session.unhandledExceptions

App.Close:

System.DebugLog( "APP - SHUT DOWN in:  " + CurrentMethodName  + " App Closing" )
System.Log( System.LogLevelError, " APP - SHUT DOWN in:  " + CurrentMethodName  + " App Closing" )


Dim Flds() As String = Array("APP - SHUT DOWN",  CurrentMethodName , "")
Logs.AddFields Flds
Logs.LogFileAppend( Logs.K_Except_App_Err )


Try
  Raise New OutOfBoundsException
  
Catch e as OutOfBoundsException
  Dim st() as String = e.stack
  Dim s As String =  cstr(st())
  
  
  // System.Log( System.LogLevelError, CurrentMethodName.ToText + "  e.stack = " + st() )
  System.DebugLog( CurrentMethodName.ToText + "  e.stack = " + s )
  
  // write the stack to a log for runtime use
  Logs.AddFields st()
  Logs.LogFileAppend(Logs.K_Except_App_Err)
End Try


Try 
  Raise New RuntimeException
  
Catch e As RuntimeException
  System.Log( System.LogLevelError,  CurrentMethodName + " RuntimeException Reason = " + e.Reason + " RuntimeException Err Number = " + cstr(e.ErrorNumber) + " RuntimeException Message= " + e.Message )
End Try

App.Unhandled Exceptions:


System.DebugLog( "Unknown exception in: " + CurrentMethodName  + " no error type available  " + CStr( error.Type) + "  Err Msg = " + Error.Message )
System.Log(  System.LogLevelError , " Unknown exception in: " + CurrentMethodName  + " no error type available  " + CStr( error.Type) + "  Err Msg = " + Error.Message )
Return False

None of these makes it to a log entry however…

Ideas?
Thanks,
Tim

Rick_Araujo · October 27, 2023, 10:54pm

Maybe an OOM Kill

https://www.kernel.org/doc/gorman/html/understand/understand016.html

Tim_Seyfarth · October 27, 2023, 11:51pm

Hi Rick.

Thank you for your response. This is really weird since it just started happening. None of the apps shutdown code is executed (it appears). And happens in seemingly different time intervals. As far as memory, this and other of my apps are running on a Pi4 with 8GB RAM, using about 500MB. Only about 6% of the CPU is in constant use too…

Is there a way to catch whatever is causing this app to quit? Systemd restarts it, pretty much immediately, but since the it restarts, the user has to log back on… a real PIA! Or are there other tests or traps I can institute to find the cause of the problem?

Thanks.
Tim

Rick_Araujo · October 28, 2023, 12:21am

Not sure what could cause, you will need to research. Non-responding loops, memory leaks, I don’t know… The system may detect a bad process, send a TERM and later a KILL.

Tim_Seyfarth · October 28, 2023, 12:38am

Well, that’s more info than I had a few minutes ago!
Thank you! I will look into those things and probably more…

Tim

Rick_Araujo · October 28, 2023, 7:04am

Check the swap space of your system, if you can increase it to proper values you may minimize the problem, but if you have a leak, it may delay the problem but not avoid it.

Increasing the system RAM is also an option if your app is ok but just demanding RAM at peak times.

Have in mind that the process being killed isn’t always the process that caused the OOM, it may be just the chosen one to be killed based on those rules I listed.

DerkJ · October 29, 2023, 11:17am

This happens if systemd stops or restarts a service it does not come from your app.

Tim_Seyfarth · October 29, 2023, 10:57pm

Hi DerkJ

How to find out if systemd is responsible, and more importantly why?

Tim

DerkJ · October 30, 2023, 1:19pm

This part
systemd[1]:

Rick_Araujo · October 30, 2023, 3:40pm

This just says “the service axcys-embedded-viewer.service, that I, systemd started, just exited, due to a kill (no cause explained) and the status was 15, it means, it received a SIGTERM”

An OOM could cause the same.

A sudo pkill -SIGTERM axcys-embedded-viewer should do the same too.

Maybe a system shutdown, because a shutdown will send “terminate your work and exit please” to everything too.

DerkJ · October 30, 2023, 5:56pm

Killing the service does the same yes, but it looks like the TS did not do that…

Tim_Seyfarth · October 30, 2023, 8:32pm

Thank you guys.

The system did not reboot. Only the app.

Tim

Rick_Araujo · October 30, 2023, 8:39pm

If you could replicate the problem in a sample with instructions on how to reproduce the condition in a report, people could try to find out what’s going on. Sadly, as it is happening with the unsupported Web1 Xojo may not help if it needs a framework update.

Tim_Seyfarth · October 30, 2023, 8:47pm

Hi Rick,

Creating a sample would be challenging since the Web App is large, with lots of elements and I don’t think it would easily replicate the problem. However, the problem appears to be consistent. These RPi’s are V4 with 8Gig RAM.

How to find out how large the swap is, and how to change the settings?

Here is what webmin says:

Thanks,
Tim

Rick_Araujo · October 30, 2023, 8:51pm

Your current workload is not “asking” for extra memory. The reason may be something else or you may have a leak and in few days it may occur.

Being on ARM linux would be another important info for Xojo if you had a sample.

Inspecting swap should be done in a terminal session with root (sudo) access.

I guess this LPIC still current about swap space:

Rick_Araujo · October 30, 2023, 9:08pm

Is this instance in a shared server? Any chance of an admin of some other management process taking it down?

Tim_Seyfarth · October 30, 2023, 11:49pm

It is a stand alone Pi. There are other apps running, that we built, but this is the only Web App. The others are console to do special stuff.

It is re-creatable, happens within a few minutes of use - as in just watching the screen.

This is something that just started recently. I could go back and try earlier releases to see if they are also a problem - but have not gotten any negative feedback either.

What is interesting and frustrating is that the Session.Close nor the App.Close /Stop/Unhandled exceptions are never raised.

One thing that I did notice is there are System logs referring to Fonts.

Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: line 5: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: line 6: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: line 9: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/10-hinting-slight.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/10-hinting-slight.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/10-hinting-slight.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/10-scale-bitmap-fonts.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/10-scale-bitmap-fonts.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/10-scale-bitmap-fonts.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig error: "/etc/fonts/conf.d/10-scale-bitmap-fonts.conf", line 76: non-double matrix element
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig error: "/etc/fonts/conf.d/10-scale-bitmap-fonts.conf", line 76: non-double matrix element
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/10-scale-bitmap-fonts.conf", line 84: saw unknown, expected number
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/11-lcdfilter-default.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/11-lcdfilter-default.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/11-lcdfilter-default.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/20-unhint-small-vera.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/20-unhint-small-vera.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/20-unhint-small-vera.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/30-metric-aliases.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/30-metric-aliases.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/30-metric-aliases.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/40-nonlatin.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/40-nonlatin.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/40-nonlatin.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/45-generic.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/45-generic.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/45-generic.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/45-latin.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/45-latin.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/45-latin.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/49-sansserif.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/49-sansserif.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/49-sansserif.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/51-local.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/51-local.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/51-local.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/60-generic.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/60-generic.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/60-generic.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/60-latin.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/60-latin.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/60-latin.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/65-fonts-persian.conf", line 34: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/65-fonts-persian.conf", line 35: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/65-nonlatin.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/65-nonlatin.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/65-nonlatin.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/69-unifont.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/69-unifont.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/70-no-bitmaps.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/70-no-bitmaps.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/70-no-bitmaps.conf", line 8: unknown element "description"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/80-delicious.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/80-delicious.conf", line 5: unknown element "its:translateRule"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/90-synthetic.conf", line 4: unknown element "its:rules"
Oct 30 19:46:16 axcys AxcysEmbeddedViewer[12514]: Fontconfig warning: "/etc/fonts/conf.d/90-synthetic.conf", line 5: unknown element "its:translateRule"

Also, this error which I am looking into but do not yet know what it pertains to or means, other than the obvious.

Oct 30 19:48:29 axcys AxcysEmbeddedViewer[12514]: Error 116 of type 33554432: WriteFText: Operation canceled!

Tim

Rick_Araujo · October 31, 2023, 7:19am

After removing all noticeable errors, the only non-obvious thing I can think is what I said, some older than AxcysEmbeddedViewer task having processing peaks requesting loads of memory that we did not see in your snapshot. The memory management prioritize some older tasks, and may select newer tasks (AxcysEmbeddedViewer? Any user task consuming a good quantity of RAM started after the older ones needing memory) to kill (gracefully, terminate), trying to keep the long time running tasks alive.

Good luck in your research.

Tim_Seyfarth · October 31, 2023, 6:40pm

Thanks again Rick.

I expect that we will adjust the swap space.

Tim

Daniel_Mullins · October 31, 2023, 11:55pm

Adding swap to a Pi is not the answer. You will be swapping to the sd-card (assuming your not using a nvme hat/compute module), both reducing speed, and wearing out the sd-card.