Memory usage of an app

I am currently writing on a logfile parser for a computer game. The log consists of the events happening to the game server like players connecting or disconnecting or shooting each other. The parser goes through the log, analyses each line and creates an object for each player found in the log. This player is added to an array of player objects. As the logfiles can be huge (hundreds of megs) and there are hundreds of thousands of events stored, the app quickly uses vast amounts of memory. For like 9000 players with all their log events the app uses about 120 MB of memory. This is not the problem though.

The problem is a different one: you can reparse the logfile and the list and everything gets populated again. But reparsing makes the applications memory hunger grow. So after reparsing the same logfile again, the amount of memory used grows up to about 200 MBs. Of course I am redimming the player array before repopulating it, so there should be no reference to the Player-objects anymore. You can even see it that when I start reparsing, the apps memory shrinks about 40 - 5 MBs and then grows again.

To me it appears as if some objects are not destroyed even if there is no reference pointing to them anymore. Is there a good methodology how to find out were my app is leaking memory? I hoped I could do this using the debuggers runtime view, but it only shows a mess of thousands of unordered objects… :confused:

Are you using this to clear the arrays?

ReDim playerArray(-1)

I was not accurate enough in my first description. Actually the player-objects are stored in a dictionary which is part of the ApplicationController class. When I need to iterate I use the values()-method of the dictionary. Anyway, each time the file is reparsed, a new instance of the dictionary is created. So the memory used after reparsing should be nearly identical than before, nevertheless it increases.

Try using Runtime.IterateObjects to see what objects actually exist : you can look at Runtime.ObjecdtRefs to see how many references there are to a given object. A common bug that leaks memory is to have more than one reference, or to have circular references.

See http://documentation.xojo.com/index.php/Runtime

I just posted a thread on memory usage a couple weeks ago. I was doing some tests of an app where I do a large amount of image updates to canvases - up to about 10 times a second.

In Windows I was seeing my memory start out at around 40 Megabytes but then it would gradually grow and grow and grow. I didn’t have any circular references and all my objects were being cleared from what I could tell. I took the advice here of some and checked the number of runtime objects and sure enough - it was steady but memory kept growing.

Suddenly, around 400 Megabytes of memory, Windows cleared everything up and memory usage dropped back to to 40 Megabytes or so and began climbing again. I let it run for several days and it did this consistently. So it seems like Windows may take some amount of time before it actually finishes all the garbage collection and releases memory. Of course, if it needed it, I am sure it would clean it faster.

I did find one spot in the app where my object count grows and will continue to grow the more I do a particular action. However, it’s unlikely that this particular action (going into the setup menu for the app) is going to be performed over and over and over and over and over again…So that’s one that I can’t find that I am ignoring for now…

One of the problems is that it’s not easy to define “memory used” - for example, there’s the number given by Runtime.MemoryUsed. Windows Task Manager shows another number, but in Windows 10 I’m finding the most useful number is to go to TaskManager, switch to the Details tab, then right-click the header and select the Memory (private working set) column, as that seems to correspond more closely to reality. Another good item to watch is “Handles” as that could show a leak of non-Xojo objects.

@Natascha Chrobok
Have you considered putting the data into a SQLite database? You could even do one completely in memory and it may give you better memory and access performance.

It was circular referencing… I wrote a routine that cleans up. Now I am fine – thanks for the hint.

@Greg O’Lone

I have parts of the app in a sqlite database. I already considered using a in memory database and transform the in game classes to tables in the database. But it would mean rewriting a lot of code. So far the performance is acceptable.