Memory Leak?

Martin_T · May 2, 2018, 12:35pm

Hello,

I have a method that goes through a 670 MB text file line by line. The method searches for specific letter combinations at the beginning of a line (4 characters long) and inserts their line number into a Dictionary, depending on the type. Even after I have opened the program for 20 minutes, the RAM consumption does not decrease and remains constant at 488.5 MB. And this, although I have emptied and released all dictionaries (=Nil). Why does the memory consumption remain constantly so high, although I release the memory internally?

file = TextInputStream.Open(GetOpenFolderItem("txt")) fileLength = file.Length

[code]Private Sub RunAnalyzeThread(sender As Thread)
myDict1 = New Dictionary
myDict2 = New Dictionary
myDict3 = New Dictionary
myDict4 = New Dictionary
myDict5 = New Dictionary
myDict6 = New Dictionary
myDict7 = New Dictionary

Const chunkSize = 5000

Dim mCurrentLineNumber As Integer
Dim residual, mCurrentLine As String
Dim tArray() As String
Dim result As String

While Not file.EOF
Dim bytesToRead As Integer = Min(fileLength - file.PositionB, chunkSize)

result = file.Read(bytesToRead)
ReDim tArray(-1)
tArray = result.Split(&uA)

' insert residual text of incomplete line from previous read
tArray(0) = residual + tArray(0)
Dim nLines As Integer = tArray.Ubound - 1
' save incomplete line at end of current chunk for next read
residual = tArray(nLines + 1)

For i As Integer = 0 To nLines
  ' Process complete lines of text here
  mCurrentLine = tArray(i)

  Select Case mCurrentLine.Left(4)
  Case "AAAA"
    myDict1.Value(mCurrentLineNumber) = Nil
  Case "BBBB"
    myDict2.Value(mCurrentLineNumber) = Nil
  Case "CCCC"
    myDict3.Value(mCurrentLineNumber) = Nil
  Case "DDDD"
    myDict4.Value(mCurrentLineNumber) = Nil
  Case "EEEE"
    myDict5.Value(mCurrentLineNumber) = Nil
  Case "FFFF"
    myDict6.Value(mCurrentLineNumber) = Nil
  Case "GGGG"
    myDict7.Value(mCurrentLineNumber) = Nil
  End Select

  mCurrentLineNumber = mCurrentLineNumber + 1
Next

' If file doesn't end with an EndOfLine, then the residual string
' will contain the final partial line which still needs to be processed.
If residual <> "" Then
  ' Process final partial line of text
End If

Wend

ReDim tArray(-1)
result = “”
residual = “”
mCurrentLine = “”
file.Close
file = Nil

myDict1.Clear
myDict2.Clear
myDict3.Clear
myDict4.Clear
myDict5.Clear
myDict6.Clear
myDict7.Clear

myDict1 = Nil
myDict2 = Nil
myDict3 = Nil
myDict4 = Nil
myDict5 = Nil
myDict6 = Nil
myDict7 = Nil
End Sub
[/code]

Christian_Schmitz · May 2, 2018, 12:44pm

Do you have a test project for this?
and are you sure this compiles?

Martin_T · May 2, 2018, 1:03pm

[quote=385818:@Christian Schmitz]Do you have a test project for this?
and are you sure this compiles?[/quote]
Of course, I just made a test project. You can see it here: Test-Projekt
The working memory remains constant at approx. 109 MB after analysis.

Christian_Schmitz · May 2, 2018, 1:15pm

I think the thread and class may never be released as they depend on each other.
Could that be?

AddHandler causes a new reference.

Martin_T · May 2, 2018, 1:22pm

[quote=385826:@Christian Schmitz]I think the thread and class may never be released as they depend on each other.
Could that be?[/quote]
What do you mean?

If I add this at the end of “RunAnalyzeThread”

Xojo.Core.Timer.CallLater(0, AddressOf Clean)

and add the new method “Clean”

AnalyzeThread.Kill RemoveHandler AnalyzeThread.Run, AddressOf RunAnalyzeThread AnalyzeThread = Nil
nothing happens. The memory consumption remains on top.

Christian_Schmitz · May 2, 2018, 1:27pm

You have a circular reference.

e.g. you could use WeakAddressOf here to make it a weak connection to the run handler.

Please remove the AnalyzeThread.Kill call here and setting to nil doesn’t help here, too.

Christian_Schmitz · May 2, 2018, 1:31pm

When RunAnalyzeThread finishes, you could set the AnalyzeThread property to nil, so the reference is gone there.

Martin_T · May 2, 2018, 1:36pm

Thank you for the note with WeakAddressOf. That’s what I would have tried next.
However, the RAM still remains high. Something seems to be not working with the memory management of Xojo…!?

Christian_Schmitz · May 2, 2018, 1:39pm

It’s maybe just 1 MB which may be leaked. Don’t worry.

Christian_Schmitz · May 2, 2018, 1:39pm

The memory management is fine. The last years it was always the developer at fault.

Martin_T · May 2, 2018, 1:42pm

[quote=385832:@Christian Schmitz]It’s maybe just 1 MB which may be leaked. Don’t worry.
The memory management is fine. The last years it was always the developer at fault.[/quote]
OK, and that means that there is a bug in my code somewhere, or that I just have to accept the memory allocation because it is normal (and the memory is not freed)?

Christian_Schmitz · May 2, 2018, 1:45pm

Thread references class
Class references thread

As both references are staying, the memory can’t be released.
But with weak reference and setting back thread reference to nil on the end of the run method, it should be away.

Martin_T · May 2, 2018, 1:53pm

If I’m right at the end of the method from above add sender = Nil or AnalyzeThread = Nil does not change. I didn’t either, if I sender/AnalyzeThread = Nil as described above into a Clean method by inserting Xojo.Core.Timer.CallLater(0, AddressOf Clean) at the end of RunAnalyzeThread.

Or could it be that the dictionaries are not released properly?

Thank you for your time, patience and input.

Beatrix_Willius · May 2, 2018, 3:50pm

You can analyse your app with Instruments. Select leaks. I did a quick run and the memory leaks are very small. A couple of hundred bytes.

The question is now: is the value from the Activity Viewer correct? This doesn’t go down. Isn’t there a function System.MemoryUsed or similar that you can check?

Martin_T · May 2, 2018, 4:05pm

[quote=385854:@Beatrix Willius]You can analyse your app with Instruments. Select leaks. I did a quick run and the memory leaks are very small. A couple of hundred bytes.

The question is now: is the value from the Activity Viewer correct? This doesn’t go down. Isn’t there a function System.MemoryUsed or similar that you can check?[/quote]
Hello, Beatrix,

with Instruments, we looked at the matter yesterday and observed exactly the same thing as you.

The note with the function Runtime.MemoryUsed is very interesting. If I query the value before the deletion calls of the dictionaries, it moves around 91 MB (approximately the display of the task manager). If I query the value again after deleting the dictionaries, it shows approx. 21 MB. This means that the program seems to really free up memory, but the display in the task manager of macOS is not updated. On the other hand, it is very strange that the task manager still displays the wrong value after 20 minutes. Very interesting, at least if the value returned is correct!

Carlo_Rubini · May 3, 2018, 6:19am

Excuse me if my question does not make sense: is the job done in a different window, and at the end of the job that window gets closed? Because threre is that old bug that closing a window (Mac) does not release its memory, unless in the cancelclose event of that window one does not put the proper declare (case #43484). But I’m not a pro, and my reasoning might be utterly out of mark.

Beatrix_Willius · May 3, 2018, 11:10am

@Carlo Rubini: this would be a couple of MBs only. But not a bad idea.

It’s not the thread. After eliminating the thread the memory consumption stays the same. When I click on the button a couple of times the memory used goes up only a couple of MBs.

However, when I change from 64bit to 32bit then I only get 29MB in the Activity Monitor.

Martin_T · May 3, 2018, 11:28am

I didnt tried this, but thats also interesting. So could be a 64-bit Bug in Xojo?

Beatrix_Willius · May 3, 2018, 12:02pm

@Martin T: could be. At least it looks very odd.

I’ve done a bit goggling about the differences between Activity Monitor and what Instruments shows. Have a look at

https://stackoverflow.com/questions/4596037/why-is-there-a-difference-between-the-reported-memory-usage-of-an-app-by-activit
or
https://stackoverflow.com/questions/27916048/get-memory-usage-of-a-process-from-c

When I execute the code a couple of times the memory in Activity Monitor goes up. This shouldn’t be the case.

Christian_Schmitz · May 3, 2018, 2:55pm

Well, first you need to know that the memory management functions in the C library will allocate memory in chunks, e.g. 64 KB.
The app may ask for small peaces like 16 byte. But the 64K block may not be freed until all smaller blocks inside it are released. So fragmentation may cause Activity Monitor to show more allocated memory than really is used.

The only way to find a leak is to make a sample app:

A method with the calls which may leak
A button which calls that method a dozen times.

Now run the app and click the button a few times. After each click, you may see a memory increase or not.
First call may fill some caches, so please don’t complain if first run uses memory.