Memory Leak Under Load

Targets Windows
1

I’m struggling at the moment with an application that runs continually and generates documents every few minutes. A timer kicks off a batch process of generating the documents and then waits a few minutes before going again.

In the live environment, the memory usage of the application climbs until eventually it crashes. I’ve been going through the application with a fine tooth comb trying to discover where I might be causing the leak but to no avail. I’ve tried parring back the code until I find the offending line or routine that I think causes the problem and I’ve even got down to a line of code that when comment it out, stops the leak. (It’s was a FolderItem.CreateAsFolder call). But when I then bring back other parts of the application, the leak goes off again. And if I create a test app to hammer the CreateAsFolder call, there’s no leak.

I’ve tried creating a test application so I can show the Xojo guys but what I’ve found is, it only happens as you increase the load on the application. So I can run one part of the application over and over with no leak. I can run a second part over and over with no leak. But, when I run them together, off it goes. The 2 parts do not share any objects or routines so I’m at a loss as to what’s going on.

Has anyone else had any experience of this or got any suggestions about how I can find out what the issue is?

I have the test application available if anyone is interested.

Thanks in advance.

2

Maybe try the hints given here
https://forum.xojo.com/25201-how-to-find-a-memory-leak

3

Thanks Dirk. I’ve put the code snippet in to test the objects in use and the object counts all remain static but you can just watch the memory usage go up.

Time to raise an issue I think.

4

Jim - what memory statistic are you looking at to determine that it’s leaking?

5

In a few applications, I’ve had to include the flush application from Microsoft, when the bugs were Xojo framework based. Basically, the flush toolkit scans through the application in memory, and forces memory that is not used by the application (has been released, but not freed…thus the memory leak) to free up. Generally, on Windows I’ve noticed when Xojo is left open for days, its memory will increase to almost a Gigabyte of ram usage, when not in use. So, I wrote a tool to invoke the flush toolkit on Xojo when the memory reaches a certain value, and once that value is reached, the memory is freed. It does not affect Xojo when using it, and does not cause an application to behave badly or ‘not as intended,’ it merely frees up the released memory that has not been freed by the application (usually by bad programming…not always the developer’s fault in Xojo).

If you want to try it, it’s downloadable here:
http://www.xojodevspot.com/empty.zip

using it, is as easy as running from the command line as:

empty.exe applicationname.exe

if there is a space in the app name you must use the application PID,

empty.exe PID

Putting it all together and using in Xojo:

Function GetPID as Integer
#if targetwin32 then
Soft Declare Function GetProcessID Lib “Kernel32” alias “GetCurrentProcessID” () as Integer
#endif
return GetProcessID
end function

and use the shell class as:

Dim xShell as new Shell
Dim ePath as String = GetfolderItem…path to .Child(“empty.exe”).ShellPath
xShell .Execute ePath + " " + str(GetPID)
xShell.Close

It’s not hackery, and quite the reason why Microsoft developed the toolkit in the first place; to handle memory leaks that are not in the control of the end-user (or Xojo developer in the cases when the framework is the culprit).

You can replace the extension with DLL and place the file in Libs/Resources folder when compiling for software distribution, just don’t forget to make the extension change in your code.

Just for giggles, open task manager and run empty.exe against a few applications with large ram in the command line and you’ll be quite surprised how many pieces of software actually have ‘slight’ memory leaks in them (moreso bad programming of releasing memory but not freeing it). After freeing unused memory by applications which have bad programming, you’ll notice they’re more responsive, and your system has more resources to use for other importing things.

6

Jim is this a threaded app? I just ran into situation where under some circumstances, I was closing objects from within a thread - it didn’t trigger any errors and seemed to be working, but in fact something was going wrong, with terrible leaking and bizarre behavior.

I solved it by putting

Class.Close
   if app.currentThread<>nil then BigUglyWarningMessageAboutThisNotBeingAllowed()
end
Class.Destructor
   if app.currentThread<>nil then BigUglyWarningMessageAboutThisNotBeingAllowed()
end
7

Is there any way to define the “certain value”?
Is the source code for empty.exe available?

8

Thanks @Matthew Combatti for sharing … very useful for all Windows applications running 24/7.
I used to have such solutions >15 years ago, back in my VB6 days, using the main executable only for launching, keeping eyes on and relaunching ActiveX-executables running the real business logic. It ran unattended for >8 years on several NT and Win2003 servers, but isn’t this why tech guys hate Windows for 24/7 processes.

9

Thanks for all your replies.

@Michael Diehr
Is this a threaded app? Well, I’m launching a process via a timer so that’s a new thread but nothing else is running at the same time.

@Michael Diehr
What memory statistic am I looking at to determine that it’s leaking? I’m using the Task Manager in Windows and I’ve also using Runtime.MemoryUsed. I in fact used the Snippet of code suggested by @Dirk Cleenwerck earlier in this thread.

@Matthew Combatti
Thanks for your suggestion. I’ll implement this and see what happens.

I’ve also raised this as issue #40566. The example app is attached.

What I’ve had to do as a work around is to create the batch process as a run once, stand alone EXE and then have another very simple launcher app that has a timer which does a Shell to the stand alone app. A bit desperate but it seems to be working.

10

Long long ago, I used this Windows API call which has allways worked on a Win2003 server.

11

I tried your test app in Windows 10 x64 - after 55 runs making 550 folders, memory usage is up to 21092K, which is up from 20024K after the first run. So roughly 1MB leak for 500 folders - perhaps a 2K leak per folder? Is that the kind of numbers you are seeing?

With that sort of leak, your app should be able to create roughly 500,000 to a million folders before running out of memory.

Note: just to be sure it’s the folderItems leaking (and not, say your logging code which logs the data to the window) I replaced your logs with System.DebugLog() - which can then be viewed with https://technet.microsoft.com/en-us/Library/bb896647.aspx DbgView.exe.

Then I see 550 folders taking 20905-20418 = 487K, perhaps 1K leak per folder?