Memory problem in console app

Targets Linux
1

Hi,
I have a memory problem in a console application running on a Debian 11 virtual machine.

The application is really simple:

  • run a big query on a MySQL database.
  • for every row launch a method that:
    – selects the single database row
    – sends an API request using URLConnection with a Json content
    – receives a Json content from the POST request
    – updates the row accordingly to the data received.

On the Debian system the memory usage goes slowly up, until the app is killed by the system (at aroung 66000 rows)

I have tryed to:
make the URLConnection property global and reuse it (insted of a new instance for every row)
pause the execution every N rows using DelayMBS (maybe giving time for some garbage collection)
split the big query using a cycle with a limit value, and every time close the connection and reconnect

But no way.

It works just fine ion my MacOS Sequoia, but terribly slower!

What could be the cause?

Thank you!

2

Check the count of your objects to see if you have a memoryleak somewhere. As the app also runs on macOS you can use Instruments to do a leak check. This might require to setup special entitlements (get task allow).

DelayMBS is only needed in really rare cases. I have some cases where I need to use NSAutoreleasePoolMBS. But this is macOS only - for AppleScripts.

3

Thank you so much!
Sincerely I didn’t know about ‘Instruments’ and its possible use to investigate problems.
Now I need to figure out how to use it, but this is a really really good tip.

4

Have a look at Blog: News and Tutorials about Mail Archiver . The interface has changed a bit. But the actions are the same.

3 Likes
5

There is a confirmed Memory Leak with URLConnection on Linux, less than 1 KB per URLConnection.
https://tracker.xojo.com/xojoinc/xojo/-/issues/78053

Also, URLConnection doesn’t release its memory immediately. There can be a small delay of 10 seconds.

If your app is running in a tight loop, you could add some sort of 20s pause every 5000 URLConnections to make sure the app is releasing memory when needed.

In my case, it is in a WebApp where each connected user will perform 5-10 URLConnections.
After 100.000 connected users, the memory usage grows to several GB.

1 Like
6

Thanks Jeremie for the info.
I have tryed to set a longer pause.
mind you, my loop is intended to be for more than a million elements…

7

I once did something similar for 40 million rows of a database.

You could rewrite your code to use workers that process 100.000 rows at a time. That will make sure the memory is correctly released as soon as the worker is killed after processing all assigned rows.

8

OK, Good idea.
Tha bad is thet the memory leak makes my app be killed when arriving at around 56000…
I have added myself to the issue, the last message was by William Yu telling that an update to libsoup-3 in 2025r1 could resolve the issue, but I’m indeed using 2025r1