Resizing large Arrays creates an app crash on windows

General
1

Hi

I’m starting to receive complaints from my costumer about application crashes. My costumer uses my application to analyze data collected from oscilloscopes. Oscilloscopes can create huge amount of data points to analyze. I have traced down the problem to be caused by the redim or resizeto functions of arrays when the array contains large number of data points.

Consider the following 7 rows of code:

var n as integer = 300000000

var arr1(-1) as double
var arr2(-1) as double

arr1.ResizeTo( n )
arr2.ResizeTo( n*2 ) // It is possible ta define a much larger array than n , but you can’t resize it

var m as Integer = n + 1 //Any number m <> n will create an app crash!

arr1.ResizeTo( m ) // Results in an app crash with Xojo 2006r2.1 and earlier on windows 10, 11 equipped with 64Gb Ram

Do anyone knows anything about this behavior?

Best regards Håkan N

2

What does the crash log say? Have you calculated how much memory your large arrays need? I’ll bet with you that the app runs out of memory.

3

Is this a mistype?

4

Hi Beatrix and thanks for your answer.

When I run this 7 lines of test code and removed the definition of the arr2 ( It was just a test to find out if I was close to a memory limitation ), I have 54GB free RAM. The arr1 size allocated is only 2.3GB. My conclusion is that it shouldn’t be any issue with the memory if xojo doesn’t do any weird things when resizing arrays. It look like an access violation for some reason. The dump file contains the following:

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr
(410.1de8): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtWaitForMultipleObjects+0x14:
00007ff9`61360ed4 c3

The crash also occurs in Xojo debug mode and when it crashes a window is popping up containing the following text:

Your application crashed due to an access violation. The debugger may show invalid values. Check the selected line, make corrections, and try again…

What can cause access violations? Interference with windows virtual memory paging file? Xojo not 64bit clean?

/Håkan

5

Yes it shoud be 2026r2.1

6

I guess it should be repeatable in Xojo test machines. Looks like some bug in the ResizeTo() internals.

Create a small sample, assure it crashes, zip the sample project, make an issue report and sending it attached to.

7

An access violation happens when the OS kills your app for trying to access memory it’s not allowed to. Things like null pointers, trying to access another process’s memory, etc.

I can confirm the crash on this code. This is a framework bug. You should submit an Issue with this code.

Here is a workaround that makes a new array instead of resizing an existing one… It doesn’t hit the access violation, but it’s really slow because it has to copy all that data in the for loop.

Var n As Integer = 300000000

Var arr1(-1) As Double
arr1.ResizeTo(n)

Var newArr(-1) As Double
newArr.ResizeTo(n + 1)

For i As Integer = 0 To arr1.LastIndex
  newArr(i) = arr1(i)
Next

arr1 = newArr

Might get a little boost if you enable aggressive compiler optimization but ultimately the framework bug needs to be fixed. My guess is somewhere in the framework they’re storing the element count * element size in a signed 32-bit variable and it’s overflowing once the array reallocation grows larger than 2.1 GB.

If you don’t want to wait for that, I’d probably rearchitect this using memoryblocks instead of arrays, and chunking the data into, say, a million doubles each (8 MB) so you’re not just trading huge arrays for huge memoryblocks. Try using pointers to see if it speeds up vs calling .doublevalue over and over. You could wrap it all up in a class to make it prettier.

8

Hi @Hakan_Nyberg ,

your are fortunate to know where the applicatios crashes.

I sugguest trapping errors using a try/catch block. Since we do not know which execption is thrown, I suggest trapping for all exceptions. When one occurs, then examine the exceptions contents. If it’s a framework issue, chances are the specefic place in the framework be identified.

HTH

9

Unfortunately it doesn’t exhibit as an exception, it’s a hard crash that is not trappable.

10

Christian

Thank you for the support and the the crash confirmation.

I will submit a case on this.

Regards /Håkan N

11

Rick

Thank you for the support.

/Håkan N