Large memoryblocks under 64 bit

Targets macOS
1

I’m in the process of porting my app to 64 bit, ran into some issues and investigated with this simple code snippet:

dim n as integer = 5e9 ’ allocate 5GB
dim m as new MemoryBlock(n)
for j as integer = n-1 downto 0
m.UInt8Value(j) = 128 ’ OutOfBoundsException at the first iteration
next

Unless I’m missing something obvious, there is something really amiss.

  • allocating a 5GB block is seemingly successful, but as shown the first iteration throws an outOfBounds
  • allocating a 4GB block succeeds and the loop runs fine, but curiously the size of the memblock is reported as negative in the debugger
  • allocating <= 2GB succeeds, the loop runs fine, and the size of the memblock is correctly reported in the debugger

So it seems that:
a) m.Size is still an Int32 even under 64 bit, even tho ‘integer’ should be a 64 bit Int64
b) m is limited to 4GB (32 bits) in size (at least the indexing of its elements, the block itself can be allocated bigger it seems) under 64 bit

Even tho the docs clearly state “There is no practical limit for 64-bit apps” .

What am I missing? If I can’t even get this to work, no hope for porting to 64 bits!

(and yes, my compiler is certainly set for 64 bit and I have 32GB RAM tho that should not matter).

Thx for any insights,
Peter.

2

The size parameter is of type Integer, which is 64-bits when built as 64-bit.

Now, one thing to consider… even though you have 32GB of RAM, you may not have 5GB of contiguous memory available. You may be able to check this by rebooting.

Also, right after creating the Memoryblock, try this:

dim tmp as uint64 = m.size

I’d be curious if you’re getting a smaller block than you requested.

3

I ran the test in the latest Beta [Edit: run as a Debug, not compiled app] and

msgBox str(m.size) gives 705032704 which is a weird size: 672.3615 MB

Seems like a bad bug to me: allocating a 5GB memory block on a 64 bit machine with 16GB physical ram should “just work”.

4

Aha: 5e9 - 4GB is exactly equal to 5e9-410241024*1024 = 705032704 which is the size of the memoryBlock I’m given.

So it looks like some sort of 32 bit overflow or rollover in the memoryBlock constructor?

5

Thanks Peter, I submitted <https://xojo.com/issue/52898> as a bug against the most recent beta.

6

Thx Greg. I don’t think contiguity of memory matters these days because I was under the impression that the caller receives a logically contiguous block of memory, and the memory manager takes care of the actual physical locations that can be scattered all over various pages in physical RAM and VM. But I could be wrong.

Regardless, I did what you suggested, rebooted and the results were the same.

To answer your tmp as UInt64 question, if I request 5e9 bytes, tmp shows 705MB but I think this is wrong, I think its simply a 32 bit integer overflow and wrap because as it turns out, 5e9-2^32 is exactly the value that both m.Size and tmp show.

If I request 4e9 bytes, tmp = 18446744073414584320 which is also wrong, simply reflecting the cast from m.Size = -294967296 to a UInt64.

Now even more curious, if I do:

dim p as Ptr = m for j as integer = n-1 downto 0 p.UInt8(j) = 128 next
It works. Sort of. The loop runs for a while then crashes:

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x0000000114e8bc5f
Exception Note: EXC_CORPSE_NOTIFY

Termination Signal: Bus error: 10
Termination Reason: Namespace SIGNAL, Code 0xa
Terminating Process: exc handler [0]

VM Regions Near 0x114e8bc5f:
CoreUI image data 0000000114c59000-0000000114c6b000 [ 72K] rw-/rwx SM=PRV
–> mapped file 0000000114c6b000-0000000114ea2000 [ 2268K] r–/rwx SM=COW Object_id=7e90559f
CoreUI image data 0000000114ea2000-0000000114ed9000 [ 220K] rw-/rwx SM=PRV

So was the memblock allocated or not? It seems yes, because when I use my C plugin fn:

fastFillPtr8(p, 127, n)

that does what its name says, it returns with no problem. Now checking the values of p in the debugger:

https://www.dropbox.com/s/c0t6j5egz0eifka/PtrBug.png?dl=0

Many bytes report as 00 when they should all be 7F. So were the values written incorrectly? No, because my other plugin fn:

dim b as Boolean = checkPtr8values(p, 127, n)

reports that all 5e9 bytes were in fact 7F.

Bottom line: new memoryblock allocates correctly, but accessing the bytes is totally messed up for large blocks.

Peter

7

[quote=399317:@Michael Diehr]Aha: 5e9 - 4GB is exactly equal to 5e9-410241024*1024 = 705032704 which is the size of the memoryBlock I’m given.

So it looks like some sort of 32 bit overflow or rollover in the memoryBlock constructor?[/quote]
Either that or in the Size method.

8

I have changed it to a non-beta bug though as this problem is not new to 2018r2.

9

Confirmed it also happens in Windows. Hopefully this is fixed at the same time as <https://xojo.com/issue/52211>

10

As a workaround, can you use the new framework?

Dim mb as new xojo.core.memoryBlock( 5e9 ) Dim tSize as Uint64 = mb.size

Returns the correct value here (2017r3, macOS 10.11.6).

11

Yes Xojo.Core.(Mutable)MemoryBlock seems to work, but I don’t see how I can get a Ptr to this block, both for faster access from within Xojo and to pass to my many plugin functions. Is this possible like with classic Memoryblocks? If not, a non-starter for me unfortunately.

p.

12

From what I recall.

Dim mb as new xojo.core.memoryBlock( ptr, 5e9 )

Should do it, no?

13

Dim mb As New Xojo.Core.MutableMemoryBlock(5e9) Dim p As Ptr = mb.Data Dim tSize As UInt64 = mb.Size

14

But the LR states that p = mb.Data is a read-only Ptr. This would imply that I can neither write to this Ptr in Xojo, nor pass it to a plugin to modify the contents.

And Sam, your suggestion creates a xojo.core.memoryBlock from a pre-existing Ptr. What is needed is a R/W Ptr to a newly created memoryblock, like we can do with classic mb’s.

Generally speaking, what is the fundamental difference between classic mb’s and xojo.core.memoryBlock’s? When should we use one vs the other? Seems redundant to me.

15

BTW Greg, I also noticed that Ptr addresses are reported incorrectly in the debugger: their values are truncated to 32 bits (8 hex digits) even tho the Ptr is 64 bits (convert the Ptr to an integer and display in hex and you’ll see what I mean). You may want to fix that while you’re at it.

16

That just means you cant alter the pointer, which would be a good thing.

17

It was my understanding that this memoryblock operates the same as the ‘classic’ memoryblock, but I have to confess that I’ve never actually used this version.

One was meant to replace t’other one.

I’m not at my ‘work’ computer, so I don’t have Xojo to hand, but a quick Google, reveals that you may want to look at the “MutableMemoryBlock” instead.
http://developer.xojo.com/xojo-core-mutablememoryblock

it’s constructors are the same, but in theory having mutable in the name, means that it can be edited. This is one thing from the Obj-C world that I dislike, is having two sets of classes, mutable and immutable. I’ve heard the advantages and technically understand them, but I still prefer having one class to rule them all.

18

It would be helpful to have a separate bug report as these fall under vastly different areas.

19

<https://xojo.com/issue/52902>

20

More followup. So I played with MutableMemoryBlock:

[code]Dim mb as new Xojo.Core.MutableMemoryBlock( 5e9 )
Dim tSize as integer = mb.size
dim p as Ptr = mb.Data

fastFillPtr8(p, 127, tSize)
b = checkPtr8values(p, 127, tSize)

for j as integer = tSize-1 DownTo 0
p.UInt8(j) = 127
next
[/code]

Here the size is reported correctly, my C plugin fills the bytes, and checks the values correctly (even for ridiculously large blocks of 20e9 bytes). However, the XOJO loop crashes after a few seconds (see the crash log at the end). If I reduce the size to 2e9, the xojo loop completes correctly and all bytes verify.

So MutableMemoryBlock allocates correctly, returns the correct size, returns a valid Ptr in mb.Data, but there is an addressing problem also for large MutableMemoryBlocks.

As an aside, my C plugin fills 2e9 bytes in 0.1 seconds (in fairness, that’s the second time around, the first time it takes about 10x longer, probably the memory manager is mapping requested pages into RAM or something) vs the xojo for-next loop that takes 52 seconds.

I did include these pragmas to give xojo its best chance:

#pragma BackgroundTasks Off #pragma BoundsChecking Off #pragma BreakOnExceptions Off #pragma NilObjectChecking Off #pragma StackOverflowChecking Off
While I expected xojo to be a bit slower than C, but 500-fold?

P.

System Integrity Protection: enabled

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x000000011b125fff
Exception Note: EXC_CORPSE_NOTIFY

Termination Signal: Bus error: 10
Termination Reason: Namespace SIGNAL, Code 0xa
Terminating Process: exc handler [0]

VM Regions Near 0x11b125fff:
CoreUI image data 000000011aedb000-000000011aeef000 [ 80K] rw-/rwx SM=PRV
–> mapped file 000000011aeef000-000000011b126000 [ 2268K] r–/rwx SM=COW Object_id=7e90559f
MALLOC_LARGE 000000011b126000-000000011b234000 [ 1080K] rw-/rwx SM=PRV

Application Specific Information:
Performing @selector(performClick:) from sender XOJButton 0x6040001513b0

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 memblock2Integer.debug 0x000000010e788d28 Window1.Window1.PushButton3_Action%%o<Window1.Window1>o + 3192 (/Window1:97)
1 memblock2Integer.debug 0x000000010e78ce24 Delegate.IM_Invoke%%o + 52
2 memblock2Integer.debug 0x000000010e78ce74 AddHandler.Stub.15%% + 52
3 com.apple.AppKit 0x00007fff38eb375a -[NSApplication(NSResponder) sendAction:to:from:] + 312
4 com.apple.AppKit 0x00007fff38959933 -[NSControl sendAction:to:] + 86
5 com.apple.AppKit 0x00007fff3895985b __26-[NSCell _sendActionFrom:]_block_invoke + 136
6 com.apple.AppKit 0x00007fff38959761 -[NSCell _sendActionFrom:] + 183
7 com.apple.AppKit 0x00007fff3899aa18 -[NSButtonCell _sendActionFrom:] + 97
8 com.apple.AppKit 0x00007fff38957fca -[NSCell trackMouse:inRect:ofView:untilMouseUp:] + 2438
9 com.apple.AppKit 0x00007fff3899a75f -[NSButtonCell trackMouse:inRect:ofView:untilMouseUp:] + 777
10 com.apple.AppKit 0x00007fff38956a64 -[NSControl mouseDown:] + 965
11 XojoFramework 0x000000010e91b337 0x10e843000 + 885559
12 com.apple.AppKit 0x00007fff39052959 -[NSWindow(NSEventRouting) _handleMouseDownEvent:isDelayedEvent:] + 5891
13 com.apple.AppKit 0x00007fff3904f5b0 -[NSWindow(NSEventRouting) _reallySendEvent:isDelayedEvent:] + 2359
14 com.apple.AppKit 0x00007fff3904e85c -[NSWindow(NSEventRouting) sendEvent:] + 497
15 XojoFramework 0x000000010e926b45 0x10e843000 + 932677
16 com.apple.AppKit 0x00007fff38eaf617 -[NSApplication(NSEvent) sendEvent:] + 307
17 XojoFramework 0x000000010e9165cb 0x10e843000 + 865739
18 memblock2Integer.debug 0x000000010e6f5ec5 Application._CallFunctionWithExceptionHandling%%op + 181
19 XojoFramework 0x000000010ea96e53 CallFunctionWithExceptionHandling(void (*)()) + 262
20 XojoFramework 0x000000010e916546 0x10e843000 + 865606
21 com.apple.AppKit 0x00007fff38710d9d -[NSApplication run] + 812
22 XojoFramework 0x000000010ea9517f RuntimeRun + 40
23 memblock2Integer.debug 0x000000010e750a53 REALbasic._RuntimeRun + 19
24 memblock2Integer.debug 0x000000010e792bde _Main + 846 (/#main:87)
25 memblock2Integer.debug 0x000000010e792123 main + 19
26 libdyld.dylib 0x00007fff62b0d115 start + 1