MemoryBlock Limit problem

General
1

I am trying to read the full binarystream into a memoryblock.
When the file size exceeds 2GB I get a zero sized memoryblock.
This happens with both 32 and 64 Bit builds.
Any help ?

Dim bstream As BinaryStream
Dim mb As MemoryBlock
MsgBox str(bstream.Length)
mb = bstream.Read(bstream.Length)
MsgBox str(mb.Size)

2

Compile a 64Bit-App

Look at: https://documentation.xojo.com/api/language/me.htmlmoryBlock

3

I don’t have an answer to your question, but am wondering why you do this. What do you do with the contents of the MemoryBlock? And can you do it in chunks instead?

1 Like
4

Misreading your post. It seems to be a bug in Xojo or Xojo-documentation.
Process your file in chunks.

5

don’t do that.
Use string as Read function gives string.

6

I am actually trying to read in an entire pdf file so that I can access randomly any part of that file.
Theoretically the documentation says there is no practical limit for memoryblocks in a 64 bit build.
But the problem exists in both 32 and 64 bit builds.
Which is the best way to solve this problem ?

7

Try this:

#Pragma BackgroundTasks       False

Var blockSize As Integer = 1024 * 1024 * 128
Var Offset As Integer
Var s As String

Var readFile As FolderItem = FolderItem.ShowOpenFileDialog("")

If readFile <> Nil And readFile.Exists Then
  Var ReadStream As TextInputStream = TextInputStream.Open(readFile)
  Var mb As MemoryBlock = New MemoryBlock(readFile.Length)
  
  Do Until ReadStream.EOF
    s = ReadStream.Read(blockSize)
    mb.StringValue(Offset, s.Bytes) = s
    Offset = Offset + s.Bytes
  Loop
  
  readStream.Close
  
End If

Btw. The LR is OK, there is no practical limit in 64Bit-Apps, but Strings are limited to 2GB.

8

Reading chunk can still be done:
first of all, read the toc (famous last words… Err: last part of the file)
then read what you want ?

My understanding of the pdf

9

Var mb As MemoryBlock = New MemoryBlock(readFile.Length)

In the above statement, if readFile.Length exceeds 2GB then the size of the memoryblock mb becomes negative. This happens with the 64 Bit build. In the 32 Bit build, it gives an out of memory error.
When the file size is 2203001756 bytes, the size of the memoryblock mb becomes -2091965540 and the statement mb.StringValue(Offset, s.LenB) = s gives an Out of Bounds error.

I am using Xojo Version : 2017 Release 2.1

Any suggestions please ?

10

In Xojo 2020R1 on iMac19,1 macOS 10.15.6 it works.

11

You probably need bs.close after read to set the length of the memoryblock

12

Can you map the segments of the file instead? When you need to search it, you can load up segment by segment.

13

I believe MemoryBlock was updated in 2019r2 to fully support 64-bit sizes.