I’m trying to optimize the saving of a file with tens of thousands of writes. I’ve found it’s faster to make a memory block and write it all at once. Now I’m trying to speed it up even more by not using binary stream writes to the memory block.
I want to build the memory block as I go and find I can use memblock = memblock.AppendMBS(“string”) for strings since the string is automatically converted to a memory block.
How can I convert a Uint32 to a memory block on the fly? Something like:
This also raises another question. I’m saving data from an array of possibly thousands of classes (they define songs full of MIDI data). Is there a faster/better way to save the data without plucking each property (and imbedded array of classes) out of each element of the class?
UPDATE: I just realized someone might wonder why I’d be saving thousands of songs full of MIDI data. I’ve written my own app which sits in the background or foreground and captures (records) all MIDI events. If 5 seconds goes by with no events, it’s saved as a song. It captures whatever I play on my Yamaha Disklavier grand piano. So over a period of several months, there are thousands of “songs”.
Xojo supports creating a BinaryStream that is backed up by a MemoryBlock. This lets you use all the normal BinaryStream functions with the output being stored in memory. The coolest feature is that the MemoryBlock will automatically be expanded as your BinaryStream needs more room. Here’s what the syntax looks like:
Dim m as New MemoryBlock(100)
Dim b as New BinaryStream(m)
b.WriteUINT32(someNumber)
When you’re done writing, make note of the position of the BinaryStream and close it:
Dim dataLength as Integer = b.Position
…and retrieve the contents:
Dim writtenData as string = MemoryBlock.StringValue(0, datalength)
I know about the relationship of MemoryBlock and BinaryStream. I tried this with my routine for reading in the data and it was much faster. But then I was able to cut the time in half again by avoiding BinaryStream reads and using my own pointer to parse the MemoryBlock.
I’m hoping I can save a similar amount of time by avoiding thousands of BinaryStream writes while creating a MemoryBlock for saving.
The only (other) way to accomplish this is to precalculate how big your MemoryBlock needs to be, so you can create it large enough and blindly increment your write position without worrying about going off the end of the MemoryBlock.
This is a nice and useful thing to do in any case because it allows you to pre-flight your memory and disk usage - you’ll know before you write whether you have enough room, and can gracefully fail and let the user know why.
Eric, I thought about allocating the block first, but until you put it that way, I hadn’t realized what a good idea that is. My largest file so far is still under 200MB. I think I’ll do that instead of banging my head trying to append to a MemoryBlock. Appending and resizing the block probably costs a bit of time, too.
I still wonder why there isn’t an easier way to “build” a MemoryBlock as you go, much like appending to arrays.
If you’re curious, do a little looking into how computers allocate memory, and then the way that BinaryStream+MemoryBlock operate will make as lot more sense.
Essentially, growing a MemoryBlock a little at a time is extremely inefficient because it requires a sequence of allocation, copy, and deallocation tasks. When BinaryStream runs into the end of a MemoryBlock, it enlarges it by a large factor to avoid the cost of multiple smaller expansions.
Fundamentally, it does what you originally were looking for, and with a pretty logical approach. But, as you’ve discovered, there are good reasons to take a different path.
The problem is how memoryblocks are allocated. They’re just a pointer to memory with a certain amount of guaranteed space. When you ask to “append” to an existing block, the framework would need to create a new block of the slightly bigger size, add the data and then release the previous one. It’s slow to do with large blocks.
