Encodings, Codepoints and Memory Blocks

Background: I am finally trying to update some very old code (RB when we started it) in a desktop application we use to talk to our instruments via serial. During download process we create a temp .bin file that is then parsed using our various record definitions. Having occasional issues with some users with non-US-English PC settings where parsing goes haywire.

Actual question: In app we create a memory block and then pass as string to a parse function. The first thing is to determine the type of record and this was done with:

type=Asc(mid(RecIn,1,1)) //where RecIn is memblock passed as string

Our type definitions go to 0xFF, so we have non-ASCII bytes in that position sometimes. I note that if I define encodings

dim rec as String = RecIn.DefineEncoding(encodings.ASCII)

it still works, but if I use UTF8, I get 0 for any value > 0x7F. Maybe that’s correct, but seems odd.

I also tried using Codepoints here, which seems more appropriate, but I consistently get access violation crashes when doing so. For instance:

Var rec as String = RecIn
for each cp as integer in rec.Codepoints
type = cp
next

crashes. I don’t see much guidance in the documentation on how to use Codepoints correctly, but this looks very much like the example given. (have tried it with and without encodings)

It looks like most of this can be better handled with memoryblock.uint#value() than the home-rolled conversions and that’s where I’m headed, but am curious about above on codepoints and encodings.

This is the correct approach if you are dealing with binary data – using the string functions will be slower and you will inevitably run into text encoding issues if the data isn’t really a string. If there are actual strings in the data, you can use the string functions on those as appropriate; for everything else, use the MemoryBlock’s other functions.

Am wondering if the codepoint error has something to do with trying to assign a value by reading it directly? So

type = cp

would cause it? It doesn’t say so in documentation (that I can find), but is it iterable? (It looks like it could be from example.) Looking at iterable documentation it says if you change an Iterable while iterating over it you’ll get an exception.

We’d need to see more code to know for sure. I have to say the variable name “type” is a yellow flag - seems like that would be a reserved word in many languages - but if so, it shouldn’t compile.

In any case, you really should move away from thinking of these individual bytes as bits of a string if they are binary data. Just reorient yourself to modeling them as integers and you’ll be better off.

Asc() and Mid() are API1 functions that process characters not bytes, and this may be tripping you up, since the # of characters counted may not equal the number of bytes, depending on the encoding and byte values.

You could try the binary versions of these functions:

AscB()
MidB()

which only operate on individual bytes.