Fast character access of a string

General
1

Hello,

is there any way to access a single character of a string like mystring[3] for accessing the fourth character?
I know that I can access it with mystring.middle(3,1) but that function is too slow for fast string manipulations.

Thanks in advance!

2

memoryblock?
https://documentation.xojo.com/api/language/me.htmlmoryBlock

3

There is an example Advanced/Memoryblock/ Fast string append

4

The MiddleBytes function is faster as the normal Mob one as it doesn’t need to count utf-8 characters.

5

Thank you, I will try that one!

6

It depends. Will any of the potential characters be multi-byte, like © or :slight_smile: ?

7

Yes, I am going to use UTF8…

8

Then you should use Middle or you will only get a single-byte of a multi-byte character.

9

But if you need to do this repeatedly, use Split to get all the characters into an array, then use that for processing.

1 Like
10

I guess, this will not really speedup my procedure because I have to parse a table with about 5.000.000 rows… :thinking:

11

If you are reading them sequentially then perhaps wrapping it in a BinaryStream will help:

Dim mb As MemoryBlock = GetTheRawData()
Dim stream As New BinaryStream(mb)
Do Until stream.EOF
   Dim nextchar As String = stream.Read(1)
   [...]
Loop

Or

Dim mb As MemoryBlock = GetTheRawData()
Dim stream As New BinaryStream(mb)
stream.Position = 3
Dim fourthchar As String = stream.Read(1)
[...]
12

Thank you, I can try this.
But would that work for UTF8 characters as well?

13

Ok, I have used stream.read(1.encodings.utf8) now.
But unfortunately, after profiling I can see that my method is about 12.7% slower if using the MemoryBlock.
It seems I have to stick to the middle() function…

14

Can you code the sql to do what you need and return the results?

1 Like
15

No, I am reading a csv-file.

16

You may find that a combination of techniques is best for speeding up the data processing. For example, if you need to find the end of line characters in order to separate the rows of data, then using the split function, with the end of line character as delimiter, is probably the fastest way to do that. Once you have your separate rows, you may find that another technique is faster for the remaining processing.

17

That’s exactly what I am doing. I just thought there has to be a function to access a character at a position of a string with a length of 1. The middle function has always to lookup for the number of characters which has to be slower. However I don’t know the compilation or optimisation details.

18

I would do this:

  • read the entire file into memory
  • split lines into an aray using SplitB (since UTF8 guarantees that all ASCII chars are one byte)
  • for each line - compare the length using Len() and LenB()
  • if they are the same, use a fast mode where all your code uses the byte versions, e.g. MidB(), LeftB() etc.
  • if the 2 lengths differ, then you do have actual Unicode characters, and you should process that line using the Unicode-aware versions (Len(), Mid() etc.) which are slower

Post some code and we’d be happy to critique

1 Like
19

Not a bad idea.

I am new to Xojo and therefore my code will not look perfect. :grin:
It’s my own version of Split() because I have to take care about the double quotes as well.

However, this is my fastest version until now:

method MySplit (extends source as string, delimiter as string = " "): string()
var cnt,len as integer
var arr() as String
var str,ch as string
var instr as Boolean
instr=false

len=source.Length-1
for cnt=0 to len
ch=source.middle(cnt,1)
if ch<>delimiter and ch<>"""" then
str=str+ch
Continue
end if
if ch=delimiter and instr=false then
arr.Add(str)
str=""
Continue
end if
if ch="""" and instr=false then
instr=true
elseif ch="""" and instr=true then
instr=false
end if
next
arr.Add(str)
return arr()

I am sorry, for the bad formatting of copy/paste.

20

First, there are already CSV parsers available and you might want to use, or at least look at, one of those.

Second, your code won’t handle escaped quotes like \".

Third, what’s probably the slowest part of the code is str=str+ch. Concatenating strings like this tends to be slow because, each time, a new string is created and the old one destroyed. There are techniques you can use to speed that up such as using an array to “build” the string, then using String.FromArray to join the characters. (If you didn’t have to worry about escaped characters, you could just track the start and end of each segment and use Middle to extract it.)

Next, if the delimiter is going to be a single byte (as is usually the case when splitting CSV), you can still use a MemoryBlock to scan the string for the delimiter, then use StringValue to extract the segments. This is a more advanced technique though and some are not comfortable with MemoryBlocks.

Lastly, please use code tags when posting code to make it easier on us.

3 Likes