New Xojo framework extremely slow - what am I doing wrong?!?

General
1

A friend of my asked my to help him parsing a data-dump from a PV-farm installation. This is a 233 MB CSV file containing minute-by-minute data from the system. The file is semi-colon separated. “No problems I said” and fired up Xojo 2017r3 on my Ubuntu 17.10 (i7, 16GB of RAM, SSD) and went to work.
I decided to actually start using the new framework (only been using the networking classes before) and came up with this method to read the file:

dim tin as Xojo.IO.TextInputStream
dim f as Xojo.IO.FolderItem
dim lines as integer = 0
dim line as Text
dim parts() as Text
dim start as integer = Ticks

f = new Xojo.IO.FolderItem("/home/msa/Tangix_Work/REALbasic Projects/PV-Viewer/data/test minut.csv")
tin = Xojo.IO.TextInputStream.open(f, Xojo.Core.TextEncoding.UTF8)

while not tin.EOF
  lines = lines + 1
  line = tin.ReadLine
  parts = line.Split(";")
wend
tin.Close
msgbox str(lines) + endofline + str(ticks - start)

Compiled it as 64-bit and tried it. After 10 minutes of maxing out one of the cores in the system I quit the application and started debugging. All looked fine, lines are read one by one and the split() worked as expected. WTF?
So, tried the old school way of doing things:

dim f as FolderItem
dim tin as TextInputStream
dim lines as integer
dim parts() as String
dim line as String
dim start as integer = Ticks
f = GetFolderItem("/home/msa/Tangix_Work/REALbasic Projects/PV-Viewer/data/test minut.csv")

tin = TextInputStream.Open(f)
tin.Encoding = Encodings.UTF8
while not tin.EOF
  lines = lines + 1
  line = tin.ReadLine
  parts = line.Split(";")
wend

msgbox str(lines) + endofline + str(ticks - start)

This code read 525618 lines in 371 ticks.

My question is bluntly: Why bother at all with the Xojo.* framework? And also - is this the reason of the painfully slow IDE - it’s using the new framework?

The sad end (for Xojo and my trust in the viability of Xojo as a go-to-tool) to this story is that I ended up coding this project in PHP and JavaScript instead and running it on the web.

2

When the new framework came out, I decided to wait two releases and then start moving to the new framework for new things to be made. Well Mattias, while everything seemed to work well, which means I did not have functional issues, I experienced that quite big performance loss compared to the good old existing framework. Long story short, nowadays I only use parts of the new framework if there is a forced need for it to do so. If Xojo might decide not to support the standard framework anymore, I am gone.
But what I don’t understand, you had an urgent need for importing the csv data, why not just go for the least risky road, which is existing framework, 32-bit, mac or windows, go with the banana and done ?

3

Main reason - the IDE on Ubuntu is pain-fully slow and quirky (text-selection and auto-complete) and I’m out of the office without access to a Windows machine.

4

a simpler way might have been to import it to SQL database using Dave S’s “Tadpole” application.

5

Yes, it is what it is. I wouldn’t use the Linux IDE for the time being. Seems that a minority of Xojo developers is using the Linux IDE at all, so I can imagine that it’s more important for Xojo to get the Windows stuff state of the art.

6

Don’t read in line by line (horribly slow), use ReadAll and split that into lines.

Text is also slower than String. You could have stayed with String.

7

@Matthias Sandström: Did you have a look at the profiler which line is the bottleneck? Either the readline or the split.

@Markus Winter: ReadLine IS slow. But then I had a look at the OPs post again and there really is no reason why the code should be so slow.
But Text is supposed to solve all your encoding problems. Hahahahah…

8

Amen to that.
New does not always mean better.

9

I maybe recalling something wrong (happens a lot), but I think some text operations on the new framework (e.g. text.split) were dogs. Search feedback and/or forum.

10

Text on non-Mac platforms is painfully slow. On the Mac, it’s about the same, or even faster, than String, if I recall. There is at least one Feedback case about this.

Stick with String. The classic framework isn’t going anywhere.

11

@Beatrix Willius :
Using the profiler to see where the code is slow I had to break down the loop into separate calls (as the profiler don’t show line-by-line execution, at least not what I could find). One call for EOF(), ReadLine() and Split(). The “helper-functions” only do the actual command and return the appropriate type.
Imagine my surprise when I found out the code was now about as fast as the old framework (19 ms vs 11 vs)!

I am reading just 100 lines from the file but doing it ten times to avoid any caching issues.
Clearly the new framework shouldn’t be used in tight loops.

12

Correction:
I now see the same loss of performance even when running it non-inline. I have no clue why.
Bottom-line still the same - the new framework is slow. Seems like it is the Text.Split() and that has been reported many times.