I made a previous post calling out threads, but I just ran a test non-threaded. The issue isn’t threads. It’s FolderItems and/or TextInputStreams.
I just made a test for this. New project, new button. Clicking the button loops through a folder of 1000 text files and simply opens each file and immediately closes it using TextInputStream.open(f.item(i))
No data is read from the file, no additional code is run. Simply loop the files, and open and then close each one.
On Xojo 2019 R1.2 this takes <1 second.
On Xojo 2020 R2 this takes 9-11 seconds. It’s about ten times slower! The problem gets even worse when you actually read data from the files and do anything with the data.
Opening files is much, much slower now.
Is this a known issue? I can provide a project file if needed.
Why are you using TextInputStreams and not BinaryStreams?
Because I have to read the data in plain text. It’s how I’ve been doing it for this app for 20+ years. When I save the data in binary I get all kinds of edge case issues with special characters and crashing.
You are doing it wrong, then. BinaryStreams work just fine for text files.
What is the advantage of using a binarystream? I’ve tried using them multiple times over the years and they do not work for my needs. More importantly, why is TextInputStream now 10 times slower in the latest versions?
EDIT: Just for fun, I changed my sample project to open the file as a BinaryStream, and it takes the exact same 11 seconds in Xojo 2020 R2. Still takes <1 second in 2019 R1.2 FYI.
I do know there are performance problems looking at a lot of files.
From the folder item lang ref
Avoid invoking functions such as Count, Child and IgnoreAlias multiple times for the same target because these functions are time-intensive (especially on macOS).
If you walk directory contents, follow these rules:
- Always iterate forward, starting at index 0 and ending with the index that matches the folder’s Count. If you iterate backward, it may get very slow if the directory contains a few hundred or even more items.
- If you want to recurse into subfolders, do not go depth-first. Instead, first collect all items into an array of FolderItems, then walk the array items and enter any folders you encounter.
- To delete items from a folder, follow the above rules as well. Do not be tempted to walk the directory items backwards (from Count downto 0), even if you see many recommendations for doing so. The proper way is to first collect all items in a loop into an array, then walk the array and delete the items accordingly.
Thank you for your reply. I already follow all of those guidelines, and this was very efficient until just the latest releases of Xojo. Something major has changed recently.
I just tested with this code, over 784 files on 2020r2 on macOS 10.14.6
dim f as FolderItem = folderitem.ShowSelectFolderDialog()
if f = nil then
quit()
end if
dim d as Double = System.Microseconds
dim t as BinaryStream
dim a as integer
for each c as FolderItem in f.Children
if c.NameExtensionMBS = "jsx" then
a = a + 1
t = BinaryStream.Open(c)
call t.Read(t.Length)
t.Close()
t = nil
end if
next
d = (System.Microseconds - d) / 1000000
break
Took 0.03 seconds, files were 2K each
1 Like
Although a valid question, it is not related to the issue described by the OP.
If it is indeed slower, this is a bug.
1 Like
Sure. But I’m a pragmatist and do whatever is necessary to make my code fast.
Anyways, tested with Xojo 2020r2 and 2019r3 with the following code:
dim startTicks as Integer = System.Ticks
GetListOfFiles(theFolderitem)
for currentFile as Integer = 0 to ListOfFiles.Ubound
if not ListOfFiles(currentFile).IsFolder then
'dim b as BinaryStream = BinaryStream.Open(ListOfFiles(currentFile))
'b.Close
dim t as TextInputStream = TextInputStream.Open(ListOfFiles(currentFile))
t.close
end if
next
dim EndTicks as Integer = system.Ticks
dim diff as Integer = EndTicks - startTicks
For about 12k files the value of diff is between 70 and 90 ticks. There is no difference between the Xojo versions and there is no difference between BinaryStream and TextInputStream.
One of the bugs I mentioned above is only visible when working with > 100k FolderItems. I did some testing with Instruments which showed me that accessing the path is what makes FolderItems slow.
1 Like
Wasn’t there a change in the API for accessing files or directories recently? As in, a 2020 or 2019 release? IIRC, this was to use the latest Apple API and therefore stop using a deprecated Apple API. This change did make file access slower, but I can’t remember which release this happened in.
If I’m remembering correctly, then there’s no use complaining about it. You’ll have to figure out how to make better use of directory access. BICBW.
The other Tim is talking about how Xojo changed macOS APIs underneath FolderItem.
iirc it doesn’t matter which Xojo API you use, you still get the new macOS API.