FolderItem speed in new API

I notice what looks like a severe speed reduction in the new API for Folderitem. Running this short scanning routine:

[code]Private Sub scan(SourceFolder as FolderItem)
If SourceFolder = Nil Then Return

For i As Integer = 1 To SourceFolder.Count
Dim f As FolderItem = SourceFolder.Item(i)

If f.Directory Then
  scan(f)
  
Else
  //save in dictionary
  diSourceFIles.value(f.NativePath) = f
  
End If
cnv1.invalidate

Next

End Sub
[/code]

Running in 2019 R1 - on 7881 files on my Desktop takes <0.8 seconds.
The same code in 2019 R3.1 takes 13 secs!

Apart from compiling in the old version - any clues as to how to speed this up again?

Could it be the call to Invalidate?

Would that make a difference in the 2 releases?

On macOS, Xojo is using a different macOS API under the hood (since 2019r2).

That’s slow anyway… .Count is calculated every time. You’ll find quite some threads about that.
What’s the difference in speed if you do: SouceFolder.Count DownTo 1 ?
Or at least: put .Count in a local variable before the loop, then: 1 To intCount

Using dim c as integer = sourceFolder.count
for i as integer = 1 to c …

R1 : 0.7secs
R3.1: 7.3secs

So reduced a bit - but still 10x slower in R3.1.

Read the performance considerations.

Try to see if the mentioned Array-way is faster.
Look at the new method: http://documentation.xojo.com/api/files/folderitem.html.Children

What? Read the manual? Me?

Actually that’s quite useful - changing the code to:

[code]Private Sub scan(SourceFolder as FolderItem)
If SourceFolder = Nil Then Return

Dim fs() As FolderItem

For Each f As FolderItem In SourceFolder.children

If f.Directory Then
  fs.append f
  
Else
  //save in dictionary
  diSourceFIles.value(f.NativePath) = f
  
End If
cnv1.invalidate

Next

For Each f As FolderItem In fs
scan(f)
Next

End Sub
[/code]

Gets R3.1 down to about 0.4 seconds.

Invalidate should be coalesced on macOS; so you can call it as many times as you like during processing; then at the end of the run loop it does one big update covering all the areas. Are you thinking of refresh?

How many times does it takes to move the displayed data (a click in the scrollbar) ?

Also: I noticed that starting around 1,000 files in the Desktop slow down the whole computer, running applications too.

Could you run the optimized code using the old API at the beginning of the loop?

dim n as integer = SourceFolder.Count
For i As Integer = 1 To n
Dim f As FolderItem = SourceFolder.Item(i)
etc.

I think you will get a result not far from about 0.4 seconds.

I haven’t run any tests recently (in the last few releases)…wondering if the issue with newer macs and folderitem was resolved. I have a very old MacBook (Core 2 Duo) that can run circles around my new i7 six core at opening large numbers of folderitems. The new system looks plenty speedy when you look up a thousand or so…but the more files you scan through…the slower it gets. Same code both machines. When you have a few million files you need to open/read/process (build a database) – I find that the old Macbook is faster than the decade newer hardware — and yet… for small numbers of files…the newer hardware is faster. It’s got to be a garbage collection type thing. In my tests… the new machine goes slower and slower…consuming more and more resources until it’s struggling with 100% of a cpu…while my old hardware cruises through the files between 10 and 100 times faster and the cpu never breaks a sweat at about 15% utilization. Because it starts out faster and slows WAY down…I have to assume it’s a memory leak, garbage collection, or some other thing.

No. The slowdown comes from the new and “fast” APFS. FileListMBS is way faster than the Xojo code. It’s even got a flag so that you avoid showing the beachball.

Refresh would be disastrous. I’ve never performance tested Invalidate but I would have thought there was some overhead calling it lots of times.

How many times a week do you manage 7881 files ?
(and from an unique folder)

That is the real question.

It is like saying “I need 7881 KB of sand and this takes time” (not saying you use your own small car). In that case, take a truck.
Smoke a cigarette, dring a cup or tea or coffe, etc.

Probably not that many times but from the perception of a user sitting and waiting it can be significant. This element was just the first in some optimisation I am trying to do. The previous version of my app would run this and other checks in 12 secs, but after changing the core code it is up to 60secs for those same 7881 files, so now I need to look for other causes as the Folderitem search is no longer that significant.

I didn’t know that existed - thanks.