FileListMBS Threaded vs. YieldTicks

What are the benefits/differences of using Threaded vs. YieldTicks for FileListMBS? So far I’ve used YieldTicks which worked fine for a medium number of selected files. For a large number of files (>100k or so) I still saw the beachball.

When using Threaded to do a FileListMBS for all Enron emails I still see the beachball. So no improvement over using YieldTicks.

Threaded will do the wirk in a preemptive thread.

If it blocks for you, please analyze the process to let me know where it hangs.

We requested the feature as FileListMBS would beachball when scanning a slow SMB share. The feature definitely improves the situation and we haven’t seen the beach all since.

Are you sure you are using FileListMBS within a Xojo thread?

Yes, there is a thread.

  • HFS
  • macOS High Sierra (yes, I know it’s old)
  • external USB 2 hard disk
  • 517 k files, most files are small, I use the Enron email corpus for testing

I’ll do more testing on the MacBook Air M1.

Well, the constructor is threaded. Your code is later and call item() function, which uses GetTrueFolderItem to build a Xojo folderitem for it.

The key of FileListMBS is to avoid FolderItem as much as possible.

Well, I need a Folderitem as result. What can I do instead? The code isn’t really complicated:

'nothing to do
if StartFolderitem = nil then 
  isRunning = False
  Return
end if
if not isRunning then Return

dim theFileList as new FileListMBS(StartFolderitem)
theFileList.Threaded = True
dim FileCount as Integer = theFileList.Count - 1
for currentFile as Integer = 0 to FileCount
  
  if not theFileList.Visible(currentFile) then
    if theFileList.TrueItem(currentFile).FileIsOnlyIcloud then
      theError = kErrorCloud.Replace("<<filename>>", theFileList.TrueItem(currentFile).DisplayPathMBS(" » "))
      isRunning = False
      exit for
    end if
  end if
  
  if theFileList.Directory(currentFile) then
    ListOfFiles.Add theFileList.TrueItem(currentFile)
    GetListOfFiles theFileList.TrueItem(currentFile)
  else
    dim FileEnding as String = theFileList.TrueItem(currentFile).NameExtensionMBS
    dim ParentFileEnding as String = theFileList.TrueItem(currentFile).Parent.NameExtensionMBS
    'if the parent is an sbd file then I'm archiving directly from Thunderbird, then msf files are not needed
    if EmailFileEndings.IndexOf(FileEnding) > -1 or (ParentFileEnding = "sbd" and FileEnding <> "msf") or theFileList.TrueItem(currentFile).Name = "mbox" then
      ListOfFiles.Add theFileList.TrueItem(currentFile)
    end if
  end if
  
next

Before doing anything I need - some - reference to the file. Afterwards the files are read which can take longer. But I need a list of the files before doing anything else.

Then you are better of using FolderItem and not FileListMBS for this.

FileListMBS is much faster than iterating over lots of files.

It looks like you are calling TrueItem lots of times for each entry. Have you tried putting TrueItem into a local variable and adjusting the code to use that?

sorry for missing that.
theFileList.TrueItem(currentFile) should be cached in a local variable to save time.

Also “theFileList.TrueItem(currentFile).Parent” should not be needed as you have the parent on the beginning.

I am curious to see a comparison in execution time before and after optimizing the code.

The time went from about 50 seconds to 30. Still seeing the beachball, though.

As you’ve managed to improve the efficiency you might be able to mask the problem by reducing the Xojo thread priority.

I think the only other solution would be to use a worker as you’ve hit the limit of Xojo’s primitive threading capabilities.

I’ve already tried to use workers. Because they are so super primitive I didn’t come very far. Getting the result back to the main app choked the app. When the bug (https://tracker.xojo.com/xojoinc/xojo/-/issues/69248) is fixed. Then I can try again.