[Declares] Fastest way to scan folders?

This has been asked in the past but I can’t find a clear resolution on what’s agreed to be the best approach.

I need a fast way to scan directories that doesn’t rely on helpers. Most Xojo-specific routines go unbearably slow, compared for example with running a simple “find” command in a shell (Linux and Mac, at least) but the App Store doesn’t like apps using CLI helpers (I believe) and it feels kind of kludgy anyway.

Are there declares in each platform that could be leveraged to get a folders contents (recursive) as fast as the OS can dish them? I’d use the “find” command as a benchmark at the very least.

I’m talking about scanning whole disks, so speed is paramount. Any introduced delay multiplies as the disk is larger. I don’t expect this to be instantaneous but it’s too slow if done directly in Xojo.

Also: This should apply the same for any type of disk. Not only for internally-mounted, specific-filesystems. As long as the OS can see them, the app should see them.

Did you try FileListMBS class in MBS plugin? It’s using the OS functions.
FindFirstFile on Windows and FSOpenIterator on Mac.
The key is to avoid folderitem class for speed.

Not true. Almost all my apps use shell. I even have one that is a GUI to a shell command.

I believed wrong, then. I had one once rejected because it used helpers extensively so it may have been a matter of excess, rather than whether it used them at all :slight_smile:

[quote=144794:@Christian Schmitz]Did you try FileListMBS class in MBS plugin? It’s using the OS functions.
FindFirstFile on Windows and FSOpenIterator on Mac.
The key is to avoid folderitem class for speed.[/quote]

Indeed, FolderItem, for all its usefulness, is a hog.

I hadn’t tried your plug-ins because I have been specifically asked to not use any third-party binaries (all source code for the app is expected to be documented in the end). I’ll see what I can find about the functions, thanks for the hint!

FSOpenIterator is deprecated in 10.8, maybe look at NSDirectoryEnumerator?

Could also be that the reviewer was more stupid than usual :wink:

The following will give you a list of directory entries, very quickly (on Win32 target at least).

Function FastItems(extends f as FolderItem) As FolderItem()
  #if Not TargetWin32
    return nil
   #endif
    // Return all of the files within a folder in the quickest way possible.
    // courtesy: http://forums.realsoftware.com/viewtopic.php?f=1&t=13692&st=0&sk=t&sd=a&hilit=folderItem+aaron&start=15
    // With modificaitons by me to use NativePath, corrected a minor bug and ignore ".", ".." entries.
    // NOTE: This is windows only....XP to present.
    
    Soft Declare Function FindFirstFileA Lib "Kernel32" ( path as CString, data as Ptr ) as Integer
    Soft Declare Function FindFirstFileW Lib "Kernel32" ( path as WString, data as Ptr ) as Integer
    Soft Declare Function FindNextFileA Lib "Kernel32" ( handle as Integer, data as Ptr ) as Boolean
    Soft Declare Function FindNextFileW Lib "Kernel32" ( handle as Integer, data as Ptr ) as Boolean
    Declare Sub FindClose Lib "Kernel32" ( handle as Integer )
    
    dim ret( -1 ) as FolderItem
    dim handle as Integer
    
    // Sanity check
    if not f.Directory then return ret
    
    dim data as MemoryBlock
    if System.IsFunctionAvailable( "FindFirstFileW", "Kernel32" ) then
      data = new MemoryBlock( 592 )
      handle = FindFirstFileW( f.NativePath + "*.*", data )
    else
      data = new MemoryBlock( 318 )
      handle = FindFirstFileA( f.NativePath + "*.*", data )
    end if
    
    if handle <> -1 then
      // Loop over all of the items in using the handle and the find data
      dim done as Boolean
      do
        // Add the current item
        dim temp as FolderItem
        if System.IsFunctionAvailable( "FindFirstFileW", "Kernel32" ) then
          temp = GetFolderItem(f.NativePath+data.WString(44),FolderItem.PathTypeNative)   //FolderItem( data.WString( 44 ), FolderItem.PathTypeAbsolute )
          
          // Loop to the next item
          done = FindNextFileW( handle, data )
        else
          temp = GetFolderItem(f.NativePath+data.WString(44),FolderItem.PathTypeNative)   //FolderItem( data.WString( 44 ), FolderItem.PathTypeAbsolute )
          
          // Loop to the next item
          done = FindNextFileA( handle, data )
        end if
        
        // Add the current item to our list
        // Ignore . and ..
        Dim tmpString as String = temp.Name
        If tmpString <> "." and tmpString <> ".." then
          ret.Append( temp )
        End If
      loop until not done
      
      FindClose( handle )
    end if
    
    return ret
End Function

Good point. Will check it out instead.

Funny how nobody would disagree with the initial premise in this sentence.

I put together an example NSDirectoryEnumerator module to see how fast it is… pretty fast. (enumerated 279,000 items in 22 seconds)

DirectoryListing.xojo_binary_project

I think most of that time is the Listbox. Instead of AddRow I just appended the paths to a string array and collected 134123 items in 1.77 seconds. Very Nice :slight_smile:

That 22 seconds is with the “List paths” unchecked, so it’s just looping through the listings. I think I need more ram, or maybe my SSD is getting past it’s time…

You can also specify the options… I think you can specify whether it returns directories or just plain files, hidden files, traverse bundles… I didn’t want to get too deep though :wink:

Nice work Jim!

I’m getting an ObjCException if I select certain directories (copied over from my old Mac for some reason).

and this is the line it errors on:
[quote]nsenum.handle=enumeratorAtURLincludingPropertiesForKeysoptions(fm,nsurl,nil,0,nil)[/quote]

Try a search and replace and replace initWithString with initFileURLWithPath. I reuploaded the example with the oops fixed and the class is now self contained… I might add some of the options if I have some free time.

Ok, last iteration…
DirectoryListing.xojo_binary_project

I changed it to a module with a single method that returns an array of filepaths. It accepts options to ignore hidden files, do a shallow listing, and not include bundle contents.
I’m using the allObjects selector so that I can use redim on the output array rather than appending to the array… should be super-ultra-fast now. :slight_smile:

It’s working brilliantly now…thank you!

it no longer works in Xojo2014R3

error on

  if ShallowCheck.Value then options=options + NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsSubdirectoryDescendants
  if NoBundlesCheck.Value then options=options + NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsPackageDescendants
  if NoHiddenCheck.Value then options=options + NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsHiddenFiles

Undefined Operator Type NSDirectoryEnumerator.Options does not define “+” with Type NSDirectoryEnumerator.Options

got it working in Xojo2014R3

PushButton1

  Listbox1.DeleteAllRows
  
  dim f As FolderItem=SelectFolder
  dim options As NSDirectoryEnumerator.Options
  dim res() as string= NSDirectoryEnumerator.ItemsInDirectory(f.NativePath,options)

  if ListCheck.Value then 
    for each item as String in res
      if ShallowCheck.Value then options=NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsSubdirectoryDescendants
      if NoBundlesCheck.Value then options=NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsPackageDescendants
      if NoHiddenCheck.Value then options=NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsHiddenFiles 
      Listbox1.AddRow item
    next
  end if
  Label1.Text=str(res.Ubound+1)+" items"
Function ItemsInDirectory(mypath as string,options as NSDirectoryEnumerator.Options=NSDirectoryEnumerator.Options.Default) As string()
  declare function NSClassFromString lib "Cocoa" (classname as CFStringRef) as ptr
  declare function allObjects   lib "Cocoa" selector "allObjects" (obj as ptr) as Ptr
  declare function count   lib "Cocoa" selector "count" (obj as ptr) as integer
  declare function path lib "Cocoa" selector "path" (obj as ptr) as CFStringRef
  declare function objectAtIndex   lib "Cocoa" selector "objectAtIndex:" (obj as ptr,index as integer) as Ptr
  declare function alloc   lib "Cocoa" selector "alloc" (obj as ptr) as Ptr
  declare sub release   lib "Cocoa" selector "release" (obj as ptr)
  declare function init   lib "Cocoa" selector "init" (obj as ptr) as Ptr
  declare function initFileURLWithPath   lib "Cocoa" selector "initFileURLWithPath:" _
  (obj as ptr,path as CFStringRef) as Ptr
  Declare Function enumeratorAtURLincludingPropertiesForKeysoptions lib "Cocoa" _
  selector "enumeratorAtURL:includingPropertiesForKeys:options:errorHandler:" _
  (obj_ref as ptr,url as ptr,keys as ptr,mask as NSDirectoryEnumerator.Options,handler as ptr) as ptr
  
  dim fm As ptr=init(alloc(NSClassFromString("NSFileManager")))
  dim nsurl as ptr=initFileURLWithPath(alloc(NSClassFromString("NSURL")),mypath)
  dim res() As String
  dim myenum as ptr=enumeratorAtURLincludingPropertiesForKeysoptions(fm,nsurl,nil,options,nil)
  dim allres As ptr=allObjects(myenum)
  ReDim res(count(allres)-1)
  
  for i as integer=0 to res.Ubound
    res(i)=path(objectAtIndex(allres,i))
  next
  
  release(fm)
  release(nsurl)
  
  Return res
End Function

Xojo used to implicitly convert enums to integers and back so you could add them like that. Now you need to explicitly cast those lines, which is a mouthful…

if ShallowCheck.Value then options = NSDirectoryEnumerator.Options( integer(options) + integer(NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsSubdirectoryDescendants) ) if NoBundlesCheck.Value then options = NSDirectoryEnumerator.Options( integer(options) + integer(NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsPackageDescendants) ) if NoHiddenCheck.Value then options = NSDirectoryEnumerator.Options( integer(options) + integer(NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsHiddenFiles) )

Or maybe there’s a way to define + on enums?

I made an example of the code

ListFiles_OSX

Hmmm - it does not even remotely come close to the speed of Thomas Tempelmann’s “Find File” tool. I wonder how he did that …
http://apps.tempel.org/FindAnyFile/