This has been asked in the past but I can’t find a clear resolution on what’s agreed to be the best approach.
I need a fast way to scan directories that doesn’t rely on helpers. Most Xojo-specific routines go unbearably slow, compared for example with running a simple “find” command in a shell (Linux and Mac, at least) but the App Store doesn’t like apps using CLI helpers (I believe) and it feels kind of kludgy anyway.
Are there declares in each platform that could be leveraged to get a folders contents (recursive) as fast as the OS can dish them? I’d use the “find” command as a benchmark at the very least.
I’m talking about scanning whole disks, so speed is paramount. Any introduced delay multiplies as the disk is larger. I don’t expect this to be instantaneous but it’s too slow if done directly in Xojo.
Also: This should apply the same for any type of disk. Not only for internally-mounted, specific-filesystems. As long as the OS can see them, the app should see them.
Did you try FileListMBS class in MBS plugin? It’s using the OS functions.
FindFirstFile on Windows and FSOpenIterator on Mac.
The key is to avoid folderitem class for speed.
I believed wrong, then. I had one once rejected because it used helpers extensively so it may have been a matter of excess, rather than whether it used them at all
[quote=144794:@Christian Schmitz]Did you try FileListMBS class in MBS plugin? It’s using the OS functions.
FindFirstFile on Windows and FSOpenIterator on Mac.
The key is to avoid folderitem class for speed.[/quote]
Indeed, FolderItem, for all its usefulness, is a hog.
I hadn’t tried your plug-ins because I have been specifically asked to not use any third-party binaries (all source code for the app is expected to be documented in the end). I’ll see what I can find about the functions, thanks for the hint!
The following will give you a list of directory entries, very quickly (on Win32 target at least).
Function FastItems(extends f as FolderItem) As FolderItem()
#if Not TargetWin32
return nil
#endif
// Return all of the files within a folder in the quickest way possible.
// courtesy: http://forums.realsoftware.com/viewtopic.php?f=1&t=13692&st=0&sk=t&sd=a&hilit=folderItem+aaron&start=15
// With modificaitons by me to use NativePath, corrected a minor bug and ignore ".", ".." entries.
// NOTE: This is windows only....XP to present.
Soft Declare Function FindFirstFileA Lib "Kernel32" ( path as CString, data as Ptr ) as Integer
Soft Declare Function FindFirstFileW Lib "Kernel32" ( path as WString, data as Ptr ) as Integer
Soft Declare Function FindNextFileA Lib "Kernel32" ( handle as Integer, data as Ptr ) as Boolean
Soft Declare Function FindNextFileW Lib "Kernel32" ( handle as Integer, data as Ptr ) as Boolean
Declare Sub FindClose Lib "Kernel32" ( handle as Integer )
dim ret( -1 ) as FolderItem
dim handle as Integer
// Sanity check
if not f.Directory then return ret
dim data as MemoryBlock
if System.IsFunctionAvailable( "FindFirstFileW", "Kernel32" ) then
data = new MemoryBlock( 592 )
handle = FindFirstFileW( f.NativePath + "*.*", data )
else
data = new MemoryBlock( 318 )
handle = FindFirstFileA( f.NativePath + "*.*", data )
end if
if handle <> -1 then
// Loop over all of the items in using the handle and the find data
dim done as Boolean
do
// Add the current item
dim temp as FolderItem
if System.IsFunctionAvailable( "FindFirstFileW", "Kernel32" ) then
temp = GetFolderItem(f.NativePath+data.WString(44),FolderItem.PathTypeNative) //FolderItem( data.WString( 44 ), FolderItem.PathTypeAbsolute )
// Loop to the next item
done = FindNextFileW( handle, data )
else
temp = GetFolderItem(f.NativePath+data.WString(44),FolderItem.PathTypeNative) //FolderItem( data.WString( 44 ), FolderItem.PathTypeAbsolute )
// Loop to the next item
done = FindNextFileA( handle, data )
end if
// Add the current item to our list
// Ignore . and ..
Dim tmpString as String = temp.Name
If tmpString <> "." and tmpString <> ".." then
ret.Append( temp )
End If
loop until not done
FindClose( handle )
end if
return ret
End Function
I think most of that time is the Listbox. Instead of AddRow I just appended the paths to a string array and collected 134123 items in 1.77 seconds. Very Nice
That 22 seconds is with the “List paths” unchecked, so it’s just looping through the listings. I think I need more ram, or maybe my SSD is getting past it’s time…
You can also specify the options… I think you can specify whether it returns directories or just plain files, hidden files, traverse bundles… I didn’t want to get too deep though
Try a search and replace and replace initWithString with initFileURLWithPath. I reuploaded the example with the oops fixed and the class is now self contained… I might add some of the options if I have some free time.
I changed it to a module with a single method that returns an array of filepaths. It accepts options to ignore hidden files, do a shallow listing, and not include bundle contents.
I’m using the allObjects selector so that I can use redim on the output array rather than appending to the array… should be super-ultra-fast now.
if ShallowCheck.Value then options=options + NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsSubdirectoryDescendants
if NoBundlesCheck.Value then options=options + NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsPackageDescendants
if NoHiddenCheck.Value then options=options + NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsHiddenFiles
Undefined Operator Type NSDirectoryEnumerator.Options does not define “+” with Type NSDirectoryEnumerator.Options
Listbox1.DeleteAllRows
dim f As FolderItem=SelectFolder
dim options As NSDirectoryEnumerator.Options
dim res() as string= NSDirectoryEnumerator.ItemsInDirectory(f.NativePath,options)
if ListCheck.Value then
for each item as String in res
if ShallowCheck.Value then options=NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsSubdirectoryDescendants
if NoBundlesCheck.Value then options=NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsPackageDescendants
if NoHiddenCheck.Value then options=NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsHiddenFiles
Listbox1.AddRow item
next
end if
Label1.Text=str(res.Ubound+1)+" items"
Function ItemsInDirectory(mypath as string,options as NSDirectoryEnumerator.Options=NSDirectoryEnumerator.Options.Default) As string()
declare function NSClassFromString lib "Cocoa" (classname as CFStringRef) as ptr
declare function allObjects lib "Cocoa" selector "allObjects" (obj as ptr) as Ptr
declare function count lib "Cocoa" selector "count" (obj as ptr) as integer
declare function path lib "Cocoa" selector "path" (obj as ptr) as CFStringRef
declare function objectAtIndex lib "Cocoa" selector "objectAtIndex:" (obj as ptr,index as integer) as Ptr
declare function alloc lib "Cocoa" selector "alloc" (obj as ptr) as Ptr
declare sub release lib "Cocoa" selector "release" (obj as ptr)
declare function init lib "Cocoa" selector "init" (obj as ptr) as Ptr
declare function initFileURLWithPath lib "Cocoa" selector "initFileURLWithPath:" _
(obj as ptr,path as CFStringRef) as Ptr
Declare Function enumeratorAtURLincludingPropertiesForKeysoptions lib "Cocoa" _
selector "enumeratorAtURL:includingPropertiesForKeys:options:errorHandler:" _
(obj_ref as ptr,url as ptr,keys as ptr,mask as NSDirectoryEnumerator.Options,handler as ptr) as ptr
dim fm As ptr=init(alloc(NSClassFromString("NSFileManager")))
dim nsurl as ptr=initFileURLWithPath(alloc(NSClassFromString("NSURL")),mypath)
dim res() As String
dim myenum as ptr=enumeratorAtURLincludingPropertiesForKeysoptions(fm,nsurl,nil,options,nil)
dim allres As ptr=allObjects(myenum)
ReDim res(count(allres)-1)
for i as integer=0 to res.Ubound
res(i)=path(objectAtIndex(allres,i))
next
release(fm)
release(nsurl)
Return res
End Function
Xojo used to implicitly convert enums to integers and back so you could add them like that. Now you need to explicitly cast those lines, which is a mouthful…
if ShallowCheck.Value then options = NSDirectoryEnumerator.Options( integer(options) + integer(NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsSubdirectoryDescendants) )
if NoBundlesCheck.Value then options = NSDirectoryEnumerator.Options( integer(options) + integer(NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsPackageDescendants) )
if NoHiddenCheck.Value then options = NSDirectoryEnumerator.Options( integer(options) + integer(NSDirectoryEnumerator.Options.NSDirectoryEnumerationSkipsHiddenFiles) )
Hmmm - it does not even remotely come close to the speed of Thomas Tempelmann’s “Find File” tool. I wonder how he did that … http://apps.tempel.org/FindAnyFile/
