Quick way to get file paths of all files/folders in a folder, recursively

General
1

I need a way to get the native paths to all files and folders, recursively, in a target folder into an in-memory database. In some cases it might only be a couple hundred. In others it might be a couple hundred thousand. In my current app, built many years ago, I walk through the folders and it works, but it’s not super fast. I’m refreshing that app and want to make this part quicker. In my testing I tried an MBS plugin (can’t remember which) to see how it worked and it was very quick. But I don’t want to rely on plugins, so I’m looking for some other way to do it.

The app is for Mac and Windows (primarily mac, but needs to work on both), for in-house use. I’m not selling this app so I don’t really care about how I do it as long as it’s accurate and it works - could be a native Xojo algorithm, an OS-level native function call, or an external library. Doesn’t matter to me as long as it works on both platforms. Bonus if I can also (without a massive performance hit) get a little metadata about the files - particularly the file size.

2

One option would be to use a shell and use DIR on Windows & ls -al on macOS then parse the result. This is way faster than iterating a folder using folderitem & gives you the size of the file.

1 Like
3

funny, I’ve always done it as “ls -la” for like 30+ years so I had to think about that for a second!

That returns the results of the current working directory though, so I still have to manually traverse all the directories and subdirectories and run that in each. Ideally I’d like to find something that returns a list of full paths to files from the root folder of my search, kind of like what:

find * ./

…would do, but also with the file sizes. That’s incredibly fast on a mac, but I don’t know what the equivalent to that is on Windows or if there’s anything as fast.

4

I did find this in my searches and just ran it on a really substantial folder. It’s quite fast. But I’d need a windows equivalent, and obviously I’d pare this down because there’s extra stuff I don’t really need. macos - Recursive file list of all files, list containing path+name, type, size, creation data, modification date - Stack Overflow

#!/bin/bash

find . -type f -name '*' -print0 | while IFS= read -r -d '' file
do
    name=$(basename "$file")
    path=$(dirname "$file")
    # full_path=$(readlink -f "$file") # This only works on Linux
    full_path=$(echo "$PWD/${file#./}")
    extension=${name##*.}
    size_human_readable=$(ls -lh "$file" | awk -F' ' '{print $5}')
    size_in_bytes=$(stat -f "%z" "$file")
    creation_date=$(stat -f "%SB" "$file")
    last_access=$(stat -f "%Sa" "$file")
    last_modification=$(stat -f "%Sm" "$file")
    last_change=$(stat -f "%Sc" "$file")

    printf "[$file]:\n"
    printf "\tfile name:\t\t$name\n"
    printf "\tfile path:\t\t$path\n"
    printf "\tfull file path:\t\t$full_path\n"
    printf "\tfile extension:\t\t$extension\n"
    printf "\tfile size:\t\t$size_human_readable\n"
    printf "\tfile size in bytes:\t$size_in_bytes\n"
    printf "\tfile creation date:\t$creation_date\n"
    printf "\tlast file access:\t$last_access\n"
    printf "\tlast file modification:\t$last_modification\n"
    printf "\tlast file change:\t$last_change\n"
    printf "\n"
done
5

How about DIR /s?

Where f is your selected folder use:

Var sh As New Shell

Var cmd As String = f.NativePath.Left(2) _ ' Select the Drive
 + " && cd " + f.NativePath.Middle(2) _ ' Change to the selected folder
 + " && dir /s" ' recursively list the contents

sh.Execute(cmd)

sh.result can be parsed into your in-memory database.

1 Like
6

thanks. I’ll give this a try tomorrow when I’m back in the office.

7

Haven’t had a chance to try the windows DIR command yet, but on the mac, I loaded a folder with 48,358 items in it in about 5 seconds using the find command. That’s a massive improvement.

8

I use the Chilkat plugin for this. The DirTree class doesn’t need a paid license to use.

Test app attached.
DirSpeedTester.xojo_binary_project.zip (9.8 KB)