I need to download several thousand files via SFTP. I’ve followed the MBS “CURLS sftp batch download” example pretty closely. It works fine for 800 files, but 4,800 files just sits there forever with no TransferFinished events ever firing. I’m calling the Perform method every 10ms.
I see that there are a number of “Max” parameters:
but these are not explained well in the MBS docs and the defaults are not listed. I cannot find these parameters in the CURL project’s documentation either.
Do I need to set some or all of these parameters in order to download 4,000-10,000 files? Which parameters and to what values?
If I increase MaxTotalConnections to 5000, a few files download, then I get “Could not resolve host” (return code 6) errors for the rest. The server I’m downloading from is AWS. The files are stored in six different server directories.
The magic number is ≈900 files. If 900 or less, it works. If 905 or more, TransferFinished is never called. I haven’t found any correlation with the Max___ properties.
This is a total aggregate number among five separate CURLSMultiMBS objects, each one downloading an equal number concurrently from a different server directory, i.e. 180 files per instance works but 181 files per instance doesn’t. But I know that one individual instance works for least 800 downloads, so it seems the limitation is on total simultaneous downloads for all instances.
While it was working well under MacOS as long as I stayed under the 900-file limit, it was a disaster under Windows, with many many “login denied” errors and failures creating MTOutputFiles. I rearranged the code to download each directory-worth of files sequentially so there is never more than one instance of CURLSMultiMBS at any given time - it works OK, but slower and not as much fun to watch as the simultaneous parallel downloads.