CurlsMBS.Progress

When I used to use CURLSMBS individually in threads, the Progress event would fire pretty frequently and I could display a smooth and informative ProgressBar. Using CURLSMultiMBS.SharedInstance (which I love), however, it doesn’t seem to fire until the transfer is complete when using the default buffer size. I can get it to fire more frequently by setting OptionBufferSize to a small value, but that seems to slow down the transfer unnecessarily and results in the bar moving in uniform increments that don’t seem “real”.

Don’t know if this is a limitation of the CURLS library or if it could be improved in the plugin. Maybe the SharedInstance could have its own Progress event…

Not sure. Progress is usually reduced to once a second to avoid to frequent updates.

But otherwise we don’t do something special.

My download takes 4 seconds and Progress doesn’t seem to fire until it’s complete.

Sorry, difficult to say.

Could you log debug messages from that transfer with times, so you see what happens when.

Sometime the download is quick, but the server needs a second to connect and to find the data. So time to connect and query is long and download is short.

Seems like Progress is firing once per second (with DebugLog resolution), but the arguments aren’t changing.

This is true, but if that’s the case then why is it firing with no change to dlTotal and dlNow? It should fire only when actual progress has occurred, no?

It may fire while the query is sent and we wait.

What do DebugMessage event shows?
Like if you log that, too.
Maybe the 3 second delay is SSL setup?

DebugMessages and DebugData are both empty :woman_shrugging:

No, I mean to use the event where you get debug messages live and then also log them with the time.

And in DebugMessage only log if infotype < 5.

It never fires.

Is curl.OptionVerbose=True?

1 Like

Verbose should be enabled automatically in newer plugins.

I tried a modified example and it works fine here:

So we can live process debug messages and show them.
As well as show the progress.
I had to limit progress to actually get a meaningful progress as this would otherwise run much to quickly!

Public Sub DownloadOneURL()
  If UBound(urls) = -1 Then Return
  
  Dim URL As String = URLs.Pop
  
  Dim c As New MyCURLSMBS
  
  // setup request and any options you need
  c.OptionURL = URL
  c.OptionUserAgent = "MyDownloader 1.0"
  c.OptionMaxRecvSpeed = 2000
  
  If CURLSMultiMBS.SharedInstance.AddCURL(c, AddressOf TransferFinished) Then
    Log "Downloading: "+URL
  else
    Break
  end if
  
End Sub

I did not have Verbose enabled, sorry. Now I get

12:56:17 PM: Starting Multi
12:56:17 PM: Handle progress 0 of 0
[various authentication messages]
12:56:17 Authentication complete
12:56:18 PM : Handle progress 0 of 0
12:56:19 PM : Handle progress 0 of 0
12:56:20 PM : Handle progress 0 of 0
12:56:20 [my big chunk of downloaded string data]
12:56:21 Connection #0 to host ftp…com left intact

During the 4 seconds of downloading after authentication reported completed, HandleProgress fired 3 times, with 0 bytes reported each time.

Are you getting a compressed response by chance?

I just recently ran into a very similar issue with URLConnection’s data received event having useless parameter values. It turned out that Xojo was adding a header to ask for compressed response and this was borking the framework.

This could explain why it works for some tests, but not your use case (the URL / response is different).

Not that I’m aware of… the data is present in the DebugMessage event in human-readable string form. But it is an SFTP transfer, FWIW.

Hm, I’m not sure then. It seemed similar so I thought it worth mentioning.

1 Like

Are you sure it’s using SFTP? The url suggests FTP(S). They’re not interchangeable.

Yep, pretty sure :slight_smile:

curl.OptionPort = 22
curl.OptionSSHPrivateKeyfile = keyfile.NativePath
curl.OptionSSHAuthTypes = 1 // CURLSSH_AUTH_PUBLICKEY
curl.OptionDefaultProtocol = "SFTP"

It’s an Amazon S3 bucket. The URL is just “ftp.myclient.com” and somehow it automagically selects the target “directory” based on the username - I don’t really understand the details, but for sure it’s SFTP - I don’t think S3 even supports FTP.

@Christian_Schmitz sent me his sample project but there’s no magic there, it’s the same as what I do except that I’m using SFTP instead of his http download.

I know my file is small and only takes a few seconds to download, but it’d be nice to have a progressbar even if it goes quickly. I guess I’ll have to use a wheel, ugh.

I think I can tune the plugin calling progress more often.
But for some SFTP operations, the progress may not be reported.

I do See progress here:

Please try next pre-release.

1 Like