CURLSMultiMBS Massive Speedup

I have a web app that continuously downloads listings of 18 directories residing on another server using CURLSMBS.

I was previously using CURLSMultiMBS.SharedInstance to load request curls, but I noticed that the listing times got longer and longer as the app ran, so I changed to a specific instance of CURLSMultiMBS so I could set its properties and changed the following from the defaults:

MaxHostConnections = 18
MaxTotalConnections = 18
MaxPipelineLength = 18

and wow :open_mouth: the listing times plummeted by 10-20x!

I wish the various properties of CURLSMultiMBS were better-explained in the documentation. Their meanings are only cursorily explained, no defaults are listed, and the significance of the defaults isnā€™t explained. For example, here are the defaults from the debugger:

image

Does MaxHostConnections = 0 or MaxConnects = -1 mean no limit?
Does PenaltySize = 0 mean no penalties, or everything will be penalized?

maxconnects: if >0, a fixed limit of the maximum number of entries we are allowed to grow the connection cache to.

max_host_connections: if >0, a fixed limit of the maximum number of connections per host.

max_total_connections: if >0, a fixed limit of the maximum number of connections in total.

The penalty size properties are no longer used in CURL. I may deprecate them.

So I think the only change you made is to allow more connections.

2 Likes

Thanks for the clarifications. So it seems the default MaxTotalConnections of 4 may be limiting the performance of CURLSMultiMBS.SharedInstance in some applications? Or am I wrong in assuming that the shared instanceā€™s properties canā€™t be changed?

Yes. Please increase it and it may speed up if you have a lot of parallel transfers.

What values are good ones? Could higher values lead to problems?

Back in the day when I had really slow internet more connections led RapidWeaver choke and I had to use ā€œSlowā€ or ā€œAverageā€ as far as I remember: