Several HTTPSecureSockets limit?

  1. 2 months ago

    Hi, Is there a Limitation about downloading several files, one file per socket?
    I have a String array with 758 Links
    And I created a HTTPSecureSocket array that append as many items as items has the String Array.

    And I tested it, checking the Ubound of the HTTPSecureSocket
    and truly its working.

    But dispite of know the number of sockets created and with its respective Handlers, Only Downloads 496 of 758 Links.
    I wonder if Is it a Xojo Limitation, or is a Server Limitation.
    Have anyone faced similar situation?

    Regards
    :D

  2. Kevin W

    Aug 8 Pre-Release Testers, Xojo Pro
    Edited 2 months ago by Kevin W

    There are system limits that might affect this, but I'm pretty sure it's a bit higher than 758. Did you have a lot of other things open at the time?

    Is it possible that something else is limiting your connections? Sometimes ISPs impose connection limits, firewalls might do that?

    Does it always stop at the same number? I wouldn't expect it to be that deterministic if it's some kind of system or network limit. I'd expect it to vary a little bit at least.

    Are you logging socket errors? Anything interesting there?

  3. @Kevin W There are system limits that might affect this, but I'm pretty sure it's a bit higher than 758. Did you have a lot of other things open at the time?

    Is it possible that something else is limiting your connections? Sometimes ISPs impose connection limits, firewalls might do that?

    Does it always stop at the same number? I wouldn't expect it to be that deterministic if it's some kind of system or network limit. I'd expect it to vary a little bit at least.

    Are you logging socket errors? Anything interesting there?

    Hi Kevin, In fact I'm realizing that I'm using a For each to parse all my HTTPSecureSockets, but one curious thing happens when I put a Msgbox at every Item on "FOR", and when I do this, all the 758 files are downloaded.

    Might be is right that you told me, I'll be investigating.

    Regards

  4. Tim H

    Aug 8 Pre-Release Testers Portland, OR USA

    That sounds like the server protecting itself from what it sees as an attack.

  5. Kevin W

    Aug 8 Pre-Release Testers, Xojo Pro

    How are you creating the sockets? Is it possible that some of the sockets are being allowed to go out of scope and the reference counter is reclaiming them before they've finished their work?

  6. Mattias S

    Aug 8 Pre-Release Testers, Xojo Pro Europe (Sweden, Stockholm)

    Don't know what OS you are running, but on Windows there is a limit in the maximum file handles that can be opened. Maybe you are running into this with so many downloads. When adding a msgbox some downloads may close the file handle and allow others to continue? Just thinking...

  7. @Mattias Söm Don't know what OS you are running, but on Windows there is a limit in the maximum file handles that can be opened. Maybe you are running into this with so many downloads. When adding a msgbox some downloads may close the file handle and allow others to continue? Just thinking...

    Im running on macOS, but the goal is to run on both platforms

  8. Mattias S

    Aug 8 Pre-Release Testers, Xojo Pro Europe (Sweden, Stockholm)

    On Windows an application normally only supports a maximum of 512 open files at the same time, so maybe you need to do some throttling (remember that the EXE itself opens many support files).

  9. @Mattias Söm On Windows an application normally only supports a maximum of 512 open files at the same time, so maybe you need to do some throttling (remember that the EXE itself opens many support files).

    Ok, I take note of that

  10. Kevin G

    Aug 8 Pre-Release Testers, Xojo Pro Gatesheed, England

    I suggest that you don't attempt to transfer that many at once as you will more than likely run into limitations / restrictions either on the client, firewall or server. 10 - 20 concurrent transfers may be a good number to start at.

    FYI. The last time I used the built-in HTTP(s) classes they performed poorly both in performance and CPU overhead when performing multiple transfers. I switched to the MBS CURL plugin which made things a lot better.

  11. Greg O

    Aug 9 Xojo Inc Somewhere near Raleigh, NC
    Edited 2 months ago by Greg O

    With all of that said, remember that your computer can only handle so much bandwidth anyway, especially if you are writing these files to disk. There may also be limits on the number of concurrent connections from a sinhle IP on the server as well.

    Try having only 3-4 sockets going at once and you'll probably see nearly the same overall speed as trying to do them all at once.

  12. @Greg OLone With all of that said, remember that your computer can only handle so much bandwidth anyway, especially if you are writing these files to disk. There may also be limits on the number of concurrent connections from a sinhle IP on the server as well.

    Try having only 3-4 sockets going at once and you'll probably see nearly the same overall speed as trying to do them all at once.

    Ok, I'll gonna with 4 sockets, one queue Download per socket. :D

or Sign Up to reply!