WebFile downloads killing web app responsiveness?

I’m looking for a quick/easy way to track the progress of downloads and/or off-load them to a separate thread from the main WebApp so that they don’t impact performance.

Should they be devastating performance?

My WebApps generally support dozens or hundreds of simultaneous users, but one particular standalone web app is bogging down to the point where it is unusable–even when there are only 8-10 users connected. The only unique thing about the problematic app is that users are downloading large (500MB-3GB) files using the native Xojo webfile download functionality–no plugins or add-ons.

Other Xojo standalone web apps running on the same server at the same time continue to be 100% responsive, and the server shows minimal resources in use (CPU/RAM/network are all around 5%).

Anecdotally, it seems the massive slow-downs happen when 5-10 files are being downloaded simultaneously, but I don’t have any concrete stats on this–it’s easy to track when downloads start, but I haven’t found a good way to track when downloads finish.

Does the native WebFile download functionality tie up the app? Any other thoughts?

I had the issue of large downloads bogging down the app. files of 500 meg in size or so. We didn’t solve the issue. We worked around it by dumping the files to disk and having Apache handle the downloads. We did some trickery to move the file to the /web directory on Xojo Cloud. And we delete the files after X hours of being available.

Side note: Our client reported that the “time bomb” nature of the downloads actually increased the responsiveness of their users! (“We better get that file while we still can!”)

Now if we can just solve the large upload issues…

Web apps are single threaded and IO functions are synchronous and reading from disk is slow.

If your app is reading files from disk real slowly and it cannot do anything else while waits then your backlog of requests is going to get longer and longer until your app essentially slows to a crawl. Eventually connections start dropping.

Solution #1: Store as much in cache as you can.

Solution #2: Load balance your app.

Solution #3: Don’t use Xojo Web to serve large files.

Jay, for what it’s worth, after trying a half-dozen (or more) upload solutions, the only one that solved 100% of my upload issues (lots of users simultaneously uploading multiple 2+ GB files) was Phillip’s plupload ( https://forum.xojo.com/41644-open-source-upload-control-for-xojo-web ) although I still consider it a temporary work-around since I had to get php running on the server for that, which isn’t ideal to me.

Phillip,

I’m fortunate that I’m running on heavy-duty server and storage (everything connected by 10GB links and capable of handling thousands of IOPS), and the loads on both the server and storage are barely measurable (typically less than 1-3% on the server and typical storage read latency of 1-3ms rarely peaking over 10ms.

So it seems like the single-thread issue is the culprit, and it has me wondering if I might be able to keep the solution 100% Xojo-based by offloading the downloads to another thread (or at the worst, forward it to a separate app).

Any thoughts on putting the download call into its own thread? Would that even work effectively in a Xojo web app?

[quote=359316:@Seth Ober]

Any thoughts on putting the download call into its own thread? Would that even work effectively in a Xojo web app?[/quote]

Unfortunately that would not work. Xojo threads are cooperative and as such would not solve the issue of synchronous tasks and growing requests from impacting user experience.

Your best option is just to use a ‘real’ webserver to serve the big files I think.

A 4th option is to use a storage service like S3. You can configure your web app to upload files directly to S3 and present links to download the files from S3. Avoid the Xojo web server entirely.

I’m using S3 with our Xanadu app and it’s amazing. I calculate the S3 path and use a web form that uploads directly to S# in a frame. Once uploaded, the file is shown using the S3 url to the file.