Xojo web app and bandwidth

Targets Web
1

I have never built a web app with Xojo, just desktop, but I have built a lot of web sites in the past 30 years. We’re working on a web based project that would involve customers uploading very large (tens of gigabytes, potentially up to 1-2 terabytes) of files from their desktop to an S3-compatible bucket. We would store these files with Wasabi rather than Amazon, since they have better pricing and don’t charge ingress/egress fees. The kind of files being stored would rarely be accessed, so we’d be well within their free egress policy.

The issue is getting the files to Wasabi. In some cases (many, actually), we’d upload the files to the S3 bucket from our office via a desktop app. This would be a direct connection to Wasabi. But when customers upload their own files from their desktop, we run into issues. Unfortunately, Wasabi doesn’t support CORS, which means when the customer uploads their own files through our web site (likely built on Drupal), the data has to go from their desktop->our site->S3 Bucket. That means we’re dinged twice on the bandwidth costs, and based on our projections, these are significant fees. We want to avoid this. (and hosting it at Amazon is cost-prohibitive, driving the cost of our service to too high a level for the customer base we’re targeting).

One option is to make a desktop application that the customer downloads, which uses the Wasabi API to directly upload the file to them, bypassing our web server. This is relatively easy to do, and would make for a simple, secure method of uploading the files. But there’s a fair bit of friction here, which I think would be hard for some users to deal with. (many of these customers are older and are not tech-savvy).

Another option is to embed some kind of app hosted elsewhere, such as within Xojo Cloud, within our site, so that the bandwidth is used elsewhere, even though it looks to the user like they’re uploading through the site. There are reasons we want to keep our web site where it is, but we could put the upload/download stuff in an embedded xojo app within our site, with that app being hosted on xojo cloud, no?

What are the costs for bandwidth overages with Xojo cloud? We would need minimal space, but maximal bandwidth in this scenario.

How reliable is Xojo cloud? How’s the speed? Is there a better way to make this seamless than what I’ve outlined?

2

Last month, we just developed an application to manage medical documents. That means uploading files for their consultations, analysis, and laboratory tests.

XOJO Cloud worked very well for me and the POSTGRES database supports many files. I have not tried to have 1 to 2 terabytes saved, but in theory, if there is not much query to download these, it supports it well.

I see your project as possible in XOJO Cloud. The speed depends a lot on the capacity of the server you order. To give you an idea, the capacity of a medium server without much traffic on the network uploads a relatively large file very quickly.

Hope it works, although my answer is very general.

3

Thanks. So to clarify, the bulk of the files would never live on our web server or on Xojo cloud. They’d live at Wasabi, which is an Amazon S3 compatible service. The trick is to avoid routing the files through our web server to get from a client browser to the S3 bucket, and (eventually) back to the customer when they want to restore a file.

4

Please talk to us before creating a Xojo Cloud app that uses this much bandwidth. We will need to make special provisions for a server like this and there may be significant bandwidth charges as it sounds like you’re expecting to exceed the monthly bandwidth allotment in a fraction of a day.

1 Like
5

Not at all. The vast majority of the files would be uploaded direct from our office to Wasabi, without ever touching the web server or Xojo cloud. We’re talking about how to handle cases where customers upload their own files (only a fraction of the number of customers), or when they download some of all of the files in their personal bucket from the site. The latter is an infrequent occurrence. This is about archival storage backups, not frequently accessed data.

6

Are you sure you can create a Xojo web app that will connect your user directly from their computer to Wasabi without passing where the web app is running? (Xojo cloud or another server)

As you said, a local application is possible but I don’t know if a web app can do that.

1 Like
7

Depending on the storage medium, you may find that there’s a way to create one-time or time-limited protected links to files which you could send to the browser in a WebLink. We do something similar with prerelease downloads from our CDN provider. Then the web app wouldn’t be involved in the file transfer at all.

If any of these files are as large as you indicate and are going to go through XojoCloud, they’re going to fail outright on a default server. When files are uploaded, they need to go somewhere, even temporarily, so you’d need enough drive space to take in a file even before you got a chance to process it, which you’d likely not be able to do because of their pure size. I can tell you for a fact that we didn’t consider files that big when designing the web server and file uploader components. Now it may work if you had enough disk space, but I wouldn’t count on it without extensive testing.

But what I meant about bandwidth is that we only include 2-6 TB of bandwidth per month, depending on the size of the server purchased. With huge files, you could easily eat that up.

8

No, this is not what I’m talking about - the web app would have to route the data through Xojo Cloud.

A desktop app would connect direct to Wasabi, which is how most of the files would be uploaded (from our office). The idea is to avoid

I don’t think Wasabi offers this for private files (which these would be), but only for publicly available files. I can look into that though, as it would save a fair bit of bandwidth on the download side.

The way this should be done is with CORS. While Wasabi is “S3 compatible” they’re not 100% S3 compatible in that they don’t support this protocol, which Amazon S3 does. CORS would allow us to easily get/put files in the Wasabi S3 bucket, without ever touching a web server at all. I’m not sure why they don’t support this, to be honest, but they’ve been saying for 3 years that it’s on their list. They haven’t implemented it though.

File sets could be 1-2TB. The vast majority will be under 250GB though. Within that, individual files would most commonly be in the 5-20GB range. These would be segmented for multi-part upload, into small, manageable chunks.

9

I would recommend that you find either a web hoster which gives you unlimited bandwith like contabo.de (which I can highly recommend).

Bildschirmfoto 2021-06-10 um 15.31.53
Bildschirmfoto 2021-06-10 um 15.31.531256×308 25.7 KB

Or you upload the files from the devices it self and try to fiddle around all the possible problems.

2 Likes
10

Well, the WebUploader is not even working…

And the very nature of Xojo being single threaded can cause many problems when uploading large files.

1 Like
11

Another reason to upload the files directly from the device rather than over a web app:
If you’re using S3, then you can use its CLI, which is incredibly good. So you even don’t need to write your own upload code.