Best practices to install a large amount of data

In the past, I’ve just had my app unpack a few zip files containing support files, empty sqlite databases etc out of my app bundle’s Resources folder and copied them to Application Support.

But on a new app I’ve got a large amount of data to put in Application Support, about 2 GB worth, and keeping an extra copy around in my app’s Resources folder seems like a waste.

Since I can’t delete it from my resources folder without killing the signature/notarization, what is the best approach to install this data without wasting the user’s disk space?

Download it on demand.

3 Likes

Will the data change? Is there a reason you can’t just read it out of your own Resources folder… or segregate it into read-only and writable data?

This is one of my personal pet peeves: applications that barf huge amounts of data all over the place, consuming space in a way that is difficult for the user to understand. It also complicates the process of uninstalling the app.

1 Like

The data will be modified over time and must be in a folder with write privileges. And I would love to download it on demand, but it’s free software and infrastructure to do that will get very expensive very quickly. For now I will distribute the binary via Google Drive but on-demand downloads are not sustainable on free platforms for that amount of data.

What kind of data is it?

Sqlite database files, mostly text data.

Compacted or not ?
Text data compacted could be much smaller.

Yes, compacted when I distribute it. Zipped it’s about 600 mb.

You’re paying for the bandwidth one way or another. Either it’s included with the original download, or it’s downloaded on demand, either way it’s getting downloaded. I serve around 2TB per month from multiple data centers and it costs me around $25 with Bunny.net.

But this kind of stuff is why maybe it shouldn’t be free? I released a free app that I had made for myself, but after a while of developing and supporting it, I began to get burned out. I was putting lots of time and money into a hobby. I decided to add a paid version to cover my costs. I figured I’d be happy if I made $300 from it. I sold $1800 my first month, and it changed my life. It’s now my primary source of income by a wide margin.

Granted, my story isn’t typical, but you have to decide what this is worth to you. Nothing is free. You’re paying for your software with your time and money.

4 Likes

You know you do not have to do that for sqlite database, but let your application create them when needed.

Definitely the way to do it.

Unfortunately due to the license of parts that I’m using the software must remain free. And yes, you’re correct, you “pay” for the data either way, but at least for now, it is perfectly capable of being hosted on a free Google Drive for the < 100 people that will ever download it.

Here is what I’ve come up with so far. Sorry for the crude sketch but I haven’t actually designed anything yet for it. Is this an acceptable user experience?

Wouldn’t it be easier for the user to create an installer?

You will definitely have some people that double click the app from the DMG, others that copy the app but not the data folder…

There’s not a good solution to this problem given your constraints. Here are your options as I see it:

  1. Pay a small fee for a CDN and download on demand.
  2. Include the data inside the app and copy it into AppSupport. This costs the user disk space, but saves you tech support costs.
  3. Use an installer to put the app and data in the desired places. The user can delete the installer when finished, if it doesn’t delete itself.
  4. Separate the app and data. Users will mess this up, so it’ll drive up tech support costs.

And to be clear, “tech support costs” is your time and user frustration in this case, I don’t mean literal money.

Of those options, the installer is probably the most user friendly option.

1 Like

I think an installer is the best way. What is the go-to method for creating an installer these days? I used to use Packages but it stopped working a few MacOS versions ago (I think there was an issue with one of the panels not rendering or something). I suppose I could create an installer in Xojo if I had to.

Packages are still the correct installer technique for macOS. They are produced with the pkgbuild command line tool. Like your app, an installer should be notarized, but that is done with a “Developer ID Installer” certificate instead of the “Developer ID Application” certificate. Do not attempt to produce your own installer.

1 Like

I remember having such a bug. I got a beta from the developer and at least on Sonoma Packages works fine.