Unable to Write to FolderItem on Catalina

A small but important number of my users are unable to consistently write to a FolderItem on Catalina. It seems a bit sporadic for the affected users with it failing more often than it working. Other users report no problems.

Could it be a permissions/security issue? The app is code signed and notarised. The problem happens regardless if I write to designated folderItem or the user chooses the location via a dialog. It also doesn’t make a difference if it’s saved in Application Support or Documents.

How it saves:

  • Make tempSave folderItem made in same location as previous save.

  • create a stream from the tempSave

  • Write to stream

  • Use ExchangeFilesMBS to atomically swap tempSave and the previous save.

  • Make a new reference to the tempSave and delete it. (This deletion also frequently fails in all version of MacOS.)

I’d post the code but it’s longer than I’d expect anyone to casually troll through.

Any clues/thoughts are appreciated. :slight_smile:

I think @Sam Rowlands mentioned that this form of saving is not allowed. You have to write to the FolderItem given directly by the user, and can’t write to temporary then swap.

OooOooh. That never occurred to me.

That doesn’t sound very safe, though. If the app crashed, there would be a half written file.

I wonder if I can permissibly write to the temporary folder. (Though wasn’t there as problem with that on Catalina and Xojo 2019r.1.1?)

As I understood it, the issue isn’t with writing to temporary but rather the swap action.

Update:

2019r2 changed the system API behind FolderItem to deal with Catalina related issues.

Interesting! Is there a best practice for saving over an existing file to prevent data loss?

I don’t have that information, I’m afraid. I haven’t yet had to explore the proper avenues for large file saving.

This is definitely something I’d like to know about though, so if there’s information out there I’d love to read about it.

Catalina release notes ( here ) do not talk about that.

You’ll have to ask @Sam Rowlands how he specifically discovered it, Emile.

By trying or talking with Apple Engineers, I suppose.

We do something similar except using ExchangeFilesMBS to perform the atomic swap (we just rename the files) and don’t have any problems under Catalina.
Maybe ExchangeFilesMBS is still using older file system APIs that now have problems under Catalina.

[quote=488314:@Kevin Gale]We do something similar except using ExchangeFilesMBS to perform the atomic swap (we just rename the files) and don’t have any problems under Catalina.
Maybe ExchangeFilesMBS is still using older file system APIs that now have problems under Catalina.[/quote]
For what it’s worth, I also use ExchangeFilesMBS for all my file saves, and I work on Catalina as my main machine (without a problem).

Though I am using macOS 10.15.3 because I heard rumours of file system issues with Catalina 10.15.4, or least more than the usual number of rumours. So I’m holding off on upgrading.

Edited, for clarity.

@Kevin Gale, @Scott Cadillac : What version of Xojo are you compiling with?

Sorry, good question.

2019r3.1

2017r3

Right; soooo…

Sorry for any confusion here, you’re not permitted to create temporary files in the same folder as the destination if your application is in the App Sandbox. I won’t be surprised if this rolled out to the App Sandbox lite (Harden Runtime) eventually.

Some of my users (across Sandboxed and non-Sandboxed application) found that our apps are no longer allowed to create files in the specialfolder.temporary directory (or even using different API) on macOS 10.15. This is where I was creating the temporary files, to then replace the user selected file.

The solution for the above problem was to use [NSFileManager URLForDirectory:inDomain:appropriateForURL:create:error:] to obtain a writable location, then [NSFileManager replaceItemAtURL:withItemAtURL:backupItemName:options:resultingItemURL:error:] to replace the user file with the temporary one.
However there’s an undocumented issue with this combination, whereby it will leave behind an empty folder in the temporary folder (yes the same temporary folder that my app is unable to create files in), once the number of files in the temporary folder reaches 512, your application can no longer save any files using this atomic solution.
The solution is to grab the parent folder of the temporary file, after the replace function is called, try to delete the parent folder. This sometimes works and sometimes doesn’t work (I couldn’t find a rhyme or reason for it).

I don’t know if there’s any additional file system issues on 10.15.4, but there is a bug in there that can prevent you from extending app subscriptions from the App Store.

Once I can confirm it, I think that there’s a bug with TextOutputStream.append in Xojo, that’s causing a lot of broken file accesses to be attached to your application. If you use this function, @Beatrix Willius has suggested to try using a binarystream instead.

If you use NSTask, please be advised that there’s also a bug when requesting the OS sends your app a Notification on dataAvailable, which leaks a preemptive thread and broken file access.

I really don’t know what the best solution is here; at one time I thought it was best to run the latest OS, no matter what. Then I started to get into issues with the new system, and I’d spend a lot of time trying to figure out if the issue is me, or yet another bug in the macOS. So I caved and went back and only use new versions of the macOS for testing. However since macOS 10.14, I’ve started to notice more and more issues that don’t occur during testing, you need to be actually using the latest OS. The first time was people reporting my apps exporting sold black images. Took me ages to realize, this only happens after the machine has been running for about 10 days or so. Once I could reproduce it, I used an alternative API (because Apple has many ways to accomplish the same thing) to workaround the bug.

There’s nothing worse than dropping $3,500 on a brand new Mac, and having to go back to your old one running an older OS to do actual development, because you spent so much time trying to figure out why your code wasn’t working, and it’s not you, it’s Apple.

Hard times.

Thanks for the great insight @Sam Rowlands!

So, noob question. How does one accomplish this in Xojo? I’ve little experience in declares. Happy to use a MBS call…

Have you read this MacWorld UK article ?

Link:
https://www.macworld.co.uk/news/mac-software/catalina-bug-crashes-3787693/

I would imagine that these functions are available in the MBS plugin. If you don’t use MBS, I will be selling my own internal library from next month, so you’ll be able to pick up the declares there (and all the supporting declares for crossing between Xojo and the macOS).

[quote=488465:@Emile Schwarz]Have you read this MacWorld UK article ?

Link:
https://www.macworld.co.uk/news/mac-software/catalina-bug-crashes-3787693/[/quote]
Was genuinely shocked that there were regressions made in 10.15.4, it should be getting more stable by now, not worse!