Unable to Write to FolderItem on Catalina

  1. 8 weeks ago

    Stephen D

    May 12 Testers, Xojo Pro

    A small but important number of my users are unable to consistently write to a FolderItem on Catalina. It seems a bit sporadic for the affected users with it failing more often than it working. Other users report no problems.

    Could it be a permissions/security issue? The app is code signed and notarised. The problem happens regardless if I write to designated folderItem or the user chooses the location via a dialog. It also doesn't make a difference if it's saved in Application Support or Documents.

    How it saves:

    * Make tempSave folderItem made in same location as previous save.

    * create a stream from the tempSave

    * Write to stream

    * Use ExchangeFilesMBS to atomically swap tempSave and the previous save.

    * Make a new reference to the tempSave and delete it. (This deletion also frequently fails in all version of MacOS.)

    I'd post the code but it's longer than I'd expect anyone to casually troll through.

    Any clues/thoughts are appreciated. :)

  2. 7 weeks ago

    Tim P

    May 12 Testers, Xojo Pro Rochester, NY

    @Stephen D * Use ExchangeFilesMBS to atomically swap tempSave and the previous save.

    I think @Sam R mentioned that this form of saving is not allowed. You have to write to the FolderItem given directly by the user, and can't write to temporary then swap.

  3. Stephen D

    May 12 Testers, Xojo Pro

    OooOooh. That never occurred to me.

    That doesn't sound very safe, though. If the app crashed, there would be a half written file.

    I wonder if I can permissibly write to the temporary folder. (Though wasn't there as problem with that on Catalina and Xojo 2019r.1.1?)

  4. Tim P

    May 12 Testers, Xojo Pro Rochester, NY
    Edited 7 weeks ago

    As I understood it, the issue isn't with writing to temporary but rather the swap action.

    Update:

    @Stephen Dodd Though wasn't there as problem with that on Catalina and Xojo 2019r.1.1

    2019r2 changed the system API behind FolderItem to deal with Catalina related issues.

  5. Stephen D

    May 12 Testers, Xojo Pro

    Interesting! Is there a best practice for saving over an existing file to prevent data loss?

  6. Tim P

    May 12 Testers, Xojo Pro Rochester, NY

    @Stephen D Interesting! Is there a best practice for saving over an existing file to prevent data loss?

    I don't have that information, I'm afraid. I haven't yet had to explore the proper avenues for large file saving.

    This is definitely something I'd like to know about though, so if there's information out there I'd love to read about it.

  7. Emile S

    May 12 Europe (France, Strasbourg)

    Catalina release notes ( here ) do not talk about that.

  8. Tim P

    May 12 Testers, Xojo Pro Rochester, NY

    You'll have to ask @Sam R how he specifically discovered it, Emile.

  9. Emile S

    May 12 Europe (France, Strasbourg)

    By trying or talking with Apple Engineers, I suppose.

  10. Kevin G

    May 12 Xojo Pro Gatesheed, England

    @Tim P I think @Sam R mentioned that this form of saving is not allowed. You have to write to the FolderItem given directly by the user, and can't write to temporary then swap.

    We do something similar except using ExchangeFilesMBS to perform the atomic swap (we just rename the files) and don't have any problems under Catalina.
    Maybe ExchangeFilesMBS is still using older file system APIs that now have problems under Catalina.

  11. Scott C

    May 12 Testers, Xojo Pro twitter.com/ScottCadillac
    Edited 7 weeks ago

    @Kevin G We do something similar except using ExchangeFilesMBS to perform the atomic swap (we just rename the files) and don't have any problems under Catalina.
    Maybe ExchangeFilesMBS is still using older file system APIs that now have problems under Catalina.

    For what it's worth, I also use ExchangeFilesMBS for all my file saves, and I work on Catalina as my main machine (without a problem).

    Though I am using macOS 10.15.3 because I heard rumours of file system issues with Catalina 10.15.4 , or least more than the usual number of rumours. So I'm holding off on upgrading.

    Edited, for clarity.

  12. Stephen D

    May 12 Testers, Xojo Pro

    @Kevin G, @Scott C : What version of Xojo are you compiling with?

  13. Scott C

    May 12 Testers, Xojo Pro twitter.com/ScottCadillac

    @Stephen D @Scott C : What version of Xojo are you compiling with?

    Sorry, good question.

    2019r3.1

  14. Kevin G

    May 12 Xojo Pro Gatesheed, England

    @Stephen D @Kevin G, @Scott C : What version of Xojo are you compiling with?

    2017r3

  15. Sam R

    May 12 Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan

    Right; soooo....

    @Tim P I think @Sam Rowlands mentioned that this form of saving is not allowed. You have to write to the FolderItem given directly by the user, and can't write to temporary then swap.

    Sorry for any confusion here, you're not permitted to create temporary files in the same folder as the destination if your application is in the App Sandbox. I won't be surprised if this rolled out to the App Sandbox lite (Harden Runtime) eventually.

    Some of my users (across Sandboxed and non-Sandboxed application) found that our apps are no longer allowed to create files in the specialfolder.temporary directory (or even using different API) on macOS 10.15. This is where I was creating the temporary files, to then replace the user selected file.

    The solution for the above problem was to use [NSFileManager URLForDirectory:inDomain:appropriateForURL:create:error:] to obtain a writable location, then [NSFileManager replaceItemAtURL:withItemAtURL:backupItemName:options:resultingItemURL:error:] to replace the user file with the temporary one.
    However there's an undocumented issue with this combination, whereby it will leave behind an empty folder in the temporary folder (yes the same temporary folder that my app is unable to create files in), once the number of files in the temporary folder reaches 512, your application can no longer save any files using this atomic solution.
    The solution is to grab the parent folder of the temporary file, after the replace function is called, try to delete the parent folder. This sometimes works and sometimes doesn't work (I couldn't find a rhyme or reason for it).

    @Scott C Though I am using macOS 10.15.3 because I heard rumours of file system issues with Catalina 10.15.4 , or least more than the usual number of rumours. So I'm holding off on upgrading.

    I don't know if there's any additional file system issues on 10.15.4, but there is a bug in there that can prevent you from extending app subscriptions from the App Store.

    Once I can confirm it, I think that there's a bug with TextOutputStream.append in Xojo, that's causing a lot of broken file accesses to be attached to your application. If you use this function, @Beatrix W has suggested to try using a binarystream instead.

    If you use NSTask, please be advised that there's also a bug when requesting the OS sends your app a Notification on dataAvailable, which leaks a preemptive thread and broken file access.

    I really don't know what the best solution is here; at one time I thought it was best to run the latest OS, no matter what. Then I started to get into issues with the new system, and I'd spend a lot of time trying to figure out if the issue is me, or yet another bug in the macOS. So I caved and went back and only use new versions of the macOS for testing. However since macOS 10.14, I've started to notice more and more issues that don't occur during testing, you need to be actually using the latest OS. The first time was people reporting my apps exporting sold black images. Took me ages to realize, this only happens after the machine has been running for about 10 days or so. Once I could reproduce it, I used an alternative API (because Apple has many ways to accomplish the same thing) to workaround the bug.

    There's nothing worse than dropping $3,500 on a brand new Mac, and having to go back to your old one running an older OS to do actual development, because you spent so much time trying to figure out why your code wasn't working, and it's not you, it's Apple.

  16. Emile S

    May 12 Europe (France, Strasbourg)

    Hard times.

  17. Stephen D

    May 13 Testers, Xojo Pro

    Thanks for the great insight @Sam R!

    The solution for the above problem was to use [NSFileManager URLForDirectory:inDomain:appropriateForURL:create:error:] to obtain a writable location, then [NSFileManager replaceItemAtURL:withItemAtURL:backupItemName:options:resultingItemURL:error:] to replace the user file with the temporary one.

    So, noob question. How does one accomplish this in Xojo? I've little experience in declares. Happy to use a MBS call...

  18. Emile S

    May 13 Europe (France, Strasbourg)

    Have you read this MacWorld UK article ?

    Link:
    https://www.macworld.co.uk/news/mac-software/catalina-bug-crashes-3787693/

  19. Newer ›

or Sign Up to reply!