Unable to Write to FolderItem on Catalina

  1. ‹ Older
  2. 7 weeks ago

    Tim P

    May 12 Testers, Xojo Pro Rochester, NY

    You'll have to ask @Sam R how he specifically discovered it, Emile.

  3. Emile S

    May 12 Europe (France, Strasbourg)

    By trying or talking with Apple Engineers, I suppose.

  4. Kevin G

    May 12 Xojo Pro Gatesheed, England

    @Tim P I think @Sam R mentioned that this form of saving is not allowed. You have to write to the FolderItem given directly by the user, and can't write to temporary then swap.

    We do something similar except using ExchangeFilesMBS to perform the atomic swap (we just rename the files) and don't have any problems under Catalina.
    Maybe ExchangeFilesMBS is still using older file system APIs that now have problems under Catalina.

  5. Scott C

    May 12 Testers, Xojo Pro twitter.com/ScottCadillac
    Edited 7 weeks ago

    @Kevin G We do something similar except using ExchangeFilesMBS to perform the atomic swap (we just rename the files) and don't have any problems under Catalina.
    Maybe ExchangeFilesMBS is still using older file system APIs that now have problems under Catalina.

    For what it's worth, I also use ExchangeFilesMBS for all my file saves, and I work on Catalina as my main machine (without a problem).

    Though I am using macOS 10.15.3 because I heard rumours of file system issues with Catalina 10.15.4 , or least more than the usual number of rumours. So I'm holding off on upgrading.

    Edited, for clarity.

  6. Stephen D

    May 12 Testers, Xojo Pro

    @Kevin G, @Scott C : What version of Xojo are you compiling with?

  7. Scott C

    May 12 Testers, Xojo Pro twitter.com/ScottCadillac

    @Stephen D @Scott C : What version of Xojo are you compiling with?

    Sorry, good question.

    2019r3.1

  8. Kevin G

    May 12 Xojo Pro Gatesheed, England

    @Stephen D @Kevin G, @Scott C : What version of Xojo are you compiling with?

    2017r3

  9. Sam R

    May 12 Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan

    Right; soooo....

    @Tim P I think @Sam Rowlands mentioned that this form of saving is not allowed. You have to write to the FolderItem given directly by the user, and can't write to temporary then swap.

    Sorry for any confusion here, you're not permitted to create temporary files in the same folder as the destination if your application is in the App Sandbox. I won't be surprised if this rolled out to the App Sandbox lite (Harden Runtime) eventually.

    Some of my users (across Sandboxed and non-Sandboxed application) found that our apps are no longer allowed to create files in the specialfolder.temporary directory (or even using different API) on macOS 10.15. This is where I was creating the temporary files, to then replace the user selected file.

    The solution for the above problem was to use [NSFileManager URLForDirectory:inDomain:appropriateForURL:create:error:] to obtain a writable location, then [NSFileManager replaceItemAtURL:withItemAtURL:backupItemName:options:resultingItemURL:error:] to replace the user file with the temporary one.
    However there's an undocumented issue with this combination, whereby it will leave behind an empty folder in the temporary folder (yes the same temporary folder that my app is unable to create files in), once the number of files in the temporary folder reaches 512, your application can no longer save any files using this atomic solution.
    The solution is to grab the parent folder of the temporary file, after the replace function is called, try to delete the parent folder. This sometimes works and sometimes doesn't work (I couldn't find a rhyme or reason for it).

    @Scott C Though I am using macOS 10.15.3 because I heard rumours of file system issues with Catalina 10.15.4 , or least more than the usual number of rumours. So I'm holding off on upgrading.

    I don't know if there's any additional file system issues on 10.15.4, but there is a bug in there that can prevent you from extending app subscriptions from the App Store.

    Once I can confirm it, I think that there's a bug with TextOutputStream.append in Xojo, that's causing a lot of broken file accesses to be attached to your application. If you use this function, @Beatrix W has suggested to try using a binarystream instead.

    If you use NSTask, please be advised that there's also a bug when requesting the OS sends your app a Notification on dataAvailable, which leaks a preemptive thread and broken file access.

    I really don't know what the best solution is here; at one time I thought it was best to run the latest OS, no matter what. Then I started to get into issues with the new system, and I'd spend a lot of time trying to figure out if the issue is me, or yet another bug in the macOS. So I caved and went back and only use new versions of the macOS for testing. However since macOS 10.14, I've started to notice more and more issues that don't occur during testing, you need to be actually using the latest OS. The first time was people reporting my apps exporting sold black images. Took me ages to realize, this only happens after the machine has been running for about 10 days or so. Once I could reproduce it, I used an alternative API (because Apple has many ways to accomplish the same thing) to workaround the bug.

    There's nothing worse than dropping $3,500 on a brand new Mac, and having to go back to your old one running an older OS to do actual development, because you spent so much time trying to figure out why your code wasn't working, and it's not you, it's Apple.

  10. Emile S

    May 12 Europe (France, Strasbourg)

    Hard times.

  11. Stephen D

    May 13 Testers, Xojo Pro

    Thanks for the great insight @Sam R!

    The solution for the above problem was to use [NSFileManager URLForDirectory:inDomain:appropriateForURL:create:error:] to obtain a writable location, then [NSFileManager replaceItemAtURL:withItemAtURL:backupItemName:options:resultingItemURL:error:] to replace the user file with the temporary one.

    So, noob question. How does one accomplish this in Xojo? I've little experience in declares. Happy to use a MBS call...

  12. Emile S

    May 13 Europe (France, Strasbourg)

    Have you read this MacWorld UK article ?

    Link:
    https://www.macworld.co.uk/news/mac-software/catalina-bug-crashes-3787693/

  13. Sam R

    May 13 Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan

    @Stephen D So, noob question. How does one accomplish this in Xojo? I've little experience in declares. Happy to use a MBS call...

    I would imagine that these functions are available in the MBS plugin. If you don't use MBS, I will be selling my own internal library from next month, so you'll be able to pick up the declares there (and all the supporting declares for crossing between Xojo and the macOS).

  14. Sam R

    May 13 Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan

    @Emile S Have you read this MacWorld UK article ?

    Link:
    https://www.macworld.co.uk/news/mac-software/catalina-bug-crashes-3787693/

    Was genuinely shocked that there were regressions made in 10.15.4, it **should** be getting more stable by now, not worse!

  15. 5 days ago

    Thomas T

    Jun 29 Testers, Xojo Pro Europe (Germany, Munich)
    Edited 4 days ago

    I ran into a similar issue with Arbed 1.9 when built with Xojo 2019r3. I was trying to use `replaceItemAtURL`, like Sam suggested, but it was tricky because I needed a true exchange (swap) operation.

    I came up with this code, including a way for Windows (though that's not thoroughly tested). It requires a little bit of MBS but could also be solved by using NSURL from the good olde macOSLib. The use of `fileSystemRepresentation` may be replaced by `NativePath`.

    (Update: I've fixed a little code issue - now the FolderItems are cloned first, even for macOS)

    Protected Function ExchangeFileContents(destFile as FolderItem, tempFile as FolderItem) As Boolean
      // use clones so that the caller's don't get mixed up
      destFile = new FolderItem(destFile)
      tempFile = new FolderItem(tempFile)
    
      #if TargetMacOS
        dim urlDest as new NSURLMBS (destFile)
        dim urlTemp as new NSURLMBS (tempFile)
        
        if urlDest.VolumeIdentifier = urlTemp.VolumeIdentifier then
          const CocoaLib = "Cocoa"
          declare function objc_getClass lib CocoaLib (aClassName as CString) as Ptr
          declare function defaultManager lib CocoaLib selector "defaultManager" (obj as Ptr) as Ptr
          static defaultMgr as Ptr = defaultManager (objc_getClass("NSFileManager"))
          
          declare function replaceItem lib CocoaLib _
                selector "replaceItemAtURL:withItemAtURL:backupItemName:options:resultingItemURL:error:" _
                (obj as Ptr, url1 as Integer, url2 as Integer, backupName as CFStringRef, opts as Integer, _
                ByRef result as Integer, err as Ptr) as Boolean
          
          // we need this to get a swap effect; otherwise we'd lose the tempFile:
          const NSFileManagerItemReplacementWithoutDeletingBackupItem = 2
          dim tempName as String = tempFile.Name
          dim backupName as String = tempName+"_"
          
          dim result as Integer
          if replaceItem (defaultMgr, urlDest.Handle, urlTemp.Handle, backupName, _
                      NSFileManagerItemReplacementWithoutDeletingBackupItem, result, nil) then
            dim newTemp as FolderItem = tempFile.Parent.TrueChild(backupName)
            newTemp.Name = tempName
            dim url_res as NSURLMBS = NSURLMBS.URLWithHandle (result)
            if StrComp (url_res.fileSystemRepresentation, urlDest.fileSystemRepresentation, 0) = 0 then
              return true
            end if
          end
          break
        end if
        
      #elseif TargetWin32
        Soft Declare Function ReplaceFileW Lib "Kernel32" ( destination as WString, source as WString, _
        backup as Integer, flags as Integer, reserved1 as Integer, reserved2 as Integer ) as Boolean
        
        if System.IsFunctionAvailable( "ReplaceFileW", "Kernel32" ) then
          Const REPLACEFILE_WRITE_THROUGH = &h1
          dim success as Boolean = ReplaceFileW( destFile.AbsolutePath, tempFile.AbsolutePath, _
                      0, REPLACEFILE_WRITE_THROUGH, 0, 0 )
          if success then
            destFile.Visible = true
            return true
          end
        end if
        
      #endif
      
      return SimpleSwapFiles (destFile, tempFile)
    End Function
    
    Private Function SimpleSwapFiles(f as FolderItem, g as FolderItem) As Boolean
      dim temp as FolderItem = f.Parent.TrueChild("."+f.Name+"-"+Format(Rnd()*10000,"#")+".tmp")
      if temp.Exists then
        temp.Delete
      end if
      f.MoveFileTo temp
      If f.LastErrorCode <> 0 or f.Exists or NOT temp.Exists then
        return false
      End if
      g.MoveFileTo f
      If g.LastErrorCode <> 0 or g.Exists or NOT f.Exists then
        return false
      End if
      temp.MoveFileTo g
      If temp.LastErrorCode <> 0 or temp.Exists or NOT g.Exists then
        return false
      End if
      
      return true
    End Function
  16. James S

    Jun 29 Testers, Xojo Pro

    Could this be related to the change in the file system APIs in some not that long ago Xojo version? That caused me quite a bit of pain as I figured out what the problem was.

    In that case a folder item object no longer follows the actual file after you do that exchange. (it also wont follow the file anymore if any parent folder changes its name or moves like it used to, it’s basically just the path under the hood, it is not an actual link to the file)

    so if you have

    dim mainFile as folderitem = something
    dim tempFile as folderitem = something else

    and then do the swap, they are not swapped. the mainFile now points to the what used to be the tempFile and the tempFile now points to what used to be the mainFile. If you’re doing something like swapping them and then deleting the mainFile thinking it’s still pointing to the old file you wanted to replace what you’re actually doing is deleting the new file you just swapped into that path.

    I don’t know if it’s related, but my blood pressure is going up again just thinking about the frustration in figuring all that out ;)

  17. 4 days ago

    Thomas T

    Jun 29 Testers, Xojo Pro Europe (Germany, Munich)

    Hi James, yes. And I just had to update my code after realizing that it would still confuse the FolderItems if I don't clone them first.

    And I also believe that the tracking behavior did change between 2012r2 and now. That's why I do the cloning.

    Oh my, I'm now getting the weirdest effects:

    1. After swapping the file with the above code of mine, Finder Aliases to the original file break in a way that Xojo can't resolve them any more (but old RS still can). Jeez!

    2. My code that's supposed to notice when an opened file got updated in the background is not working any more: It detect a change _after_ the save, even though the code is supposed to store the new saved date so that it doesn't see the modified file on disk as changed any more. I guess this is another result of the FolderItem caching behavioral change. Oh my, all these changes causing so much trouble, thanks to Xojo's changes that I have no control over.

  18. Sam R

    Jun 29 Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan

    @ThomasTempelmann I ran into a similar issue with Arbed 1.9 when built with Xojo 2019r3. I was trying to use `replaceItemAtURL`, like Sam suggested, but it was tricky because I needed a true exchange (swap) operation.

    The code I use is available as part of the Ohanaware App Kit, which is sold in the 2020 Omegabundle (http://omegabundle.net ).

  19. Norman P

    Jun 29 Testers, Xojo Pro outside admiring the sunshine,...

    @ThomasTempelmann And I also believe that the tracking behavior did change between 2012r2 and now. That's why I do the cloning.

    absolutely did since 2012 would be carbon based and newer versions would be cocoa based
    2012 would use the file manager calls from carbon and newer uses nsurl or something

  20. Thomas T

    Jun 30 Testers, Xojo Pro Europe (Germany, Munich)

    @Norman P absolutely did since 2012 would be carbon based and newer versions would be cocoa based
    2012 would use the file manager calls from carbon and newer uses nsurl or something

    Yes, but that's just an implementation detail that should not change behavior. Other than performance (such as that indexed access may become slower since the new macOS API has no calls for that).

    What we're seeing here, though, is that suddenly values are cached in a way that they do not update when the actual file's properties have changed, and there's not even an "UpdateProperties" function for triggering it. So, the person who made this change seems to not even have been aware of the consequences.

    The work-around is to create a new FolderItem from the existing one (f = new FolderItem (f)) - that will then update its properties with the current on-disk values.

or Sign Up to reply!