folderitems dangerously broken on MacOS

  1. 5 weeks ago

    James S

    Nov 13 Pre-Release Testers, Xojo Pro

    Since upgrading to 2019r2 I’ve been having sporadic reports of errors when saving to files in some of my apps. I posted 4 days ago about one where a folderitem that isn’t created yet was treated like a text path.

    The real problem is actually much worse than that! Even of the file does exist and any folder in the path changes it’s name then all folderitems referencing anything downstream of that are broken and will result in an error if you try to use it.

    In my case I am managing a folder structure inside a bundled document. I change some of the folder names in order to make it easier for the user to know where things are. Even renaming the document folder now breaks every folderitem reference inside that folder. I don’t believe I’m the only one keeping folderitem references around. Previous to 2019r2 a folderitem know what it’s parent folder was even if the parent folder changed it’s name. After 2019r2 changing any folder in the path results in the folderitem become invalid.

    Please do this little test and tell me I’m not crazy:

    // create the parent folder
    Dim f1 As FolderItem = SpecialFolder.Desktop.Child( "starting test folder")
    f1.CreateAsFolder

    // create a child file and write something to it
    Dim f2 As FolderItem = f1.Child( "inside the parent folder")

    Dim b As BinaryStream = BinaryStream.Create( f2, True)
    b.write( "hello")
    b.close

    // if you stop here everything is good, now rename the parent folder

    f1.name = “changed name”

    // and attempt to access the f2 folderitem it will cause an io exception
    // error happens on next line when running in 2019r2
    b = BinaryStream.Create( f2, True)
    b.write( "hello again")
    b.close

    I consider this a serious issue as it’s going to break code that has worked fine since the down of Xojo and yet it may not immediately show up in testing. Like me it may not show up until you ship code to users and they start to work with it. Luckily I can take my pre-API2 project back to 2019r1.1 and rebuild it from there in the meantime.

    I created a second feedback report to address this specifically:
    Feedback Case #58255

  2. Philip C

    Nov 13 Pre-Release Testers, Xojo Pro Cooroy, QLD, Australia

    You are not crazy.
    I get "An exception of class IOException was not handled. The application must shut down.
    Exception Error Number: 2"

    I tried this:

    // create the parent folder
    Dim f1 As FolderItem = SpecialFolder.Desktop.Child( "starting test folder")
    f1.CreateAsFolder

    // create a child file and write something to it
    Dim f2 As FolderItem = f1.Child( "inside the parent folder")

    Dim b As BinaryStream = BinaryStream.Create( f2, True)
    b.write( "hello")
    b.close

    // if you stop here everything is good, now rename the parent folder

    f1.name = "changed name"
    f2 = f1.child( "inside the parent folder")

    MessageBox("f2 is " + f2.DisplayName)

    b = BinaryStream.Open( f2, True)
    b.write( "hello again")
    b.close

    Does this work for you?

    Phil

  3. Sam R

    Nov 13 Pre-Release Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan

    AFAIK this is how NSURL works also.

    I am not in the ‘office’ today so I can’t test it. You could try using GetSaveInfo and such to hold onto a reference to the file.

    You can also try to use Bookmarks from a NSURL; but I’ve seen enough reports from our customers to have little faith in this solution.

  4. Garth H

    Nov 13 Pre-Release Testers, Xojo Pro

    I find this totally fascinating. I've actually been dealing with "this issue" for several years, ever since Apple changed the basis on how files are handled in the OS - not in Xojo per se, but other apps. I guess this is what Sam means by "NSURL".

    Sam, if you can agree or correct me, it's like Apple steered away from using HFS "close to the metal" as far as determining all-things-file-related - whether it exists, how big it is, what it's named, what type it is, searching and cataloging, etc. It's like now it's database-like, perhaps Spotlight etc.

    The instances I mention are that I've caught a Mac red-handed where you ask if a file exists, the API says it doesn't but it does.

    More to the point, I am using Xojo 2019r1.1 and for this very reason I'm afraid to go to 2019r2 because API 2.0 seems to use all these "up-to-date" Apple functions under the hood, and they frankly are unreliable.

    So, in one way Xojo is trying to keep pace with Apple (a good thing) at the cost of things not working as solidly as they used to (not good). So it's not Xojo's "fault", but I really hate this New Brave World where the file system cannot be relied upon.

    Comment?

  5. James S

    Nov 13 Pre-Release Testers, Xojo Pro

    I think this is horrible from Apple. If I wanted to use text paths everywhere I’d just save the shellPath into a string. Whats the point of a folderitem object at all then? Thats really besides the point. It’s a HUGE departure from the behavior of the previous version that breaks existing code in insidious ways that you won’t find all of in your first round of testing and there is no documentation that describes this danger that I can see.

    I have used code that saved off folderitems since Xojo was RealBasic was CrossBasic. The point of API2.0 was to make my old projects load up and work in the new version without my having to do anything until I was ready to switch them to API2.0.

    This breaks them horribly. I’ve been happily working in r2 but I can’t anymore. The amount of testing to make this work will be huge.

    I HATE fixing stuff that wasn’t broken yesterday!

  6. Beatrix W

    Nov 13 Pre-Release Testers, Third Party Store Europe (Germany)

    The folderitem behavior has nothing to do with API 2. Obviously, Apple doesn't want you to keep references around. Yes, that sucks.

  7. James S

    Nov 13 Pre-Release Testers, Xojo Pro

    but it comes along with API 2 ;)

    This is not how the folderitem class works on windows or Linux is it? Which makes that not cross platform. I’m knocking off for the day but I’ll play with that tomorrow. I did just read through the documentation for folderitem and the only change noted with this shift to the new API is that there was a fix that let you have question marks in the name where that would previously have truncated the name at that point.

    folderitem should work the same on all targets and it no longer does. If fixing it is not practical then there needs to be a whole lot of documentation describing the exact changes and how to work around them. I can’t think of any straight forward ways to do so right now. I could make my own replacement classes that held some kind of links to their parent class and all the childs created so that it is possible to walk up the chain from any child to re-create the path. That way if you rename any in your xojo code the children would still be able to resolve themselves. That doesn’t isolate you from the event of the user changing the name of one of the encompassing folders, or even the name of the drive which people rename all the time.

  8. Sam R

    Nov 13 Pre-Release Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan

    @Garth H Sam, if you can agree or correct me, it's like Apple steered away from using HFS "close to the metal" as far as determining all-things-file-related - whether it exists, how big it is, what it's named, what type it is, searching and cataloging, etc. It's like now it's database-like, perhaps Spotlight etc.

    I can’t do either of those.

    I have to use NSURL a lot to get at some of the stuff I use and for the App Sandbox. I can only make assumptions based on experiences.

    Bookmarks are ‘recommended’ way of storing a reference to a file. The idea is that store the bookmark in memory, when you need that file, you resolve it to a NSURL pass that to the API.

    I would say these Bookmarks are about 70% reliable. For most users these seem to work, except when they don’t. Apple give you zero information when they fail, so you don’t even know the name of the file. So you have to store a backup path for when they fail.

    I also have reports from customers that OS updates can hose them, but personally I’ve not witnessed this. I also have reports of people moving a lower level folder or renaming it, hoses them. Again I’ve personally witnesses this, except for when my wife moved a bunch of stuff to an external drive.

    There is one huge advantage NSURL has; in that you can build a complex path, if the file doesn’t exist, you can ask the OS to create the entire path for you, which is nice.

  9. Sam R

    Nov 13 Pre-Release Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan
    Edited 5 weeks ago

    I do recall that in the past there was a way to get the ID of the file and the ID of the volume it resides on. Which IMHO is a pretty elegant way of storing a referefence to a file (along with meta data incase it’s missing). However I can only find a way to get the VolumeUUID with NSURL (which only works for Apple formatted volumes). There is a unique idetenfier for the file, but this changes everytime the file is modified.

    Edit: this is only what I’ve noticed, it may exist and I’ve simply not noticed it.

  10. 4 weeks ago

    James S

    Nov 14 Pre-Release Testers, Xojo Pro

    @Sam R There is one huge advantage NSURL has; in that you can build a complex path, if the file doesn’t exist, you can ask the OS to create the entire path for you, which is nice

    heh, except that if you do that with the current system it would re-create the path that it had before one of the parent folders was renamed ;) So now you’ve got the newly renamed folder with no data file in it and then next to it a new folder with the old name and the data file in it... thats not exactly the behavior that anyone would want I don’t think.

  11. Sam R

    Nov 14 Pre-Release Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan

    @James S thats not exactly the behavior that anyone would want I don’t think.

    I'll admit that in your circumstance it doesn't help, but I have started to use it myself when creating a new document package, as I don't need to create the entire folder structure to write a single file.

    In regards to your issue; I can only suggests experimenting with Bookmarks; but I also caution you to store enough meta data in-case the thing goes wonky, at least you can display the name of the file to the user, they can then use spotlight to figure out where they moved it too. I personally wouldn't recommend using Spotlight in code, it can return multiple results and with Apple's focus on "Security" the user may not be able to open the file directly from within your application.

    Not ideal, but it's all I can think of right now, that offer a glimmer of help/hope.

  12. @James S — I am referring to your original message. OK things have changed but I don't see how it would be a big problem.

    The new rule seems to be that you cannot keep access to a file if you rename any folder in its path. I suspect that this was driven by security concerns that the current discussion does not address.

  13. @Beatrix W"Obviously, Apple doesn't want you to keep references around. Yes, that sucks."

    Apple has different ways to keep a reference to a file with or without sandbox, so I am not sure what you are talking about

  14. Norman P

    Nov 14 Pre-Release Testers, Xojo Pro under THE bus

    The newer folder item code underpinnings are not nearly as capable as the old FSRef based ones at tracking files when they move

  15. Philip C

    Nov 14 Pre-Release Testers, Xojo Pro Cooroy, QLD, Australia

    Check my code:

    f1.name = "changed name"
    f2 = f1.child( "inside the parent folder")

    Need to reset f2 to point to f1 after changing the name

    Hioe that helps

  16. James S

    Nov 14 Pre-Release Testers, Xojo Pro

    @Philip C Need to reset f2 to point to f1 after changing the name

    yes :) but before 2019r2 you didn’t need to do this and there is no indication anywhere that this behavior had changed until I started getting errors. Even if the change is a simple fix you can’t fix it or plan for how much time it will take to fix it or how much it will cost you to fix it if you don’t know that it needs fixing ;)

    That example was the simplest form I could build to demonstrate the problem. The real problem in real code is more difficult with many hooks and changes that need to get tested. I can describe my real world situation. You create a new “document” which is stored inside a root folder. Each “document” gets it’s own folder. That folder is initially created with a temporary UUID type file name. Inside that folder is a binary data file that describes the document. Another binary file that contains keyed data entries related to the object. Then there are 2 folders for scripts. Inside those script folders are a binary file containing the compiled script data, a file containing the script text before compiling. A backup file of the compiled script data. A keyed data file containing script context information and 5 more files containing the text of the script at the last 5 compiles so that you can revert to previous working versions of the script if you hose it up. All this is one “document” inside the bundle. So I’ve got the document object which hold a reference to it’s folderitem inside which all of it’s data is stored. Then another folder item for the binary data, another for the keyed value data file and 2 more for the 2 script folders. The keyed data is loaded and parsed by a separate class that the document holds a reference to. It also needs to know it’s output file and once loaded or created just keeps a reference to that. The 2 scripts are managed by objects in xojo as well. They store a reference to their root folder, which is the document folder, and then also to their own compiled script file, backup script text, script text and keyed data context file which is managed by yet another object that it holes a reference to that knows about the file it’s supposed to write to.

    Now the user renames the “document” and ALL that structure is instantly broken and must be rebuilt. It’s certainly possible to do so but in the real world it’s not just 2 lines of code it’s hours of testing in a complex environment that has worked perfectly this way since 2005 or so when I started this project ;)

    None of these problems are insurmountable, but if you don’t know about this new gotcha it’s going to get you.

  17. Norman P

    Nov 14 Pre-Release Testers, Xojo Pro under THE bus

    sounds like the answer is to NOT hold references to the innards and recreate them as needed
    sucky but what else do you do if the new folderitem stuff is this fragile on a rename ?

  18. jim m

    Nov 14 Pre-Release Testers, Xojo Pro piDog.com

    Shouldn't FolderItem be using fileReferenceURL internally?

    File reference URLs use a URL path syntax that identifies a file system object by reference, not by path. This form of file URL remains valid when the file system path of the URL’s underlying resource changes.

  19. Norman P

    Nov 14 Pre-Release Testers, Xojo Pro under THE bus

    its been a long time since I looked into the guts of the folderitem impelmentation of the Xojo framework which was much more modern than the old desktop framework code
    dont know if its been even updated again since then

    either way I think, as you note, there is a way to get a URL that does track the file if the name changes or if the path changes

    based on the posts so far I would guess thats not whats being used

  20. Arnaud N

    Nov 15 Pre-Release Testers, Xojo Pro

    @James S Now the user renames the “document” and ALL that structure is instantly broken and must be rebuilt. It’s certainly possible to do so but in the real world it’s not just 2 lines of code it’s hours of testing in a complex environment that has worked perfectly this way since 2005 or so when I started this project ;)

    Have you considered to use a VirtualVolume? The user can't change the name of the inner files…

  21. Newer ›

or Sign Up to reply!