Catalina unzip File Permissions Problems

  1. 6 weeks ago

    Michael D

    Feb 17 Pre-Release Testers, Xojo Pro

    My xojo app has a feature to Zip an entire app (using ZipMBS https://www.monkeybreadsoftware.net/compression-zipmbs-method.shtml ) as a convenience feature. This .app.zip file can then be sent to another mac user, who can unzip and run it. This has worked great for years on mac OS 10.10 through 10.14

    But these Zip files no longer work on Catalina. Double-clicking one invokes Archive Utility (as expected) and you get an App file (as expected) but when you double-click the app, the Finder says "The application XXX can't be opened".

    Further digging reveals permissions problems:

    for Contents/MacOS/
    	Catalina: drwxrwxr-x   (775)  (wrong)
    	Mojave:   drwxr-xr-x   (755)  (right)
    
    for Contents/MacOS/MyApp
    	Catalina: -rw-rw-r--   (664)  (wrong)
    	Mojave:   -rwxr-xr-x   (755)  (which works)

    This is weird - the MacOS folder (which should be 755) permissions is getting 775 (which is less secure) but the executable inside it is getting 664 (which prevents it from running at all).

    • This doesn't seem to be a Gatekeeper issue - the problem shows up whether or not the .app.zip file is quarrantined.
    • manually fixing the permissions after unzipping fixes it:
       chmod +x MyApp.app/Contents/MacOS/MyApp

    * Using the built-in Archiver utillity (e.g. Right-Click in Finder and choose "Compress") in Catalina doesn't have this issue - the Zip files it creates can be unzipped on Catalina and work fine.

    This really feels like a Catalina bug to me. Any ideas?

    p.s. the way that zip files handle unix permissions is explained here: https://unix.stackexchange.com/questions/14705/the-zip-formats-external-file-attribute
    Basically there is a flag field in which you set the rwxrwxrwx bits for the desired permission. Nothing has changed in my Zip code and I'm pretty sure these are correct.

    And a solution which apparently works:

    Catalina is more strict (or perhaps just "broken in a new way") when processing zip files. So, when creating the ZipMBS file for an app which includes exectuable files, you need to set the "version created by" field to version # 2.0 and the OS to "unix" which is done by setting the version value to 0314 hex as shown here:

    dim bRaw as boolean = false
    dim zipWindowbits as integer = 15  // ignored
    dim zipMemLevel as integer = 9 // ignored
    dim zipStrategy as integer = ZipMBS.StrategyDefault
    dim zipPassword as string=""
    dim zipCRC as integer = 0
    dim zipFlagBase as integer =0
    dim zipFileComment as string =""
    dim bUseZip64 as boolean = true
    
    ' 4.4.2 version made by (2 bytes)
    ' 
    ' 4.4.2.1 The upper byte indicates the compatibility of the file
    ' attribute information.  If the external file attributes 
    ' are compatible with MS-DOS and can be read by PKZIP for 
    ' DOS version 2.04g then this value will be zero.  If these 
    ' attributes are not compatible, then this value will 
    ' identify the host system on which the attributes are 
    ' compatible.  Software can use this information to determine
    ' the line record format for text files etc.  
    ' 
    ' 4.4.2.2 The current mappings are:
    ' 
    ' 0 - MS-DOS and OS/2 (FAT / VFAT / FAT32 file systems)
    ' 1 - Amiga                     2 - OpenVMS
    ' 3 - UNIX                      4 - VM/CMS
    ' 5 - Atari ST                  6 - OS/2 H.P.F.S.
    ' 7 - Macintosh                 8 - Z-System
    ' 9 - CP/M                     10 - Windows NTFS
    ' 11 - MVS (OS/390 - Z/OS)      12 - VSE
    ' 13 - Acorn Risc               14 - VFAT
    ' 15 - alternate MVS            16 - BeOS
    ' 17 - Tandem                   18 - OS/400
    ' 19 - OS X (Darwin)            20 thru 255 - unused
    ' 
    ' 4.4.2.3 The lower byte indicates the ZIP specification version 
    ' (the version of this document) supported by the software 
    ' used to encode the file.  The value/10 indicates the major 
    ' version number, and the value mod 10 is the minor version 
    ' number.  
    
    dim zipVersionMadeBy as uint16 = &h0314 // 03 = unix. 14 hex = 20 decimal  =  version 2.0
    
    z.CreateFile(relativePath+name,zi,"","", zipFileComment, z.MethodDeflated, z.CompressionBestCompression, bUseZip64,bRaw, zipWindowBits, zipMemLevel, zipStrategy, zipPassword, zipCRC, zipVersionMadeBy, zipFlagBase)
    

    Note: zip files traditionally use no compression for Folders (I don't think this matters, but...) when adding a folder (rather than a file) you might change it slightly to not use any compression at all:

    z.CreateFile(relativePath+name,zi,"","", zipFileComment, z.MethodNone, z.CompressionNo, bUseZip64,bRaw, zipWindowBits, zipMemLevel, zipStrategy, zipPassword, zipCRC, zipVersionMadeBy, zipFlagBase)

    Note: see also below for code to set the permissions flags: https://forum.xojo.com/conversation/post/475862

  2. Sam R

    Feb 17 Pre-Release Testers, Xojo Pro, Third Party Store Hengchun, Pingtung, Taiwan

    If you use Finder to Zip up your application and send to another Mac, does that work?

  3. Rick A

    Feb 18 Pre-Release Testers (Brazil. UTC-3:00)

    Catalina tries to follow what is made inside your system and what came from the outside (don't know how, stored hashes?) so I think that can be a "Gatekeeper" feature, removing the execution bits from certain files. Not tested, but probably a zip created in your system behaves differently from one "imported" (downloaded, copied).

  4. Rick A

    Feb 18 Pre-Release Testers (Brazil. UTC-3:00)
    Edited 6 weeks ago

    Maybe Catalina just prefer signed DMGs as the compressed "secure" transport option of executables.

  5. Tim J

    Feb 18 Pre-Release Testers, Xojo Pro N. Phoenix, AZ
    Edited 6 weeks ago

    If you use Compress from Finder, I am pretty certain that you'll get back what you put in. However, using Zip from another platform (i.e.: building a macOS app on Linux, or, in your case, using Christian's standard Zip methods) and zipping there will result in the execution bits being cleared. So, it's not a Catalina "bug" so much as a "feature." Apple's Catalina nanny-mode is highly infuriating.

  6. Julia T

    Feb 18 Sandy Hook, Connecticut

    Putting Xojo considerations aside for the moment, does this mean that if I use the Finder to compress an app on my High Sierra Mac and send that zip to a Catalina user, it won't run after un-zipping under Catalina? What about a file zipped under Catalina and un-zipped on another Catalina machine?

  7. Michael D

    Feb 18 Pre-Release Testers, Xojo Pro

    More thoughts:

    • this only seems to affect Zip files made "by code" - e.g. using ZipMBS. I don't know if other Zip libraries are affected.
    • Finder Zip (right-click, "Compress My App") always works in both directions (compress on Catalina, works on Mojave, and vice versa)
    • My app is cross-platform (e.g. the Windows side can zip up a mac app) so this is a big problem for me. I'm pretty sure I can't build a DMG on Windows.
    • How does Xojo do it? (I purely run Xojo IDE on mac). If you build a Mac app on Xojo under Windows, does it build a Zip file? If so, is that now broken too?
  8. Tim J

    Feb 18 Pre-Release Testers, Xojo Pro N. Phoenix, AZ

    @Michael D How does Xojo do it? (I purely run Xojo IDE on mac). If you build a Mac app on Xojo under Windows, does it build a Zip file? If so, is that now broken too?

    The resulting built apps for macOS on Windows and Linux are just folders. It's up to you to decide how to "wrap" them to get them to macOS.

    @Michael D this only seems to affect Zip files made "by code" - e.g. using ZipMBS. I don't know if other Zip libraries are affected.

    If it wasn't created with Finder's hooks into the Zip functions in libarchive.so , Catalina strips the executable bits on extract.

  9. Tim P

    Feb 18 Pre-Release Testers, Xojo Pro Rochester, NY

    @Rick A Maybe Catalina just prefer signed DMGs as the compressed "secure" transport option of executables.

    Myself as well as the developers of many apps I use distribute as ZIP without issue.

    Michael you've explained how permissions on ZIP work, but I don't see any methods with ZipMBS for actually setting them. Have you set the flags on the files within the zip or perhaps had they just previously been assumed and are no longer?

  10. Michael D

    Feb 18 Pre-Release Testers, Xojo Pro

    Using the built-in osx utility "zipinfo" I'm able to see the difference between the files:

    ' Zip created by ZipMBS
    ' zipinfo MyApp.App.zip
    ' -rw----     0.0 fat        0 b- defX 20-Feb-17 18:19 MyApp.app/
    ' -rw----     0.0 fat        0 b- defX 20-Feb-17 18:19 MyApp.app/Contents/
    ' -rw----     0.0 fat        0 b- defX 20-Feb-17 18:19 MyApp.app/Contents/MacOS/
    ' -rw----     0.0 fat 16300608 b- defX 20-Feb-17 18:19 MyApp.app/Contents/MacOS/MyApp
    
    ' Zip created by Catalina:
    ' zipinfo MyApp.zip
    ' drwxrwxr-x  2.0 unx        0 bx stor 20-Feb-17 17:17 MyApp.app/
    ' drwxrwxr-x  2.0 unx        0 bx stor 20-Feb-17 17:17 MyApp.app/Contents/
    ' drwxrwxr-x  2.0 unx        0 bx stor 20-Feb-17 17:17 MyApp.app/Contents/MacOS/
    ' -rw-rw-r--  2.0 unx 16300608 bX defN 20-Feb-17 17:17 MyApp.app/Contents/MacOS/MyApp

    Notice the differences - the 'version created by' is 'fat' which means DOS, the version # is 0.0 instead of 2.0 and the file permissions are wrong as a result.

    If I can't fix this inside ZipMBS, I'll see about writing a post-processor which updates the version # fields in the zip file afer creation.

  11. Michael D

    Feb 18 Pre-Release Testers, Xojo Pro Answer
    Edited 6 weeks ago

    And a solution which apparently works:

    Catalina is more strict (or perhaps just "broken in a new way") when processing zip files. So, when creating the ZipMBS file for an app which includes exectuable files, you need to set the "version created by" field to version # 2.0 and the OS to "unix" which is done by setting the version value to 0314 hex as shown here:

    dim bRaw as boolean = false
    dim zipWindowbits as integer = 15  // ignored
    dim zipMemLevel as integer = 9 // ignored
    dim zipStrategy as integer = ZipMBS.StrategyDefault
    dim zipPassword as string=""
    dim zipCRC as integer = 0
    dim zipFlagBase as integer =0
    dim zipFileComment as string =""
    dim bUseZip64 as boolean = true
    
    ' 4.4.2 version made by (2 bytes)
    ' 
    ' 4.4.2.1 The upper byte indicates the compatibility of the file
    ' attribute information.  If the external file attributes 
    ' are compatible with MS-DOS and can be read by PKZIP for 
    ' DOS version 2.04g then this value will be zero.  If these 
    ' attributes are not compatible, then this value will 
    ' identify the host system on which the attributes are 
    ' compatible.  Software can use this information to determine
    ' the line record format for text files etc.  
    ' 
    ' 4.4.2.2 The current mappings are:
    ' 
    ' 0 - MS-DOS and OS/2 (FAT / VFAT / FAT32 file systems)
    ' 1 - Amiga                     2 - OpenVMS
    ' 3 - UNIX                      4 - VM/CMS
    ' 5 - Atari ST                  6 - OS/2 H.P.F.S.
    ' 7 - Macintosh                 8 - Z-System
    ' 9 - CP/M                     10 - Windows NTFS
    ' 11 - MVS (OS/390 - Z/OS)      12 - VSE
    ' 13 - Acorn Risc               14 - VFAT
    ' 15 - alternate MVS            16 - BeOS
    ' 17 - Tandem                   18 - OS/400
    ' 19 - OS X (Darwin)            20 thru 255 - unused
    ' 
    ' 4.4.2.3 The lower byte indicates the ZIP specification version 
    ' (the version of this document) supported by the software 
    ' used to encode the file.  The value/10 indicates the major 
    ' version number, and the value mod 10 is the minor version 
    ' number.  
    
    dim zipVersionMadeBy as uint16 = &h0314 // 03 = unix. 14 hex = 20 decimal  =  version 2.0
    
    z.CreateFile(relativePath+name,zi,"","", zipFileComment, z.MethodDeflated, z.CompressionBestCompression, bUseZip64,bRaw, zipWindowBits, zipMemLevel, zipStrategy, zipPassword, zipCRC, zipVersionMadeBy, zipFlagBase)
    

    Note: zip files traditionally use no compression for Folders (I don't think this matters, but...) when adding a folder (rather than a file) you might change it slightly to not use any compression at all:

    z.CreateFile(relativePath+name,zi,"","", zipFileComment, z.MethodNone, z.CompressionNo, bUseZip64,bRaw, zipWindowBits, zipMemLevel, zipStrategy, zipPassword, zipCRC, zipVersionMadeBy, zipFlagBase)

    Note: see also below for code to set the permissions flags: https://forum.xojo.com/conversation/post/475862

  12. Michael D

    Feb 18 Pre-Release Testers, Xojo Pro
    Edited 6 weeks ago

    @Tim P I don't see any methods with ZipMBS for actually setting them. Have you set the flags on the files within the zip or perhaps had they just previously been assumed and are no longer?

    Yes, I didn't show that code, but it is working. You set in the external file attribute flags - here's the pseudo code:

    dim zi as new ZipFileInfoMBS
    
    [...] do other stuff here [...]
    
    
    if isAFolderThen
      // to construct this value, enter the Unix permissions
      //  (e.g. 755) in Octal and convert to Hex (1ED) and then the value is  
      // 4xxx4000  repalcing the xxx with the Hex version of permissions
    
      flags = &h41ED4000   //  this gives proper 755 permission for folder when copied to OSX
    
    elseif isAFile then
    
     if IsAnExecutable then
        // to construct this value, enter the Unix permissions
        // (e.g. 755) in Octal and convert to Hex (1ED) and then the value is
        // 8xxx4000  repalcing the xxx with the Hex version of permissions
        flags = &h81ED4000   // this gives 755 permission when copied to OSX
      else
        // a regular old file - non executable
       // to construct this value, enter the Unix permissions 
      // (e.g. 644) in Octal and convert to Hex (1A4) and then the value is
      // 8xxx4000  repalcing the xxx with the Hex version of permissions
         flags = &h81A44000 // gives 644 permissions on OSX
      end if
    else
    
    zi.ExternalFileAttributes = flags
  13. Michael D

    Feb 18 Pre-Release Testers, Xojo Pro
    Edited 6 weeks ago

    @Tim J If it wasn't created with Finder's hooks into the Zip functions in libarchive.so , Catalina strips the executable bits on extract

    In initial testing, I don't think this is true: after making my fixes, I'm again able to create zip files using ZipMBS on either Mac or Windows OSs, copy the zip file to Catalina, double-click to decompress, and the resulting app now runs fine on catalina.

  14. Tim P

    Feb 18 Pre-Release Testers, Xojo Pro Rochester, NY

    I'm very glad that you found the root of the issue and solved it. Even more so glad that you shared the solution with us! Thank you!

  15. Michael D

    Feb 18 Pre-Release Testers, Xojo Pro

    You are welcome! I emailed Christian of MBS as well to ask that he document these on his side.

  16. Norman P

    Feb 18 Pre-Release Testers, Xojo Pro outside avoiding snowflake
    Edited 6 weeks ago

    FYI &O works for octal literals just like &H works for hex and &B for binary ones

    // 4xxx4000  replacing the xxx with the Hex version of permissions
    
    Dim base As Integer = &h40004000 
    
    Dim hflags As Integer = &h41ED4000                          //  this gives proper 755 permission for folder when copied to OSX
    Dim oflags As Integer = &O755                               // &b1 1110 1101 ->  &h1ED 
    Dim bflags  As Integer = &b01000001111011010100000000000000 //                  &h41ED4000 
    
    Dim result As Integer = Bitwise.ShiftLeft(oflags , 16) + base
    
    Break
    
  17. Michael D

    Feb 18 Pre-Release Testers, Xojo Pro

    Thanks, Norman. Adding to the confusion, in C and (old) Python, the syntax for Octal is a leading 0 (zero) - e.g. "010" means "10 Octal" which is 8 decimal. Thankfully, Xojo doesn't use this.

    Thus, many of the numbers shown on https://unix.stackexchange.com/questions/14705/the-zip-formats-external-file-attribute are in Octal format. Once you figure that out that mess it's a little easier.

  18. Norman P

    Feb 18 Pre-Release Testers, Xojo Pro outside avoiding snowflake

    @Michael D Thanks, Norman. Adding to the confusion, in C and (old) Python, the syntax for Octal is a leading 0 (zero) - e.g. "010" means "10 Octal" which is 8 decimal. Thankfully, Xojo doesn't use this.

    OH gawd yes !!!!

  19. Arnaud N

    Feb 21 Pre-Release Testers, Xojo Pro

    @Michael D dim zi as new ZipFileInfoMBS [...] do other stuff here [...] if isAFolderThen // to construct this value, enter the Unix permissions // (e.g. 755) in Octal and convert to Hex (1ED) and then the value is // 4xxx4000 repalcing the xxx with the Hex version of permissions flags = &h41ED4000 // this gives proper 755 permission for folder when copied to OSX elseif isAFile then if IsAnExecutable then // to construct this value, enter the Unix permissions // (e.g. 755) in Octal and convert to Hex (1ED) and then the value is // 8xxx4000 repalcing the xxx with the Hex version of permissions flags = &h81ED4000 // this gives 755 permission when copied to OSX else // a regular old file - non executable // to construct this value, enter the Unix permissions // (e.g. 644) in Octal and convert to Hex (1A4) and then the value is // 8xxx4000 repalcing the xxx with the Hex version of permissions flags = &h81A44000 // gives 644 permissions on OSX end if else zi.ExternalFileAttributes = flags

    If the original item has different permissions (e.g. the user wants to deny access for the “other users”), you lose the setting in the conversion. I guess you'd take the original item's permissions and apply them in the code above.

  20. Michael D

    Feb 21 Pre-Release Testers, Xojo Pro

    @Arnaud N If the original item has different permissions (e.g. the user wants to deny access for the “other users”), you lose the setting in the conversion. I guess you'd take the original item's permissions and apply them in the code above.

    Correct - my app is actually generating these .app files, so it knows what the permissions should be. If you are writing an app to zip up arbitrary files, it may be smarter to use the on-disk permissions instead. (Although this is not as simple as it seems - what should you do for example if you read an executable with the wrong permissions? Keep the incorrect permissions? Fix them? etc.)

  21. Newer ›

or Sign Up to reply!