Catalina unzip File Permissions Problems

Targets macOS
1

My xojo app has a feature to Zip an entire app (using ZipMBS https://www.monkeybreadsoftware.net/compression-zipmbs-method.shtml ) as a convenience feature. This .app.zip file can then be sent to another mac user, who can unzip and run it. This has worked great for years on mac OS 10.10 through 10.14

But these Zip files no longer work on Catalina. Double-clicking one invokes Archive Utility (as expected) and you get an App file (as expected) but when you double-click the app, the Finder says “The application XXX can’t be opened”.

Further digging reveals permissions problems:

for Contents/MacOS/
	Catalina: drwxrwxr-x   (775)  (wrong)
	Mojave:   drwxr-xr-x   (755)  (right)

for Contents/MacOS/MyApp
	Catalina: -rw-rw-r--   (664)  (wrong)
	Mojave:   -rwxr-xr-x   (755)  (which works)

This is weird - the MacOS folder (which should be 755) permissions is getting 775 (which is less secure) but the executable inside it is getting 664 (which prevents it from running at all).

  • This doesn’t seem to be a Gatekeeper issue - the problem shows up whether or not the .app.zip file is quarrantined.
  • manually fixing the permissions after unzipping fixes it:
   chmod +x MyApp.app/Contents/MacOS/MyApp
  • Using the built-in Archiver utillity (e.g. Right-Click in Finder and choose “Compress”) in Catalina doesn’t have this issue - the Zip files it creates can be unzipped on Catalina and work fine.

This really feels like a Catalina bug to me. Any ideas?

p.s. the way that zip files handle unix permissions is explained here: https://unix.stackexchange.com/questions/14705/the-zip-formats-external-file-attribute
Basically there is a flag field in which you set the rwxrwxrwx bits for the desired permission. Nothing has changed in my Zip code and I’m pretty sure these are correct.

2

If you use Finder to Zip up your application and send to another Mac, does that work?

3

Catalina tries to follow what is made inside your system and what came from the outside (don’t know how, stored hashes?) so I think that can be a “Gatekeeper” feature, removing the execution bits from certain files. Not tested, but probably a zip created in your system behaves differently from one “imported” (downloaded, copied).

4

Maybe Catalina just prefer signed DMGs as the compressed “secure” transport option of executables.

5

If you use Compress from Finder, I am pretty certain that you’ll get back what you put in. However, using Zip from another platform (i.e.: building a macOS app on Linux, or, in your case, using Christian’s standard Zip methods) and zipping there will result in the execution bits being cleared. So, it’s not a Catalina “bug” so much as a “feature.” Apple’s Catalina nanny-mode is highly infuriating.

6

Putting Xojo considerations aside for the moment, does this mean that if I use the Finder to compress an app on my High Sierra Mac and send that zip to a Catalina user, it won’t run after un-zipping under Catalina? What about a file zipped under Catalina and un-zipped on another Catalina machine?

7

More thoughts:

  • this only seems to affect Zip files made “by code” - e.g. using ZipMBS. I don’t know if other Zip libraries are affected.
  • Finder Zip (right-click, “Compress My App”) always works in both directions (compress on Catalina, works on Mojave, and vice versa)
  • My app is cross-platform (e.g. the Windows side can zip up a mac app) so this is a big problem for me. I’m pretty sure I can’t build a DMG on Windows.
  • How does Xojo do it? (I purely run Xojo IDE on mac). If you build a Mac app on Xojo under Windows, does it build a Zip file? If so, is that now broken too?
8

[quote=475833:@Michael Diehr]How does Xojo do it? (I purely run Xojo IDE on mac). If you build a Mac app on Xojo under Windows, does it build a Zip file? If so, is that now broken too?
[/quote]
The resulting built apps for macOS on Windows and Linux are just folders. It’s up to you to decide how to “wrap” them to get them to macOS.

[quote=475833:@Michael Diehr]this only seems to affect Zip files made “by code” - e.g. using ZipMBS. I don’t know if other Zip libraries are affected.
[/quote]
If it wasn’t created with Finder’s hooks into the Zip functions in libarchive.so, Catalina strips the executable bits on extract.

9

Myself as well as the developers of many apps I use distribute as ZIP without issue.

Michael you’ve explained how permissions on ZIP work, but I don’t see any methods with ZipMBS for actually setting them. Have you set the flags on the files within the zip or perhaps had they just previously been assumed and are no longer?

10

Using the built-in osx utility “zipinfo” I’m able to see the difference between the files:

' Zip created by ZipMBS
' zipinfo MyApp.App.zip
' -rw----     0.0 fat        0 b- defX 20-Feb-17 18:19 MyApp.app/
' -rw----     0.0 fat        0 b- defX 20-Feb-17 18:19 MyApp.app/Contents/
' -rw----     0.0 fat        0 b- defX 20-Feb-17 18:19 MyApp.app/Contents/MacOS/
' -rw----     0.0 fat 16300608 b- defX 20-Feb-17 18:19 MyApp.app/Contents/MacOS/MyApp

' Zip created by Catalina:
' zipinfo MyApp.zip
' drwxrwxr-x  2.0 unx        0 bx stor 20-Feb-17 17:17 MyApp.app/
' drwxrwxr-x  2.0 unx        0 bx stor 20-Feb-17 17:17 MyApp.app/Contents/
' drwxrwxr-x  2.0 unx        0 bx stor 20-Feb-17 17:17 MyApp.app/Contents/MacOS/
' -rw-rw-r--  2.0 unx 16300608 bX defN 20-Feb-17 17:17 MyApp.app/Contents/MacOS/MyApp

Notice the differences - the ‘version created by’ is ‘fat’ which means DOS, the version # is 0.0 instead of 2.0 and the file permissions are wrong as a result.

If I can’t fix this inside ZipMBS, I’ll see about writing a post-processor which updates the version # fields in the zip file afer creation.

11

And a solution which apparently works:

Catalina is more strict (or perhaps just “broken in a new way”) when processing zip files. So, when creating the ZipMBS file for an app which includes exectuable files, you need to set the “version created by” field to version # 2.0 and the OS to “unix” which is done by setting the version value to 0314 hex as shown here:

dim bRaw as boolean = false
dim zipWindowbits as integer = 15  // ignored
dim zipMemLevel as integer = 9 // ignored
dim zipStrategy as integer = ZipMBS.StrategyDefault
dim zipPassword as string=""
dim zipCRC as integer = 0
dim zipFlagBase as integer =0
dim zipFileComment as string =""
dim bUseZip64 as boolean = true

' 4.4.2 version made by (2 bytes)
' 
' 4.4.2.1 The upper byte indicates the compatibility of the file
' attribute information.  If the external file attributes 
' are compatible with MS-DOS and can be read by PKZIP for 
' DOS version 2.04g then this value will be zero.  If these 
' attributes are not compatible, then this value will 
' identify the host system on which the attributes are 
' compatible.  Software can use this information to determine
' the line record format for text files etc.  
' 
' 4.4.2.2 The current mappings are:
' 
' 0 - MS-DOS and OS/2 (FAT / VFAT / FAT32 file systems)
' 1 - Amiga                     2 - OpenVMS
' 3 - UNIX                      4 - VM/CMS
' 5 - Atari ST                  6 - OS/2 H.P.F.S.
' 7 - Macintosh                 8 - Z-System
' 9 - CP/M                     10 - Windows NTFS
' 11 - MVS (OS/390 - Z/OS)      12 - VSE
' 13 - Acorn Risc               14 - VFAT
' 15 - alternate MVS            16 - BeOS
' 17 - Tandem                   18 - OS/400
' 19 - OS X (Darwin)            20 thru 255 - unused
' 
' 4.4.2.3 The lower byte indicates the ZIP specification version 
' (the version of this document) supported by the software 
' used to encode the file.  The value/10 indicates the major 
' version number, and the value mod 10 is the minor version 
' number.  

dim zipVersionMadeBy as uint16 = &h0314 // 03 = unix. 14 hex = 20 decimal  =  version 2.0

z.CreateFile(relativePath+name,zi,"","", zipFileComment, z.MethodDeflated, z.CompressionBestCompression, bUseZip64,bRaw, zipWindowBits, zipMemLevel, zipStrategy, zipPassword, zipCRC, zipVersionMadeBy, zipFlagBase)

Note: zip files traditionally use no compression for Folders (I don’t think this matters, but…) when adding a folder (rather than a file) you might change it slightly to not use any compression at all:

z.CreateFile(relativePath+name,zi,"","", zipFileComment, z.MethodNone, z.CompressionNo, bUseZip64,bRaw, zipWindowBits, zipMemLevel, zipStrategy, zipPassword, zipCRC, zipVersionMadeBy, zipFlagBase)

Note: see also below for code to set the permissions flags: https://forum.xojo.com/conversation/post/475862

12

Yes, I didn’t show that code, but it is working. You set in the external file attribute flags - here’s the pseudo code:


dim zi as new ZipFileInfoMBS

[...] do other stuff here [...]


if isAFolderThen
  // to construct this value, enter the Unix permissions
  //  (e.g. 755) in Octal and convert to Hex (1ED) and then the value is  
  // 4xxx4000  repalcing the xxx with the Hex version of permissions

  flags = &h41ED4000   //  this gives proper 755 permission for folder when copied to OSX

elseif isAFile then

 if IsAnExecutable then
    // to construct this value, enter the Unix permissions
    // (e.g. 755) in Octal and convert to Hex (1ED) and then the value is
    // 8xxx4000  repalcing the xxx with the Hex version of permissions
    flags = &h81ED4000   // this gives 755 permission when copied to OSX
  else
    // a regular old file - non executable
   // to construct this value, enter the Unix permissions 
  // (e.g. 644) in Octal and convert to Hex (1A4) and then the value is
  // 8xxx4000  repalcing the xxx with the Hex version of permissions
     flags = &h81A44000 // gives 644 permissions on OSX
  end if
else

zi.ExternalFileAttributes = flags
13

In initial testing, I don’t think this is true: after making my fixes, I’m again able to create zip files using ZipMBS on either Mac or Windows OSs, copy the zip file to Catalina, double-click to decompress, and the resulting app now runs fine on catalina.

14

I’m very glad that you found the root of the issue and solved it. Even more so glad that you shared the solution with us! Thank you!

15

You are welcome! I emailed Christian of MBS as well to ask that he document these on his side.

16

FYI &O works for octal literals just like &H works for hex and &B for binary ones

// 4xxx4000  replacing the xxx with the Hex version of permissions

Dim base As Integer = &h40004000 

Dim hflags As Integer = &h41ED4000                          //  this gives proper 755 permission for folder when copied to OSX
Dim oflags As Integer = &O755                               // &b1 1110 1101 ->  &h1ED 
Dim bflags  As Integer = &b01000001111011010100000000000000 //                  &h41ED4000 

Dim result As Integer = Bitwise.ShiftLeft(oflags , 16) + base

Break
17

Thanks, Norman. Adding to the confusion, in C and (old) Python, the syntax for Octal is a leading 0 (zero) - e.g. “010” means “10 Octal” which is 8 decimal. Thankfully, Xojo doesn’t use this.

Thus, many of the numbers shown on https://unix.stackexchange.com/questions/14705/the-zip-formats-external-file-attribute are in Octal format. Once you figure that out that mess it’s a little easier.

18

[quote=475888:@Michael Diehr]Thanks, Norman. Adding to the confusion, in C and (old) Python, the syntax for Octal is a leading 0 (zero) - e.g. “010” means “10 Octal” which is 8 decimal. Thankfully, Xojo doesn’t use this.
[/quote]
OH gawd yes !!!

19

[quote=475862:@Michael Diehr][code]

dim zi as new ZipFileInfoMBS

[…] do other stuff here […]

if isAFolderThen
// to construct this value, enter the Unix permissions
// (e.g. 755) in Octal and convert to Hex (1ED) and then the value is
// 4xxx4000 repalcing the xxx with the Hex version of permissions

flags = &h41ED4000 // this gives proper 755 permission for folder when copied to OSX

elseif isAFile then

if IsAnExecutable then
// to construct this value, enter the Unix permissions
// (e.g. 755) in Octal and convert to Hex (1ED) and then the value is
// 8xxx4000 repalcing the xxx with the Hex version of permissions
flags = &h81ED4000 // this gives 755 permission when copied to OSX
else
// a regular old file - non executable
// to construct this value, enter the Unix permissions
// (e.g. 644) in Octal and convert to Hex (1A4) and then the value is
// 8xxx4000 repalcing the xxx with the Hex version of permissions
flags = &h81A44000 // gives 644 permissions on OSX
end if
else

zi.ExternalFileAttributes = flags
[/code][/quote]
If the original item has different permissions (e.g. the user wants to deny access for the “other users”), you lose the setting in the conversion. I guess you’d take the original item’s permissions and apply them in the code above.

20

Correct - my app is actually generating these .app files, so it knows what the permissions should be. If you are writing an app to zip up arbitrary files, it may be smarter to use the on-disk permissions instead. (Although this is not as simple as it seems - what should you do for example if you read an executable with the wrong permissions? Keep the incorrect permissions? Fix them? etc.)