Strip unused images (resources) from builds

Short version:
There is a way/option to copy only used (i.e. code referenced) resources into the executable build?

Context:
It seems reasonable to have a shared folder of resources that you can re-use on multiple projects (imagine a folder with many subfolders that contains icons, background, sound, cursors…)
During development the entire parent folder is dragged into the project to have access to all resources but this also means that all those resources are copied (and exposed) inside the final product.
Sure that you potentially load resources at run-time from code (i.e. by opening the file) but if you don’t do this and you use direct references in code, it should be very useful (and not so difficult simple to implement) to have an option that skip the copy on unused resources during compilation.

[quote=243960:@Pier Luigi Covarelli]Short version:
There is a way/option to copy only used (i.e. code referenced) resources into the executable build?[/quote]

Not at the moment. The compiler currently doesn’t associate resources with code at the granularity required to know what should survive dead code stripping and what shouldn’t, though I’d like it to.

Thank you for your quick answer.

I’m wondering if it is possible to walkaround this.
Now, if I write:
myCanvas.backdrop=myimage
and myimage in not in project I got an error.
This means that there is already a check that lookup the referenced resource.
Assuming that the copy of all resources in project is handled by the compiler, when you check a referenced resources you could add code that insert items into a list of used resources and later, AFTER the compilation stage, just delete those not into the list into the build.
If also the above check is made by compiler, a way could be to pre-analize the code before compilation and build the above list.

[quote=243971:@Pier Luigi Covarelli]This means that there is already a check that lookup the referenced resource.
[/quote]

Sort of. The IDE generates code to load the resource at runtime, which is basically just a module with a global function in it, and hands that code off to the compiler. The compiler doesn’t get enough information to know that if that global function is unused the resource can be discarded.

To know which images are used, remove them all, then note which ones are missing, then put back only those.

Yes Michel , but this is tedious thing with a large number of files inside nested folders.

But this could be a way to walkaround the compiler limitations that could be automatized by Joe :wink:

  1. start the compilation process with resources (of type images) disabled/ignored (like if the user removed all of them from the project)
  2. collect the errors of missing items to build the list of used items
  3. compile the project normally
  4. strip from the build the resources not into the list above

Pier, you’ve been around long enough you should know that suggesting specific implementation details is pointless, as is making comments along the lines of “that should be simple”. Request the feature/function and let it go.

Facts suggests that a complier approach should be complex for the simple reason that RB/RS/XOJO is around from many years and unused resource stripping is a reasonable but still missing feature.
I’ve been around long enough to know that discussions could lead to alternative ideas and solutions for a given problem thus I use a forum as a forum and not as a wall for requests/complaints.

Actually there is no way to “know” with 100% certainity if any particular resource (be it an image, a text file or any other in-project item) is being (or will possibly ever be used during the execution of the app.

case in point (and this is not syntax correct, just an illustration)

s="myimage"
if answer="yes" then s=s+"01"
f=app.executable.parent.parent.resource(s+".png")

myImage01.png might be used… but there is no design time indication, no could the compiler really know either.

Another example is when the resource is used but not directly. All my apps use an html page inside the Resources folder, which itself uses pictures. For the compiler they are unused, but yet they are necessary for my program.

If there were an automatic removal of resources not used directly by the compiler, I would have to resort to an extra copyfile instead of simply dragging them into the project.

Yes Dave and Michael but, as I said into the first post, I’m referring to resources that are referenced directly (statically) in code with their name into the project and not to files that are opened dynamically at runtime. Also the behavior should be controllable by the user (at a file or parent folder level) .

The point is… if they ARE referenced, then you DON’t want to strip them out
If they are NOT referenced DIRECTLY, you don’t KNOW if they are necessary

So this is a folly that XOJO engineers have better things to fix than something as worthless as this.

Hi Dave,
your second sentence in not always true:

  • you’ve a folder organized in subfolders that contains your full collection of icons and artworks shared between projects
  • to avoid individual file import and have all resources availables during development, you drag the entire parent folder into the project (imagine that 1000 items are imported)
  • drag a canvas into the window ad set it backdrop to one of that images
  • build the project anche check Resourced folder of the app
    All 1000 images are inside the Resources folder but 999 of them are not referenced directly from code AND YOU ALSO KNOW (since you’re the developer of the app) that you’ll never use and load them dynamically at runtime

If you’ve an option that automatically not include the “unused images” (i.e not directly referenced AND known to not be loaded at runtime) , the final build is much smaller and also you not distribute your entire icons collection and graphics inside every app.

That said, Michel suggestion before final build is te actual solution (despite need some manual work) and I also agree that XOJO engineers have better things to fix.

let me rephrase

If you the developer dragged 1000 images in just because it was “easy”… then the onus is on you

yes, but thanks to a proper piece of code, you could retain the “easy way” and still optimize things (it’s called automation and is one of the most useful effects of software)

I think the compiler should check which global function is discarded and have a link for the compiler generated getters for global projects items to make sure they get discarded with the item they represent if not used.

If you want to include a folder of images, please do so in a build step.

[quote=244423:@Dave S]let me rephrase

If you the developer dragged 1000 images in just because it was “easy”… then the onus is on you[/quote]
Been there, just dealt with that … 1,132 images because the original coder didn’t know which ones he would eventually use …

I finally used a variation of Michel’s process above - deleted ALL of the Picture lines from my rbvcp file, did a Cmd-K and created a list of the missing image files. I then added the missing 143 files (yep 1/10 of what the original coder dropped) back into the project and built it properly.

At one time the RB compiler / linked was quite good at stripping unused stuff. For example, it would strip dynamic string constants that weren’t referenced in code and would also strip external plugins if the code that called them was not referenced.

When we migrated some projects to Xojo I was shocked to find out how dumb it had become. I vaguely remember a conversation where Norman thought that the introduction of Introspection may have changed things.

The introduction of the text data type and the fact that ICU is always included no matter if it is used or not, shows that things haven’t got any better.

Maybe one day the effort will be put in to making the process more efficient but I’m not holding my breath.

[quote=244610:@Kevin Gale]At one time the RB compiler / linked was quite good at stripping unused stuff. For example, it would strip dynamic string constants that weren’t referenced in code and would also strip external plugins if the code that called them was not referenced.

When we migrated some projects to Xojo I was shocked to find out how dumb it had become. I vaguely remember a conversation where Norman thought that the introduction of Introspection may have changed things.[/quote]

The stripping code in the linker has not changed in the last five years. It uses a simple but effective algorithm where there are a fixed set of roots, like the app object, and the linker traverses all reachable nodes* and marks them live. Only live nodes make it into the final executable.

Introspection is a double-edged sword, which allows runtime flexibility at the cost of hindering optimization. The introspection metadata references all of the methods, properties, and the types involved. This precludes optimizations like stripping out “unused” properties and methods because the compiler cannot prove at compile time they are actually unused.

Unfortunately the introspection module isn’t the only thing relying on the metadata. At the moment it’s also used for:

  • The plugin SDK’s REALLoadObjectMethod, REALSetPropValue, and REALGetPropValue functions.
  • XojoScript uses it to know what gets exposed to the script via the context object.
  • The runtime itself uses it to glue together C++ code to Xojo objects.

Now, back to the question of resources. The compiler does not currently associate resources with functions, so it has no way of knowing if it’s still required. It’s hardly an intractable problem, just a lower priority one.

  • A node is any linkable item like a method, property, or introspection metadata.

Hi Joe

We were using RB versions from approx. 8 years ago. The new algorithm is obviously not as sophisticated as the previous one.

It would be great if the process could be improved in future versions so that it could act more intelligently or possibly the ability to report on unused functions, constants, resources etc…

Kev