I am using the shell to use use some open source command line software to produce PNGs of Chemical structures… It is uses Cairo under the hood.
It is meant to output the resulting PNG to a file, but if you don’t include a file name it sends it to StdOut as string which I get in Shell.Result and convert to a Xojo picture using Picture.FromData
That works well, but only to a point… If I make the PNGs larger than some size TBD (So maybe related to data size), a picture that Xojo can display is produced, resulting picture is only partially correct… It sort of gets chopped… but mot exactly that…
But If output is to a file and then I load that file, the picture is exactly as expected…
Is there a limit on the amount out data that I get from Shell.Result or is the problem likely with Picture.From Data or is it something else?
From what I can find in-line It looks to be StdOut buffering issue …
My situation is this… I have a database with about 15000 chemical compounds… They each have a very compact (compared to a graphic) printable string in a standard format that the describe the structure. The open source software can use to generate PNGs of the structure.
I need to use the PNG’s both on screen and in reports (in PDF format)… The PDF reports can have up a 1000 PNG structures embedded… so speed matters…
The fastest option at run time would be to limit the size to a single one and create the 15000+ pngs beforehand… besides the compromise on resolution, that is problematic for reasons I don’t want to go into now.
So I want to be able to generate the pngs on the fly and in the optimal size fro teh specific use0 required… but the PNG generation is on the slow side going through shell without using a file already so I want to avoid that overhead by keeping everything in RAM.
But I don’t see a way to do it, at least with the xojo framework…
A RAMDisk might work, but Xojo does not have a true one built in (From the docs it looks VirtualVolumns are purely an Xojo things so the OS could not directly write a file to one)…
As much as it causes outer issues, I may have to go to fixed size pre-generated PNGs.
Why would pre-generating 15,000 images be more resource friendly than generating images as needed as files?
I think you’re trying to over-optimize in a way that’s not actually beneficial.
Hm…The issue is always going to be that you’re dealing with 15,000 images. There’s no fast way of getting them into memory so you can use them later on. If you read from disk you’re limited by disk I/O. Obviously you’ve run into issues with shell.
Silly thought that might be worth looking at. How slow is the compiled app at opening if you put those 15,000 images into the project so you can reference them directly? I’ve never done more than a hundred images in a project file so this might be totally impractical. Just wondering out loud.
Is there some sort of logical grouping where you can get the most used chemical structures out of the way and then only load the rest as needed?
Another thought is grouping the images into larger tiled images and then grabbing the ones you need. This is similar to how games used to speed things up. Rather than loading hundreds of small files they’d load the one big one and then algorithmically get the individual frames of an animation.
[quote=413490:@Tim Parnell]Why would pre-generating 15,000 images be more resource friendly than generating images as needed as files?
I think you’re trying to over-optimize in a way that’s not actually beneficial.[/quote]
While pre-generating means the PNG are not optimal for all uses, it makes report production MUCH faster… Using that open source software, the structure PNG generation process is expensive…
Under the hood the code needs to parse the molecule description and from that create an internal representation of the atom positions, and from that information certain things about the bonds between atoms has to be deduced (such as aromaticity) and then the 2D PNG created from that…
If I generated on the fly and saved and saved them in different sizes, I could wind up with as many as 45K pngs in time, and I need to keep copying all the files between machines (Don’t have a server at work)… If I generated the files on the fly and deleted them (besides speed issues) that is a lot of disk access and so a lot of -wear and tear on the disk.
So If I am going to have to generate files then, for speed and practicality I might as well just pre-generate them at one size and just scale as needed (with the quality issues that entails)
[quote=413502:@Bob Keeney]Hm…The issue is always going to be that you’re dealing with 15,000 images. There’s no fast way of getting them into memory so you can use them later on.
[/quote]
Thanks for teh suggestion Bob!
I never need all of them in memory at once… and loading them from a file is not THAT slow… but generating them on the fly (which would give the best quality) seems to be, unless I can find a way to speed it up…
It’s a shame Xojo does not support converting SVGs to pictures as that open source software can output the structures as SVGs. Then scaling would not be an issue!
What am doing is taking the output of an instrument that assigns possible compounds IDs to the peaks, and I am using the info in a database to pick which one is most likely teh correct one, and then to group the compounds found by type and creating a report about the sample with all the compound structures (and relative amounts)as well as a summary by type using stacked bar and line graphs.
[quote]
Another thought is grouping the images into larger tiled images and then grabbing the ones you need. This is similar to how games used to speed things up. Rather than loading hundreds of small files they’d load the one big one and then algorithmically get the individual frames of an animation.[/quote]
As pre-generation to file (though not optimal for quality or speed) allows report generation speed that is not unreasonable, I may just go with that… I was just hoping to find a way to do better.
I too had some “fun” with depicting molecular structures some years ago. At the time, molecular structures I used were stored as MDL MOL format or SMILES strings and I used OpenBabel to convert between the formats.
For the depiction itself, I used ‘mol2ps’ to create an EPS file then ‘ps2pdf’ to convert it to PDF.
I just tried another way that you may find useful: I generated an SVG image of the molecule that I converted to vector PDF with ‘librsvg’ (which is based on Cairo and available through homebrew if you work on a Mac).
I too had some “fun” with depicting molecular structures some years ago. At the time, molecular structures I used were stored as MDL MOL format or SMILES strings and I used OpenBabel to convert between the formats.
For the depiction itself, I used ‘mol2ps’ to create an EPS file then ‘ps2pdf’ to convert it to PDF.
I just tried another way that you may find useful: I generated an SVG image of the molecule that I converted to vector PDF with ‘librsvg’ (which is based on Cairo and available through homebrew if you work on a Mac).[/quote]
Thanks,
I I may look into that for teh future, but I think I have found a path forward for this project/
This needs to run both on Macs and PC’s. I am using Open Babel and starting with either a SMILES or InChI string.
These days Open babe; can produce PNGs or SVGs from those. (I also need some of its other capabilities such as substructure matching and similarity determination)
Ilam producing the PDFs using the Einhugar plugin and with that I can embed Xojo pictures in the report where needed easily.
I’m going either have Open Babel produce pngs which I can scale down when needed or SVGs using a modified form (to deal with the Openbabel SVG internal format) of Alwyn’s code to create the Xojo pictures (so scaling looks good) if it is fast enough.
If quality of the image is the main concern in that case, why not generate the images larger than needed and scale them? This is how I handle this and Xojo’s scaling is very good.
Or, as Bob mentioned, pre-generate them and add them to your project. The resulting app will be larger because the Resources folder will contain all of the PNGs, but it will be fast and you’ll have exactly the images that you need. Concerning how many images a project can contain, just look at the IDE’s Resources folder :D.