We need to do the following, on fairly high resolution images (think 14k x 10k pixels, @16bit - massive):
Color transforms
Scaling
cropping
Color inversion
3D LUT application
Machine vision (alignment images using common reference markers in the image)
Multi-channel color merging (three monochrome images to one RGB)
Potentially: image stacking and alignment, tone mapping
there’s probably a bit more, but those are the basics
OpenCV is where we’d like to do this because it would be helpful to have some of it done on a fast GPU and that’s built into some OpenCV functions, but mostly it’s that I’d rather do it from within Xojo, which I’m already comfortable with. The alternative option would be to hire someone to write some command line tools that do these functions, and we call it from Xojo, but it seems like it’d be more seamless if I can just do it within the Xojo environment.
The main questions i have are:
is there a 64bit version of this working? i’m aware of the code here but it says quite clearly that it’s 32bit. It was written several years ago and doesn’t seem to be up to date.
Is there an alternative library I can get for Xojo that integrates current OpenCV versions?
Is there an alternative to OpenCV that does some of the same stuff?
Yeah, I’ve done a bunch with ImageMagick/GraphicsMagick both inside and outside Xojo, and it can be used to do the scaling/color space/tone mapping/etc, but I don’t think it does the machine vision stuff, and because OpenCV does, and it also has the ability to do all the other things, it seems like a more natural fit.
Description: OpenCV is a library that is mainly aimed at real time
computer vision. Some example areas would be
Human-Computer Interaction (HCI), Object Identification,
Segmentation and Recognition, Face Recognition, Gesture
Recognition, Motion Tracking, Ego Motion, Motion
Understanding, Structure From Motion (SFM), and Mobile
Robotics.
Homepage: https://opencv.org
Well, our system will be running on Windows, so no. But I’ll look to see what I can find on win.
My concern with using a separate command line app is that the kind of work we’re doing would best be done in-memory. By calling an independent application this does get trickier, and may result in having to make temp files. When you’re dealing with a file that’s about 280MB in size, with three of those per image (red, green, blue channels), there’s a lot of file I/O if you’re writing temp files to disk.
So my plan is to pull the raw image data from memory off the frame grabber into Xojo, then keep it in RAM until we’re done with the processing. I haven’t tried this, but can I pipe the output of a command line tool into memory, in such a way that Xojo can access it?
You may want to scale images down first either way. We also have Machine Learning classes for macOS and Windows available.
I’d like to take a look at these - which ones are they?
I’ve tested your ImageMagick/GraphicsMagick plugins in the past and they worked nicely. I need to consult with our engineer who is designing all the color science for this system to see precisely what the various transforms will be, and then I can set up some tests.
All of this goes a little beyond my skill set, but given a recipe I can do the coding. I’ll see what I can get from him in the next couple days and follow up with a list of specific things we need to do.
OpenCV might not be able to use the GPU for dimensions as large as that so might actually fall back to the CPU.
Moving large amount of data between the GPU & CPU has a time cost. You could find that it takes longer on the GPU than CPU.
Tasks such as machine vision usually operate on a scaled down version of the picture and not the high resolution original.
We’d likely put one of the new nvidia RTX3090 cards in there, which have 24GB of RAM, so not being able to process the files there is unlikely.
There’s definitely a cost in moving the files across the bus and back, but the PC we’re building for this will be built specifically for this purpose, and the bandwidth requirements should fall well within PCIe 3.0 specs.
We have a commercially produced film scanner that can capture at up to 6.5k x 4.8k and it’s doing machine vision stuff in the GPU in real time as it’s capturing (basically aligning the perforations of the film frame by frame, to register them in the right position, color space stuff, cropping, scaling) - that’s the same thing we need to do: locate the perforations in the film, align them to known X/Y coordinates. That machine is able to do that on a pretty modest set of GPUs (ping-ponging the frames so that each one is working on a different frame). We don’t need that kind of speed, so we should be able to do this easily with a single GPU.
I’m not opposed to doing this kind of thing in the CPU, but my experience as a user of systems that rely heavily on GPU processing is that these kinds of functions: scaling, color transforms, etc, are faster than on machines that do it in the CPU. We work with several kinds of applications, and the ones that rely on GPUs are almost across the board faster than those that rely on CPU (and we’re talking about machines with 28 physical Xeon cores running at 3GHZ, so pretty high end stuff).
Yeah, I haven’t been able to find an existing command line tool that does this. We’re on Windows, but even the Mac Ports reference above seems to just be for the libraries, not a command line application. That said, it doesn’t help us since we’re not doing this on a Mac.
They do list Windows, but even if they are just libraries, you should be able to Declare into them. A lot more work than a command line tool or plugin, but should be doable.
Yes, I was just hoping to not have to do that, which is why I was looking at the OpenCV Xojo project. It’s a lot of extra work/debug time, and we’re on a bit of a deadline – with a limited budget (you don’t want to know how much the camera we’re using cost us!). This was originally going to be work done by the engineer who is doing a bunch of other low level stuff, but he’s got too much on his plate in the time we have, so I’m looking to offload some of that work from him.
…just not enough time to start messing with something that’s as nuts-and-bolts as the raw libraries.
As someone who’s done something very similar using Xojo for the Mac (machdr.com). Below are a couple of my thoughts.
You need to own the entire toolchain. Relying on third parties to support their frameworks is a recipe for disaster. I hope MS have a far better quality control than Apple, because our app features many workarounds for different versions of the OS, whereby functionality is broken or doesn’t work right. For instance macOS 10.14 doesn’t handle certain 16-Bit images correctly and on load will produce a corrupted image. If I had access to the code that loads images, I could fix it, I don’t. Not to mention that to do work on the GPU requires the use of OpenGL or Metal, both have issues on different version of the macOS. Or Apple’s GPU based image editing is supposed to support tiling, so that it can process large images on lesser machines, except that’s broken for macOS 10.13, whereby it renders one tile and then gives up. Or 10.14 just fails to render certain images with certain filters.
I know of a very talented developer who wrote their own GPU based image processing library for the Apple platform, they got fed up with each OS update breaking their library, so they switched to using Apple’s… Only to find that it gets frequently broken also. They regret the change, because now they’re no longer in control.
Performance and memory. when combining the individual exposures into a 128-Bit floating point HDR, I actually use a custom built Xojo plugin (written in Objective-C and utilizing GCD). This way I can take advantage of multi-core processors, sharing the raw image data between each thread. This part was actually way easier than I anticipated, the first draft of the plugin worked on the very first run. I had to tweak it to get maximum performance. I can use a Xojo console application and shared memory declares to do this without, but that is actually more work.
Unless you’ve got funding or a solid marketing plan, I wouldn’t recommend anyone else attempt this on their own. It’s soul destroying to spend 27 months of your life building an application and to see it disappear into the deep dark depths of the App Store.
Thanks for the feedback - This is not a commercial application we’re building, it’s for in-house use. So once we get it working, that machine is frozen in time, more or less. We don’t update most of our in-house machines unless we have a really compelling reason, for the exact reasons you mention - something always breaks, and we rely on these machines to do what they do repeatably, day in and day out. We don’t want surprises.
If we were doing this commercially, yes, we’d write it all from scratch. but once we have it working, there would be no reason to upgrade anything unless forced to by an OS upgrade or something like that - which we can easily avoid by not having the PC on the internet.
Revisiting this thread. Since my original post, we have decided to try building this on the mac, but may revert to Windows if it ends up we need more speed and have to custom build a PC.
The primary goal of my work this week is to put together a simple benchmarking app for both Mac and Windows, to see which methods of image processing work best with the massive images we’re using, and which CPUs are faster across a variety of machines.
This is what we’re dealing with:
Frames are up to 14192 x 10640, 16bit monochrome (~300MB/frame)
We need 3-4fps processing speed (preferably a bit more, just to give some headroom)
Image acquisition, from camera to our application, is about 100ms for each frame (or at least, that’s the goal)
On the Mac, it seems the only way to access OpenCV, short of writing a C/C++ library that currently doesn’t exist, is via python. We think this will pose issues in terms of performance, if called from Xojo, primarily because the files would need to be written to disk (even if we use a RAM disk) before we can send them to Python, and then written to disk again by the python script, before we get them back in Xojo. Is there a way to pass the raw data that’s in memory after the capture to a python script, then send the results back to Xojo – entirely in memory without any disk (or RAM Disk) usage? If so, how?