Hardware platform & design suggestions for a project

Apologies in advance for the long post. Part of this is just me thinking out loud…

So I’m going to start building an application soon and I’m looking for some advice on the hardware. I’m also open to suggestions on the software. There are some constraints (physical), which I’ll get into below. We have been working on the hardware design for a motion picture film scanner for in-house use. It’s very high resolution (about 14.5k x about 10k pixels). Motion control is all done on a custom hardware board using an atmel processor (so more or less arduino compatible), and our LEDs are run via a custom driver board. The camera is Cameralink, and we’re using a Windows C++ SDK made by the cameralink manufacturer to interface with that. We’re at a stage now where all the pieces are starting to be assembled and I’m going to have to start writing code.

Here’s where it gets hairy: We’re scanning color images as monochrome, using R, G and B lighting, depending on the channel we’re capturing, These channels then get mixed together into a composite color image. The camera also uses PixelShift, which takes up to nine 4.8k shots for each frame and those get combined into the 14k master file. And on top of that, we may implement multi-flash HDR, which takes two or three exposures for each frame, each exposed for a different tonal range (shadows, mids, highs) and then stitch together the best parts of each frame into the final image. So in a worst case scenario, we’d be looking at 9 x 3 x 31MB images to make up a single channel (9 pixelshifts, 3 exposures, 31MB per 5k pre-pixelshift exposure). A final 14k frame takes about 235MB. Lots of data.

So in terms of the software design, my thought is to have four separate machines do the processing. Each would run a helper app that will do the pixelshift processing and generate a single color channel. Each will have its own local storage so we avoid thrashing a common shared drive. The fourth machine handles combining these, including the HDR work if we’re processing as HDR, and it would write the final file out to a shared volume (say, on our SAN). The machines could be connected together with some fast interconnect like 10Gb Ethernet. Or if I wanted to feel really masochistic, maybe even Infiniband, which can run much faster but is probably overkill.

There would be an application with a GUI that’s handling the camera, communicating with the motion control hardware and displaying scaled down frames as they’re done processing. It could communicate with the helper applications, or those could just be monitoring a watch folder and processing files as they arrive (I’m kind of leaning in that direction). Also, the speed at which this will happen is approaching glacial. The camera itself is only capable of doing the 9 pixelshift images at a total speed of .5 frames per second (x 3 frames). So we’re looking at around 2-3 seconds/ frame, in a worst case scenario, just for the camera to do its work. We may upgrade the camera if this project takes off, to one that can do 2-3 frames/second and eliminate the pixelshift processing, but that’s a ways off. We’d probably never have this running any faster than that, simply because of camera limitations at these resolutions.

I am looking to maximize performance within these parameters, because this all gets very slow when you’re talking about scanning 25,000 frames for a single reel of film. My concerns are primarily about the volume of data. I think trying to run this on one massive machine is going to be expensive (to build) and may not work due to limitations around bus and disk speeds. A cheaper solution might be to buy an old HP Blade server (I’ve looked into it - a machine with 4 blades, each with dual Xeon 4-core CPUs, 16GB RAM, 2x 300GB 10k drives, as well as the shared 10G network interface module, the blade controller, and KVM interface is only about $1500 CTO on ebay). Granted, it’s a beast and it’s not a speed demon (dual Xeon X5570 CPUs are vintage 2009).

One other argument for the blade server, if we go this route: the chassis for the film scanner has a monstrous steel frame. We have removed almost all of the original video processing hardware from it, and that leaves us a chassis with three vertical sections inside the frame: Left, Middle, Right. The middle contains the camera and light source, so that’s off limits, but the left and right have built-in 12U 19" racks, so we can install the computers directly into the chassis. One of the reasons I was thinking about the blade server is thermal management. We can partition off the interior of the chassis so that the heat producing elements (the computers) are basically venting front to back through the unit, and not affecting the camera in the middle. With a blade server, being designed to handle heat efficiently, we can just build a simple box that directs the heat out the back. With a stack of separate servers, which would be more expensive to set up, we would have a tougher time managing all the heat generated, I think.

So I’m looking for suggestions on how to manage this kind of image processing, both from the hardware and the software side, on somewhat of a budget. …Big picture stuff, but also things like streamlining the data flow from the camera to the final image. One person suggesting piping the output of the camera to my processing software which then gets piped to the machine that does the final image output, to try to avoid the data ever touching a disk until the very end. No idea how to do that with Xojo, but I’m open to suggestions.

Thoughts?