Setting up a client-server render farm

A client of ours is now requiring that we deliver files to them in a specific media packaging format, using a specific command line application. This application is not fast. We do film scanning, and the files are massive - many terabytes for a typical high res motion picture scan. The software they’re requiring creates lossless compressed packages, but the time to create them is crazy. a 60 minute film takes about 36 hours of nonstop compute time on a pretty fast mac Studio running a 12-core M2 Max processor with 64GB RAM (and the slowdowns aren’t disk or network, we’ve tested). Then theres about 40% of that again to decompress the file and verify that it’s ok.

The software they require uses ffmpeg to do some of the work, and ffmpeg is actually pretty fast at what it does, taking advantage of multiple cores. But the rest of the app is not multi-core aware, and is painfully slow. We tried running multiple instances of the app concurrently, but while it uses minimal CPU resources, it uses maximum RAM. So two instances concurrently was actually significantly slower than running them sequentially because the system started paging memory and really slowing down.

In order to speed this up, we are looking into building a render farm of fast, low core-count headless workstations with 10GbE, which would be connected to our SAN. Probably Linux to keep the cost and complexity down. The thinking is that I will write a batch application that can be run from a desktop that’s on the SAN, and it will mange sending the jobs to available machines on the cluster as they finish up. We’d need a command line app on the linux machine to listen for jobs and accept them, and then those would get passed along to the command line app that does the compression on that local machine, then report back when it’s finished.

What is the best, simplest way to set up something like this? Are there example projects for a setup like this? (also, where are the example projects now? I couldn’t find them in 2023R4), or maybe someone has a plugin that will facilitate this kind of thing?

They are on the cloud now, you need to open a new project and select Examples to download the one you want.

1 Like

You should research a message queue, such as RabbitMQ. The concept is that you add jobs to the queue and your headless workstations work through all of the jobs and mark them when completed. You can do the same thing with a database table but a message queue is a more industrial solution. @Christian_Schmitz has a Xojo plugin for RabbitMQ.

1 Like

Thanks. I’ll check that out.

Ahh - got it. I’ve always opened them from the Examples folder, and just couldn’t find it!