Currently IDE and Compiler both work on same thread and I’d like to change that.
So currently the process seems to work:
IDE starts compiler class
Compiler initializes
IDE sends framework stuff to compiler
For each project item
IDE assembles a project item as text and calls Compile(code)
Compiler plugin compiles it.
IDE receives build item
Next
IDE passes items to linker
We could change it:
IDE starts compiler class
Compiler class starts a new pre-emptive thread on different CPU core
Compiler initialization runs
IDE sends framework stuff to compiler to queue the jobs
For each project item
IDE assembles a project item as text and calls Compile(code)
Compiler puts the job on a queue and returns build item object
IDE checks if one of the build items got flagged with error
Next
IDE calls method in Compiler Plugin to set IDEDone flag
IDE waits for Compiler plugin to have all jobs done and set CompilerDone.
IDE passes items to linker
The compiler plugin would run a thread, which itself does:
Initialize Compiler
While not IDEDone
Pick job (build item) from queue
Compile it
Put result into build item object and set finished and error flags as needed
Wend
Put in flag for CompilerDone
Exit thread
Key thing is that the whole compiler C++ code runs on one thread for the time of compilation. The Xojo objects are handled by the IDE on the main thread. You’d need a mutex to synchronize access within the Compiler plugin for the queue and error flag. Calling Compile(code) should create the BuildItem object and put it in an array. Put in source code to a property and also a Ptr to const char* for C++ code to read. Compiler thread later can read that text. If errors happens, you’d need to store error info in C++ objects, e.g. put into a std::vector as a ptr into the BuildItem data structure. The IDE then later calls method to picks that up and get that as Xojo objects in main thread. No big synchronization needed as compiler can allocate and fill the the std::vector object and then on the end put the ptr to it into a field of the BuildItem object.
If this works perfectly, the compiler code with LLVM can run on a different thread and thus the IDE can assemble the next jobs while compile compiles the last one, which may speed up the whole process. I expect that 30% of the time could be saved on every compile run.
If you need help on implementation, please reach out to me.
PS: Of course please do some measurement first to check how much time is needed for assembling items vs. compiler them and estimate how much time could be saved.
This is an issue that many long-term Xojo engineer want to fix at one time or another, I certainly did somewhere along the line and I know that several others did before me.
I’m not saying that what you you’re describing is impossible, but may simply be insurmountable. The amount of time required to make a change like this, the amount of code that would need to be touched and the possible side-effects and time needed to correct them make this a very large project that would probably require multiple dedicated experienced IDE engineers multiple months to design and accomplish. IIRC my estimate was that the “bang for buck” was not high enough for enough users. Basically, it requires separation the rendering function of the IDE from the UI part (which are tightly coupled and have been that way for at least 15 years), the same hurdle that prevents having a command-line compiler. Again, not impossible, just something that would need to offer a bigger impact for more people.
Not to get off-topic, but my suggestions to users who are having “really long build times” are:
Update to Xojo 2022r1+. There were several changes that improve project loading and compiling.
Set up a CI/CD build system so builds are automatically done on another machine. I think you’ll be pleasantly surprised about all the time it ultimately saves you in that your projects are built exactly the same every time, essentially “in the background”, whether it’s Jenkins, GoCD or some other solution.
FWIW, I use Jenkins at my new job, and it would certainly be possible to have an entire “build system” on a single M1 Mac if you wanted to. Jenkins provides a Docker image which can pretty much be used out of the box and the “builders” could be one or more virtual machines running Big Sur and/or Monterey. You might also want to use an Intel Mac Mini with 64GB of RAM. It wouldn’t be as fast, but it would be handy for running unit tests on multiple versions of macOS or for running other operating systems that are Intel only. VMs are handy because you can snapshot a last known config and quickly get back to that if the machine is somehow corrupted.
Once you’ve got a system like this set up, you can easily do on-demand, nightly or even hourly builds of your project(s).
I don‘t care about compile times when it comes to a final build. I have a build script that handles all different targets and if this one runs 15 minutes or 30, so what.
But what‘s really important to me is the build time when I do a debug run. This is often really fast after the first run, even if more than one single line is changed. But adding a simple property often requires a complete recompilation of the whole project and this is sometimes very annoying. Anyway, I think Xojo could make many developers happy by reducing compile times noticeable.