IME of SQlite, and from lurking on the SQLite Forum, this should work; each thread with its own connection (that is, do not share a connection between threads).
Iām pretty sure that using Thread.AddUserInterfaceUpdate to write to the log file will make it easy to handle multiple threads finishing at the same time.
1 Like
Iām REALLY hoping that pre-emptive threads will help with this situation:
I use GraffitiSuiteās excellent GraffitiEditor, which runs in an instance of DesktopHTMLViewer. Whenever I have an instance of GraffitiEditor visible on screen, currently I CANNOT allow any other threads to wake up and do anything on Windows, as it causes all kinds of weird issuesā¦ everything from a stack overflow (which reports a stack exactly 1 level deep) to just locking up the app forever with the spinning wheel of death.
Iāve been back and forth with @Anthony_G_Cyphers regarding this, and weāve pretty conclusively demonstrated this is a framework problem related to how the DesktopHTMLViewer handles all the Javascript stuff that has to happen with every keystroke in the GraffitiEditor.
If there are no other (current co-operative) threads running at all, the editor is rock-solid. If any other threads fire up to do anything (compute a database value, compress some images in the background, re-calculate some stats, do some networking job, etc) then I can pretty reliably cause an app crash.
This only ever happens on Windows, and generally only on Windows machines that are a little anaemic / under-powered. Especially prevalent with Windows 11 installs that were originally just barely enough computer to run Windows 10. Seems to be a race condition deep in the bowels of the Xojo frameworks manifesting itself in very unexpected ways, which is why it only really happens on older/slower hardware.
TL;DR: Iām just hoping for a bug fix caused by current threading model.
1 Like
I would expect more bugs, not fewer, for quite some time.
1 Like
Yeah, the realist in me fully has that expectation. sigh.
If preemptive threads also prevents the app to lock while input/output is being waited for (e.g. on a āslowā network, when reading a damaged CD, in folders with thousands of items, etc.), a fact which hasnāt been discussed in this forum as far as I know, I would benefit from them in many of my apps.
Iām just not sure whether currently the app hangs as a whole because the threads are cooperative or whether this doesnāt matter.
Currently, any main stop is an entire app stop, when the event loop that depends on cooperation to continue, stops.
The theory is that your needs may be satisfied if well designed under preemptive threads. But there are possibilities that you at some level, put, or need to put, some essential part of the program, any part, waiting for something, and that something taking some time, and you will feel that wait, a hiccup, a bad performance or even long pause due to the nature of such operation and framework prowess to handle it as fast as possible and with less global interruption as possible.
This is something I try to avoid in my apps, even if it comes at the expense of the background taskās performance. Fortunately, this is not a problem for our needs. 
Thanks for your reply.
My question was specific to file input/output when the app hangs, though. I know fairly well normal flow threading in Xojo (when no OS function comes into play).
My application (macOS & Windows) does a lot of background astronomical numerical computations that need to happen in real-time. Currently I do this in non-preemptive threads and it takes a lot of juggling and throttling to stop those threads from affecting UI responsiveness. Preemptive is the way to go here, for sure.
The application also has many other BG threads that, for example, read enormous data files, process, and populate a SQLite database. These would benefit greatly from preemptive threading.
I had contemplated using Workers, but the overhead of string-based communication is just not worth the effort, and Workers (which I have used in other apps) feel like a compromise/kludge.
2 Likes
If such file I/O is shared with other parts of your app, you will need to protected the parallel access by yourself (orā¦ the framework will take care of it, causing waits and queued access, not sure what will come) and such think will be felt. If your file I/O is exclusive to your service thread, you probably will be luck and have a great non blocking enhanced performance gain.
The obvious choice for preemptive threads is processing large datasets in parallel.
Taking a stats app I have, it can be asked to process 100s of analyses in a single menu command. Currently I have a thread that takes each analysis and processes it, following on to the next and the next. If I could hand off each analysis to preemptive threads to use more than one core that would greatly speed up my processing time.
I canāt really do it with workers as it can require vast datasets and loading those into each worker would take a long time and a lot of memory.
1 Like
I maintain a few desktop apps that make excessive use of my own custom helper classes, where some are accessing measurement instruments and one is driving signal generators.
In the first case helpers are mainly to keep the GUI responsive and fluffy, because some of the instrument API calls are blocking and can make the GUI behave beachbally. The drawback is of course an enormous overhead of communication going on and the problems of debugging a running console app when driven by a desktop app. Whole app package is quite big too because of the multitude of different helpers included.
The generator helpers in the second case need to have tightly timed control over the generators, and there can be a multitude of them running at the same time, so having them run as separate tasks with the desktop app only programming them was the best way to reach this performance.
In short having preemptive tasks could reduce a lot of unnecessary protocol overhead and offer much better debugging possibilities. One task per instrument/generator.
4 Likes
Similar situation to a few people here. I have a desktop app that communicates with many external systems using Serial, Sockets, and TCP/UDP. I pass off the processing to cooperative threads which works most of the time, however, as the load gets heavier the UI starts to slow down. Most my systems have at least 4 cores, it would be great to spread the load out a bit and make better use of system resources.
I also use helper apps and async shellās to do file I/O or talk with external systems as some of the files are large or slow to read (/proc, /sys, etc.). I hope that FolderItem/BinaryStream/TextStreamās are made thread safe.
I just tested reading from the same BinaryStream from multiple threads and do not (yet) have an issue. Iām not even using a Semaphore, although generally, when in doubt, that is recommended.
2 Likes
And ditto writing to the same BinaryStream from multiple threads.
I tested both in a simple way without protecting the BinaryStream and had no issue. In production code, though, I would protect it before a read or write.
When you do this kind of testing, do you set up a scenario where it is very likely/guaranteed that the threads will interrupt each other? How aggressive are you in trying to prompt that behavior?
Thatās a good question. Iām using the framework I developed to run these Xojo class/function tests on up to 10 threads simultaneously, and the functions ultimately get called 50 times across those threads. But there is no guarantee that 10 threads actually launch since my framework will only start the number of threads needed to handle the incoming data. In other words, if the data is processed faster than you can supply it, it wonāt launch additional, unneeded threads.
Having said that, based on your comment, I checked, and yes, itās using 10 threads that are simultaneously reading or writing to a file.
So pretty aggressive, Iād say.
Thanks, good information. Of course any resource that will shared should have some protection in the form of mutex or semephore. Just trying to get an idea of the scope of the initial support and how hard the failures might be.
Have you been able to cause some race conditions on purpose? How does it react, any exceptions or hard crashes?
Depends on the condition. For example, if two threads try this without protection:
SomeIntProp = SomeIntProp + 1
The value of SomeIntProp is likely to be wrong as one thread will clobber the value of another.
On the other hand (as I found out the hard way) if you try to draw a Picture into a Canvas while threads are updating it, you will get a hard crash even if you are only accessing sections that are already complete.
But thatās the nature of preemptive threads.
1 Like