My naughty WebThreads only work properly when being watched (with a Break in code but not without).

Here is a Web app that provides realtime processing and monitoring of data. The idea is for a changing series of data locations to be scanned and any data found sucked up, processed sorted and made available as one orderly queue.

I have a Container with a button action which calls a looping WebThread subclass (sleep set to 700ms) to monitor a property in a data acquisition and processing class. The looping WebThread changes a style in a Rectangle in the Container, so I know this works fine because it blinks - yellow when data is received, alternating between green and grey when the data class is checked but no data was ready. So far, so good…

FYI: The data processing class being monitored is instantiated first by the button action via another launching class which appends the data processing class to a dictionary, returning a reference to the button action that is assigned to a variant. The variant is then assigned to a property of of the Container to increase the monitored class’s scope beyond the button action event. After this the previously-mentioned looping WebThread subclass is then instantiated and also assigned to a property of the Container, and a Handler is added which includes a WebSessionContext for good measure.

The monitored data processing/acquisition class itself has three threads. The first checks an array of file locations to see if there is any data and if so, fetches the data and adds it to a dictionary. Critical sections control this operation on the Dictionary. Another thread processes any data in the dictionary, removes it and adds processed output to an array used as a stack, which processing is also all controlled by critical sections. Another thread controls the array of file locations which is also critically sectioned. The stack array of the data processing/acquisition class is monitored by the looping WebThread class for a Count (ubound+1) of > -0. If the count is greater the top item is output to a WebListbox in the WebContainer. A “RemoveTop Item” method is then called in the monitored class until the count reaches zero. All the threads have sleeps to allow the others to run.

Now here’s the thing: When the button action is pressed the monitoring starts and the first found file data is picked up, processed and presented in the WebListBox no problem. But none after that. However, if I put a break in either the input or processing threads of the data acquisition/processing class, subsequent updates to the WebList box work fine. Take away the Break and there’s nothing more. Seems my code’s logic is good but it naughtily only works after the first time if it’s being watched !

Any suggestions?

SUMMARY:
Multiple WebThreads accessing common data structures using critical sections worked when the code had a Break in it (thus triggering the IDE) but only executed one time before silently failing to “fire” under normal running Conditions. This was despite the WebTreads being put to sleep to provide ample time for the others to execute. The solution was to force a delay of WebThread code execution until extra DoEvents could be inserted and executed in the Web App’s main event loop, to allow common data structures time to update… Warning: putting App.DoEvents(100) at the top of a WebThread’s run command did not work and a Web App does not provide a Run event to put in extra DoEvents. However, the following code did work:

Dim ThisWebApp As New WebApplication = App ThisWebApp.DoEvents(100)

DETAILS:
Often this type of problem may be avoided using “helper” console apps and running them using the Shell Command. However in the case presented in the previous post this could have introduced a large number of sub-second cross-core context switches. Further, helper apps could only delay the inevitable threading required to combine and process multiple data streams. Therefore the following steps were taken to isolate the problem which was found to be in the way we used cooperative WebThreading in relation to sharing data between threads:

  • Since our code worked when the Break command was inserted to trigger the Linux IDE, we believed the issue was not OS related. Indeed, Xojo web Apps usually run very well on Linux.
  • The issue remained using both 64 and 32 bit execution, so it probably wasn’t caused by a compiler.
  • The issue persisted when the project was recompiled using Xojo 2017r1 instead of 2018r3 so it wasn’t caused by a recent change.
  • The possibility of a session-context problem was eliminated by refactoring the data class into an array of objects stored in a module property for direct access by Web sessions (i.e. not through a chain of session-context sensitive references). This reproduced the same problem.

The only remaining difference which could be observed between the Breaked working code and normal running yet non-functioning code was that debug functions supporting introspection of threads and of the App object etc must have returned program flow to the main thread during the Break, allowing inter-thread data structures to catch up with the news from various WebThreads. So we somehow needed to give Xojo’s threading a break during normal running.

The language reference states sufficient DoEvents in the main event loop (of a Console App) are required for threads and timers to work and Xojo’s finely-tuned Web treading had obviously no knowledge of the needs of our interloping threaded code. By creating a local object variable referencing the Web App within the WebThread, the code following in that Webthread must wait until the DoEvents on the main thread have completed (according to our theory anyway); While sleeping the thread at this same point for the same length of time did not have the desired effect.

CONCLUSION:
While the cooperative threading model can be simpler and more efficient when working with common data structures, the amount of CPU cycles given to the Web App’s main thread must also be considered if the Webthreads are numerous and doing a fair amount of work. Sleeping multiple WebThreads in a Web App does not guarantee that upon waking them, previously executing WebThreads will have updated common data structures - this may require extra DoEvents executed in the main event loop to achieve. Since the main event loop is not exposed in Xojo Web, the DoEvents method on the App object must be employed. However to ensure this happens before the rest of the WebTread’s code runs, the App object should be assigned to a local variable and executed that way rather than intrinsic operation (we think that’s why). Early indications are that fine-tuning of judicious allocations of Sleeps and App-object variable DoEvents will be required to allow enough breathing space for the proper execution of Xojo’s cooperative threading in a Web App.

While the above account is only based on our external observations, the result seems to work.