If you wanted to use a thread that would loop through a string array list of items and then process each one by launching another thread that does that processing and wait for that thread to complete before launching that processing thread again with the next item in the list.
The rationale for doing it this way is if I do it all in a single thread the processing slows to a crawl and eventually crashes if the list is long. I have been unable to resolve the cause of the slow down so it makes sense to me to launch the thread to do the processing for each individual item in a separate instance serially because of sending the processed data to an external system requires a pause to allow the remote system to ingest and use the data sent so the thread needs to sleep for 10 seconds before ending and somehow notifying the looping thread to start the next item.
My efforts so far have been unreliable at in the waiting part of the 1st tread for 2nd thread completion. I’m looking for ideas of how you would tackle this scenario. It doesn’t have to use two threads, that is just my current approach.
Can you say more about the algorithm you are using? My first approach would be to try to speed that up as much as possible. I would worry about threading and other issues later. Can you describe the algorithm in words, or post some sample code?
All Xojo threads run on the same single CPU core along with your non-threaded code.
If a single thread slows to a crawl then I don’t think having multiple Xojo threads will make this any better as they will just be stealing the CPU from each other.
Unfortunately, Xojo doesn’t support (or want to support) real multi-processing so you may have to take a different approach.
I suggest you first optimise your algorithm as much as possible. If that still isn’t good enough then you may need to put the algorithm into a separate console app which your web app can execute with the relevant data. You could then have a single thread in your web app which executes and tracks the progress of multiple console apps to provide better throughput (this is basically, what the Xojo workers feature does).
Public Property events() As String
Sub Opening() Handles Opening
Thread1.Start
Do Until Thread1.ThreadState = Thread.ThreadStates.NotRunning
thread.SleepCurrent(10)
Loop
events.Add "Quit"
For each ev As String in events
System.DebugLog ev
Next
app.DoEvents
Quit
End Sub
Sub Thread1.Run() Handles Run
events.Add "Started 1"
Thread2.Start
Do Until Thread2.ThreadState = Thread.ThreadStates.NotRunning
me.Sleep(40)
Loop
events.Add "Ended 1"
End Sub
Sub Thread2.Run() Handles Run
events.Add "Started 2"
For i As Integer = 1 to 100
me.Sleep(30)
Next
events.Add "Ended 2"
End Sub
I tried this yesterday and it would never resume when pause was called from within a While/Wend in the 1st thread that was within an enclosing While/Wend. However I just tried it again changing the inner While/Wend to an If/Then and suddenly it worked, and the timing for each execution was what I was looking for.
// process the jobs in the JobList
While JobList.LastIndex > -1
Logstring = "Processing Job " + JobList(0)
LogTransactions(Logstring)
Me.Sleep(200)
// ProcessingJobXJDF must be True until set False by the ProcessingXJDF thread
If ProcessingJobXJDF = False Then
ProcessingJobXJDF = True
App.ProcessXJDF = New ProcessXJDF_Thread
App.ProcessXJDF.JobToProcess = JobList(0)
App.ProcessXJDF.JobType = Me.JobType
App.ProcessXJDF.Start
Self.Pause
End If
JobList.RemoveAt(0)
Wend
MoveToDone(FileToProcess)
Logstring = "PA Prinect data processing has completed. Resuming hot folder peridic polling." _
+ EndOfLine + "—————————————————————————————————————————————————————————————————————————————————————"
LogTransactions(Logstring)
This is the output I am seeing
2024-03-14 12:56:34 Processing Job 038558-24
2024-03-14 12:56:34 Get Spec Info Data Returned Successfully for Job 038558-24
2024-03-14 12:56:35 Successfully saved the XJDF file: \\clksujpisl03.jostens.com\workflow\Prinect\XJDF\038558-24_2024314125635.xjdf
2024-03-14 12:56:35 Send POST for Job 038558-24 sent successfully
2024-03-14 12:56:36 XJDF file successfully created for Job 038558-24
2024-03-14 12:56:36 SendPrinectSubmitQueueEntry Success! The XJDF update for Job 038558-24 was successfully submitted to Prinect
—————————————————————————————————————————————————————————————————————————————————————
2024-03-14 12:56:36 Pausing 5 seconds for Prinect processing
—————————————————————————————————————————————————————————————————————————————————————
2024-03-14 12:56:41 Processing Job 015945-24
2024-03-14 12:56:41 Get Spec Info Data Returned Successfully for Job 015945-24
2024-03-14 12:56:41 Successfully saved the XJDF file: \\clksujpisl03.jostens.com\workflow\Prinect\XJDF\015945-24_2024314125641.xjdf
2024-03-14 12:56:43 Send POST for Job 015945-24 sent successfully
2024-03-14 12:56:43 XJDF file successfully created for Job 015945-24
2024-03-14 12:56:43 SendPrinectSubmitQueueEntry Success! The XJDF update for Job 015945-24 was successfully submitted to Prinect
—————————————————————————————————————————————————————————————————————————————————————
2024-03-14 12:56:43 Pausing 5 seconds for Prinect processing
—————————————————————————————————————————————————————————————————————————————————————
2024-03-14 12:56:48 Processing Job 012652-24
2024-03-14 12:56:48 Get Spec Info Data Returned Successfully for Job 012652-24
2024-03-14 12:56:49 Successfully saved the XJDF file: \\clksujpisl03.jostens.com\workflow\Prinect\XJDF\012652-24_2024314125649.xjdf
2024-03-14 12:56:49 Send POST for Job 012652-24 sent successfully
2024-03-14 12:56:50 XJDF file successfully created for Job 012652-24
2024-03-14 12:56:50 SendPrinectSubmitQueueEntry Success! The XJDF update for Job 012652-24 was successfully submitted to Prinect
—————————————————————————————————————————————————————————————————————————————————————
2024-03-14 12:56:50 Pausing 5 seconds for Prinect processing
—————————————————————————————————————————————————————————————————————————————————————
However, I now occasionally am seeing the following when processing supposedly begins on a new file it appears to skip through a list of job numbers before it actually processes one.
2024-03-14 12:56:26 Checking Hot Folder for new files to Process
2024-03-14 12:56:26 JobType = YB file name = PA_JOB_CHANGES_2024-03-11T203436-05.txt
2024-03-14 12:56:26 2631 job names are contained in PA_JOB_CHANGES_2024-03-11T203436-05.txt. Beginning processing XJDF updates to Prinect.
—————————————————————————————————————————————————————————————————————————————————————
2024-03-14 12:56:26 Processing Job 041096-24
2024-03-14 12:56:26 Processing Job 047903-24
2024-03-14 12:56:27 Processing Job 013036-24
2024-03-14 12:56:27 Processing Job 003126-24
2024-03-14 12:56:27 Processing Job 034242-24
2024-03-14 12:56:27 Processing Job 004142-24
2024-03-14 12:56:27 Processing Job 006127-24
2024-03-14 12:56:28 Get Spec Info Data Returned Successfully for Job 006127-24
2024-03-14 12:56:28 Successfully saved the XJDF file: \\clksujpisl03.jostens.com\workflow\Prinect\XJDF\006127-24_2024314125628.xjdf
2024-03-14 12:56:29 Send POST for Job 006127-24 sent successfully
2024-03-14 12:56:29 XJDF file successfully created for Job 006127-24
2024-03-14 12:56:29 SendPrinectSubmitQueueEntry Success! The XJDF update for Job 006127-24 was successfully submitted to Prinect
I haven’t debugged why yet. That may be due to an error quietly occurring that I am unaware of but I am thankful that your suggestion worked with a slightly different refactoring. of the existing code.
Yes I know and in apps where multithreading is truly important I usually shell out to a console app but in this case multithreading isn’t the goal. The goal here is regular updates to the logging in the Web UI by yielding time within a thread.
The Processing thread that does all the heavy lifting run event and the primary method that calls all the other methods and classes is shown below. I also included a screen shot. I eliminated using the UserInterfaceUpdate and instead log to a String variable and a timer updates the UI with whatever is in that String variable every second.
Sub Run() Handles Run
Var Logstring As String
Var b As Boolean
// process the JobToProcess
b = ProcessJob(JobToProcess,JobType)
If b = False Then
// There was a problem processing the Job. It will added to the aborted jobs list
Logstring = "There was a problem processing the Job " + JobToProcess + "Futher processing has been aborted and Support has been notified" _
+ EndOfLine + "—————————————————————————————————————————————————————————————————————————————————————"
LogTransactions(Logstring)
// Send a Notification Email: SendEmail(msg As String, subj As String, msgtype As string, sendtoadd As String)
Logstring = "There was a problem processing the Job " + JobToProcess + "Futher processing has been aborted."
SendEmail(LogString,App.kAppShortName + " " + JobToProcess + " Failed","text",SendToEmailAddress)
End If
Logstring = "Pausing 5 seconds for Prinect processing" _
+ EndOfLine + "—————————————————————————————————————————————————————————————————————————————————————"
LogTransactions(Logstring)
Me.Sleep(5000,False)
ProcessingJobXJDF = False
App.HotFolderTask.Resume
Me.Sleep(1000,False)
End Sub
Below is the Method directing
Public Function ProcessJob(jn As String, jt As String) As Boolean
Var LogString As String
Var jobparts(-1) As String
Var jinfo As String
Var b As Boolean
Var JobType As String
Select Case jt
Case "COM"
JobType = "Commercial"
Case "YB"
JobType = "Yearbook"
End Select
ResetSpecInfoVariables
JobName = jn
jobparts.ResizeTo(-1)
jobparts = JobName.Split("-")
jobnum = jobparts(0)
jobyr = jobparts(1)
// Construct leading zeros to add to job number so it is 6 digits in XJDF
JobNumber = jobnum
JobName = JobNumber + "-" + jobyr
JobYear = "20" + jobyr
jinfo = ""
// Query the GetSpecInfo service for job specific specifications
jinfo = GetSpecInfoService(JobNumber,JobYear)
If jinfo = "" Then
//GetSpecInfo did not return any data
LogString = "No Get Spec Info Data was returned for Job " + JobName
LogTransactions(Logstring)
Me.Sleep(200)
Return False
Else
If GetSpecInfoParseXML(jinfo) = True Then
LogString = "Get Spec Info Data Returned Successfully for Job " + JobName
LogTransactions(Logstring)
Me.Sleep(200)
// Check to make sure that GetSpecInfo did not return an empty string
b = PrinectCreateJobXJDF(jt)
If b = True Then
b = SendPrinectSubmitQueueEntry
If b = True Then
LogString = "XJDF file successfully created for Job " + JobName
LogTransactions(Logstring)
LogString = "SendPrinectSubmitQueueEntry Success! The XJDF update for Job " + JobName + " was successfully submitted to Prinect" _
+ EndOfLine + "—————————————————————————————————————————————————————————————————————————————————————"
LogTransactions(Logstring)
Me.Sleep(200)
Return True
Else
LogString = "SendPrinectSubmitQueueEntry Failure! The XJDF update for Job " + JobName + " Failed Submission to Prinect" _
+ EndOfLine + "—————————————————————————————————————————————————————————————————————————————————————"
LogTransactions(Logstring)
Me.Sleep(200)
Return False
End If
Else
// Send an error email
LogString = "XJDF file creation failed for Job " + JobName _
+ EndOfLine + "—————————————————————————————————————————————————————————————————————————————————————"
// Log the Exception
LogExceptions(Logstring)
LogTransactions(Logstring)
Me.Sleep(200)
Return False
End If
Else
LogString = "GetSpecInfoService failed for " + JobType + " Job " + JobName + " and job processing will be aborted for this job" _
+ EndOfLine + "—————————————————————————————————————————————————————————————————————————————————————"
LogExceptions(Logstring)
LogTransactions(Logstring)
Me.Sleep(200)
Return False
End If
End If
End Function
Well, good. I often launch a number of threads simultaneously, and observe that once one gets going, it often runs to absolute completion, even though it’s organised to use what data it has, then Pause, and wait for a DataAvailable event to Resume it (I do all processing in the thread, none in the socket’s event handlers). This is probably because data is arriving faster than it can be consumed.
This doesn’t matter particularly but I may try adding a Thread.YieldToNext() judiciously to see if that gives other threads a look in.
Seems to me that means your log would be incomplete, potentially, but perhaps that doesn’t matter. I log to a file to be sure I get it all.
I log to a file first then store the log line(s) in the variable I mentioned earlier that is for UI updates. I found a timer issue which was causing the skipping of job numbers and now the app is behaving as envisioned. Thanks for your help.
