Updating progress indicators during heavy processing

Hi
I am developing a small Mac app that reads in a recordset (via an ODBC connection) then iterates through the recordset, exporting the data as a text file using a stream.

The data is read in in about 1 second then the data is exported in a loop.

The exporting loop has a couple of things that I am using to try to provide some feedback on progress. A progress bar and a label with the record number being exported.

Neither is being updated, I just get the spinning wheel. Then when it is finished I just get the label updated to ‘Done’, which is my last line of code before exiting.

Here is the loop.

[code]lblExportCount.Text = Str(RecordCount)

// Loop through each row, one-by-one, and add it to the output file
If data <> Nil Then

RecordNumber = 1

While Not data.EOF

'progress bar
prgExport.Value = (RecordNumber / RecordCount) * 100

'status labels
lblExportRecord.Text = Str(RecordNumber)
lblExportRecord.Refresh

'just formats individual files for the output file
DataRow = FormatEBayUploadData(Data)

'write to the file
OutputStream.WriteLine(DataRow)

RecordNumber = RecordNumber + 1
data.MoveNext

Wend
data.Close
End If

'now close the output file for this market
OutputStream.Close
lblStatus.Text = “Done…”[/code]

In the ‘old days’ with VB etc I would have used something like DoEvents to allow the form to re-paint in the loop, but I read that this is not good for Xojo.

So how do I get the visual indicators to refresh whilst processing this loop?

Thanks

Mark

You can try @Brock Nash ´s July 22nd, 2015 approach on this thread. I adapted it to my needs and I’m quite happy with it.

https://forum.xojo.com/24232-howto-wait-indicator-in-long-processes-without-timer-or-threads

Hi
Thanks for that, my bad, should have been clear, this is a Mac Desktop app, not a web app.

Any ideas?

Thanks
Mark

Try the Task example from the examples folder.

Yes as @Beatrix Willius Betrix suggests

Look into the example projects that come with your Xojo install - Example Projects\Desktop\UpdatingUIFromThread

OK thanks I will give that a try…

I will report back :slight_smile:

Hi

OK I have looked at the examples. Using the Thread option (which is recommended) seems a real pain because the Run method, in which you have to place all your processing code, cannot take any parameters, so I can’t pass in any values or flags for the process it needs to run.
Only way I can see of doing it is having form level properties for all my variables and flags etc. Doesn’t seem like a great solution to me.

Am I missing something?

Thanks

Use properties of a module to pass parameters.

Or a dictionary if you are familiar with that.

Or subclass the thread with the properties you need, or create your own Run with the parameters you want.

Thanks for the help

“create your own Run with the parameters you want”

Can you possibly explain a little on how I would do this please?

Thanks
Mark

Hmmm, further development (excuse the pun)

As I am stepping through my code which is the intensive bit that I have moved into the Task Run event, so that I can update the GUI items to show progress, the code suddenly exits from the Run event and back to the next line in the module that called Run. Is it time based? It is almost as if a timer is kicking in and dropping me out from the code as I am debug stepping through it. It happens at random points in the code as I step through, so not a code related issue.

This is driving me nuts…

This is the code for the calling routine

[code]If UploadData <> Nil Then

'Pass control to the thread that does all the work so that the UI can be updated with progress etc
uiTask.Run

End If

'now close the output file for this market
OutputStream.Close
lblStatus.Text = “Done AZUK”

if UploadData <> Nil then
UploadData.Close
end if
[/code]

The code in the Run event steps through OK for a while then suddenly drops back to the line shown in the code above, 'OutputStream.Close, runs down to after UploadData.Close (that’s the recordset), then drops back into the processing loop in Task run. But by then the UploadData recordset has been closed and so it cant continue. This is so weird…

Thanks for any help
Mark

Hmmm I Think I have found a bug. I cant see any other explanation.

In my ‘heavy processing loop’ that is within the Task Run event, one of the things it does is call a function that reads some data from the DB and sets some values based on the data values.

Here is the code

[code]Dim Collectible as Boolean

'get the fields needed to see if this record is collectible

Dim sql As String
sql = “SELECT ““az make all collectible”” AS MakeAll, ““az make date collectible”” AS MakeDate, ““az make collectible over value”” As MakeValue, ““az pub date edit”” as PubDate, ““azon price uk”” As AzPrice FROM ““Abe Operating”” WHERE Serial = '” + Serial + “’”

Dim Colldata As RecordSet
Colldata = WBMBBDB.SQLSelect(sql)

If WBMBBDB.Error Then
MsgBox("DB Error: " + WBMBBDB.ErrorMessage)
End If

If Colldata <> Nil Then

If Colldata.field(“MakeAll”).StringValue = “Yes” then
Collectible = True
elseif Val(Colldata.field(“PubDate”).StringValue) <= Val(Colldata.field(“MakeDate”).StringValue ) then
Collectible = True
elseif Colldata.field(“AzPrice”).StringValue >= Colldata.field(“MakeValue”).StringValue then
Collectible = True
else
Collectible = False
end if

end if

Return Collectible
[/code]

The Database object (WBMBBDB) is a Global object that I create when the app starts up, open the DB connection and use it throughout the app, save keep opening DB connections.

As soon as the code i the above sample hits the line that assigns the recordset to the SQLSelect

Colldata = WBMBBDB.SQLSelect(sql)

The Run event exits back to the calling program, runs to the end if the block it is in (which closes the main recordset) then jumps back into the Run code. Which causes an error because the recordset has been closed.

Surely this cant be right?

Am I missing something?

Thanks
Mark

I’ve read through your code and most recent question pretty quickly, so I may have missed something, but I think I know what is going on:

You call run on your thread from somewhere.
The thread starts doing its thing.
Suddenly, the code right after the .run call starts executing, which closes your global DB connection.
Then the thread suddenly jumps back to try to execute, but with the DB connection closed, bad things happen.

Do I understand your issue correctly? If so, it is doing exactly what it is designed to do: the whole point of a thread is to allow code to execute more or less in parallel with other code - but the app has to swap back and forth between your thread and your main run loop to make this happen.

This means that if you call .run from somewhere, you need to understand that the thread will start running, but the rest of the code after the .run call will also execute. You posted the code from your thread, but not the code from where you call .run on your thread.

It is important to drive home this distinction: The run event is not exiting - but rather the program has started time slicing between the thread and the main event loop, so in the debugger it will jump back and forth.

TL: DR; Don’t touch stuff that a thread needs in your main event loop until after the thread has finished running.

Hey Kimball

Thanks for the reply. What you say sounds pretty much spot on about what is happening.

What is amazing though is that no matter how long I wait while stepping through code (go and get coffee…etc) it ALWAYS drops back to the outside calling code when I hit the exact same line of code in the Thread Run event. If was some kind of time slicing thing surely it would be time based? Or is it down to how many lines of code it executes before returning to the outer code?

Seems weird

Whats more, this all seems like a sh** load of work just to get a progress bar to update. I can help thinking there should be an easier way.

Thanks for the advice. If what you say is the only way forward, that means quite a restructure for my code, sigh…

Cheers
Mark

PS, Another weird thing is that it always drops back to the outer code and runs the exact same number of lines of code, before jumping back to the thread code. Always, no matter how long I wait etc. It runs 5 lines of code then jumps back into the threaded code.

Xojo will change context (i.e., switch to a different concurrent Thread or the Main Thread) at set points, and will do it on its own schedule. Those points are, among other things, loops, so it makes sense that you are seeing repeated behavior when you are stepping through the debugger.

And this behavior may change in a future version, so you should not count on it either, but there is a drop-down menu in the debugger that will let you switch between active Threads so you can see where each is at that point.

The bottom line is, be aware that a Thread will go off and do its own thing until it’s done. You should have your Thread raise an event when it’s done so you can clean up or notify the user, and raise events along the way so you can update the UI. That is what the Task class in the example is designed to help you do.

The “right” way to code this is to set up everything the Thread needs, start the Thread, then react to the events the Thread raises.

Hey Kem and Kimball
Thanks for that, yes I see what is happening now. I must admit I am a little surprised at the relative complexity of achieving something as simple as updating a progress indicator. I understand the tools are there to do it but I would have thought that the framework would have been a little more supportive of such basic UI requirements without having to resort to timers and/or threads. I used to have to run timers to do this kind of thing in VB6. Ah well, c’est la vie. Xojo is a great tool in so many ways.

Fortunately I am not too far into this project before the need to do this has come to light. I bit of re-structuring should see it done.

Thanks
Mark

There is another, sometimes simpler method for updating the UI.

The UI cannot be updated while an event has not completed.

If your code all takes place within the same event handler, you cannot update the progress bar.

But if you break your code in several event handlers, then you can very simply address the progress bar.

This can be achieved by placing a portion of the code into a timer Action event handler. Most of the time, the trick is to run code that would have been going into a tight loop into a zero milliseconds multiple timer.

You then use such a structure

[code]Static i as integer// The counter

If i < 200 // 200 iterations. You would do For i = 0 to 200 in regular code
’ do stuff
’ Update the progress bar
else
Sender.Mode = Timer.ModeOff
Return
end if[/code]

Hey

Thanks Michel, I think I get what you are saying. Just to run something by you.

This is an overview of the current flow.

  1. User is presented with a list box of output file types (about 5 options). There is a checkbox in the listbox so they can select all of them or just a couple, free choice.

  2. There is an ‘Export’ button, which is the main action event and where all the intensive code is. (Actually in a method called by the buttons Action event)

  3. The main export method then loops through the rows in the listbox, running the relevant export routine for each of the selected rows.Each of the exports are fairly intensive and result in the spinning ball and no updates to the progress indicators on the form, when no threading is implemented.

My first issue with the Task approach is that if I place the core part of the intensive export part in the Run event of a task, when the thread returns to the main export loop (at some random point in time), the main export loop then continues on to the next export selected, when in fact the first one has not actually completed, just dropped out of the Task Run Thread. By returning to the main loop and doing a bit of processing, bad things are going to happen as the next sequential export kicks off when the previous one is not complete. There is a lot of shared data and code for each of the various exports so each one needs to complete before the next is started.

Bottom line is, I dont want the first thread to drop back to the main loop until it is done. All I want is for it to run through, exporting the data, updating the progress bar as it goes. Then when completed, come out, back to the main loop, go to the next selected export routine, and do the same thing.

I do not want multiple exports of different kinds happening in a time slice fashion. All I want is to give the app enough breathing space to update a progress bar as it goes so the user can see it is progressing.

I cant help thinking I must be missing something here as it seems such a basic requirement.

Thanks
Mark

Simple but effective:

  1. Let the Thread do it’s thing in low Priority (3?).
  2. In this Thread Update a Progress Counter Property of the Window.
  3. Let the Thread start a Timer (Single Mode) in the Window from time to time.
  4. In this Timer, update the Visual Progress Indicator for the User, by using the Counter updated in the Thread in Step 3.

Hi Sascha
Thanks for jumping in.
I am not sure I fully understand what you are saying, sorry.

Isn’t the ‘Task’ class that is provided in the Xojo examples the combination of Thread/Timer you speak of anyway? That is what I am using. As I said above, when the thread doing the heavy work surrenders back to the main loop, the main loop then carries on and starts another export from the list, making bad things happen.

Is there not some way of using the ability of the thread/timer Task class to update the UI (which it is doing very nicely) but not exit until it is done with the exporting work? I am not looking for a multi-tasking app here. I want a sequential process of exports, I just want the progress to be updated as it goes.

Maybe what you describe is the answer, I am probably not understanding.

Thanks
Mark