Updating progress indicators during heavy processing

  1. 2 weeks ago

    Hi
    I am developing a small Mac app that reads in a recordset (via an ODBC connection) then iterates through the recordset, exporting the data as a text file using a stream.

    The data is read in in about 1 second then the data is exported in a loop.

    The exporting loop has a couple of things that I am using to try to provide some feedback on progress. A progress bar and a label with the record number being exported.

    Neither is being updated, I just get the spinning wheel. Then when it is finished I just get the label updated to 'Done', which is my last line of code before exiting.

    Here is the loop.

    lblExportCount.Text = Str(RecordCount)
    
    // Loop through each row, one-by-one, and add it to the output file
    If data <> Nil Then
      
      RecordNumber = 1
      
      While Not data.EOF
        
        'progress bar
        prgExport.Value = (RecordNumber / RecordCount) * 100
        
        'status labels
        lblExportRecord.Text = Str(RecordNumber)
        lblExportRecord.Refresh
        
       'just formats individual files for the output file
        DataRow = FormatEBayUploadData(Data)
    
       'write to the file
        OutputStream.WriteLine(DataRow)
        
        RecordNumber = RecordNumber + 1
        data.MoveNext
      Wend
      data.Close
    End If
    
    'now close the output file for this market
    OutputStream.Close
    lblStatus.Text = "Done..."

    In the 'old days' with VB etc I would have used something like DoEvents to allow the form to re-paint in the loop, but I read that this is not good for Xojo.

    So how do I get the visual indicators to refresh whilst processing this loop?

    Thanks

    Mark

    You're looking at the problem linearly, and I don't blame you. It's likely how you were taught, it's how we were all taught, but this requires a new approach, so some clarification is in order.

    Xojo's Main Thread is an invisible event loop that does all the heavy lifting in your app. When you choose something from a menu or implement the Action event of a Pushbutton, that code all runs in the Main Thread.

    Any other Threads you create will run concurrently with the Main Thread. Once started, they go off and complete their task, then end. If you put an endless loop into a Thread, it will never end, but will never tie up the Main Thread either. Once started ("Run"), a Thread returns flow to the calling Thread almost immediately.

    On the other hand, any loop you put into the Main Thread will tie up the Main Thread, and thus most of your app, until the loop ends. If you create a loop that starts a Thread and then waits for it to finish before moving on, you've defeated the purpose of the Thread and your UI will still be locked up.

    This is linear (or as you put it, sequential) programming, but the way to attack this problem is through event-driven code. You don't keep checking if the Thread is finished to move on, you wait until the Thread tell you it's finished, then do the next thing.

    In this case, you want part of the UI to be responsive (progress bars), but not other parts (buttons, menus, etc.), but that's not how it works, it's either all or nothing. But you can still get the results you seek with a little extra effort.

    Here's how I'd handle it:

    • Create a Task subclass designed to do this export. It will contain all the information is needs to complete its task. That can include an array of files to export.
    • When the user clicks export, store this information in the Task's properties, disable the interface, then run the Task.
    • React to the Task's events to update your progress bar. When it tells you it's done, re-enable the interface.

    At no time do you create a loop to monitor the Task, you let it feed you the information instead.

    If you don't want to handle disabling the interface, move this function to a dialog window and have it do the processing in the way I described. When the Task is finished, it will close the dialog.

  2. Leonidas B

    Sep 4 Macaé, RJ, Brazil

    You can try @Brock N ´s July 22nd, 2015 approach on this thread. I adapted it to my needs and I'm quite happy with it.

    https://forum.xojo.com/24232-howto-wait-indicator-in-long-processes-without-timer-or-threads

  3. Hi
    Thanks for that, my bad, should have been clear, this is a Mac Desktop app, not a web app.

    Any ideas?

    Thanks
    Mark

  4. Beatrix W

    Sep 4 Pre-Release Testers, Third Party Store Europe (Germany)

    Try the Task example from the examples folder.

  5. brian f

    Sep 4 Pre-Release Testers, Xojo Pro Chilly California

    Yes as @Beatrix W Betrix suggests

    Look into the example projects that come with your Xojo install - Example Projects\Desktop\UpdatingUIFromThread

  6. Edited 2 weeks ago

    OK thanks I will give that a try...

    I will report back :)

  7. Hi

    OK I have looked at the examples. Using the Thread option (which is recommended) seems a real pain because the Run method, in which you have to place all your processing code, cannot take any parameters, so I can't pass in any values or flags for the process it needs to run.
    Only way I can see of doing it is having form level properties for all my variables and flags etc. Doesn't seem like a great solution to me.

    Am I missing something?

    Thanks

  8. Michel B

    Sep 4 Pre-Release Testers, Xojo Pro RubberViews.com

    Use properties of a module to pass parameters.

    Or a dictionary if you are familiar with that.

  9. Kem T

    Sep 4 Pre-Release Testers, Xojo Pro, XDC Speakers Connecticut

    Or subclass the thread with the properties you need, or create your own Run with the parameters you want.

  10. Thanks for the help

    "create your own Run with the parameters you want"

    Can you possibly explain a little on how I would do this please?

    Thanks
    Mark

  11. Hmmm, further development (excuse the pun)

    As I am stepping through my code which is the intensive bit that I have moved into the Task Run event, so that I can update the GUI items to show progress, the code suddenly exits from the Run event and back to the next line in the module that called Run. Is it time based? It is almost as if a timer is kicking in and dropping me out from the code as I am debug stepping through it. It happens at random points in the code as I step through, so not a code related issue.

    This is driving me nuts....

    This is the code for the calling routine

    If UploadData <> Nil Then
      
      
      'Pass control to the thread that does all the work so that the UI can be updated with progress etc
      uiTask.Run
      
      
      
    End If
    
    'now close the output file for this market
    OutputStream.Close
    lblStatus.Text = "Done AZUK"
    
    if UploadData <> Nil then
      UploadData.Close
    end if

    The code in the Run event steps through OK for a while then suddenly drops back to the line shown in the code above, 'OutputStream.Close, runs down to after UploadData.Close (that's the recordset), then drops back into the processing loop in Task run. But by then the UploadData recordset has been closed and so it cant continue. This is so weird...

    Thanks for any help
    Mark

  12. Hmmm I Think I have found a bug. I cant see any other explanation.

    In my 'heavy processing loop' that is within the Task Run event, one of the things it does is call a function that reads some data from the DB and sets some values based on the data values.

    Here is the code

    Dim Collectible as Boolean
    
    'get the fields needed to see if this record is collectible
    
    Dim sql As String
    sql = "SELECT ""az make all collectible"" AS MakeAll,  ""az make date collectible"" AS MakeDate, ""az make collectible over value"" As MakeValue, ""az pub date edit"" as PubDate, ""azon price uk"" As AzPrice FROM ""Abe Operating"" WHERE Serial = '" + Serial + "'"
    
    Dim Colldata As RecordSet
    Colldata = WBMBBDB.SQLSelect(sql)
    
    If WBMBBDB.Error Then
      MsgBox("DB Error: " + WBMBBDB.ErrorMessage)
    End If
    
    If Colldata <> Nil Then
      
      If Colldata.field("MakeAll").StringValue = "Yes" then
        Collectible = True
      elseif Val(Colldata.field("PubDate").StringValue) <= Val(Colldata.field("MakeDate").StringValue ) then
        Collectible = True
      elseif Colldata.field("AzPrice").StringValue >= Colldata.field("MakeValue").StringValue then
        Collectible = True
      else
        Collectible = False
      end if
      
    end if
    
    Return Collectible

    The Database object (WBMBBDB) is a Global object that I create when the app starts up, open the DB connection and use it throughout the app, save keep opening DB connections.

    As soon as the code i the above sample hits the line that assigns the recordset to the SQLSelect

    Colldata = WBMBBDB.SQLSelect(sql)

    The Run event exits back to the calling program, runs to the end if the block it is in (which closes the main recordset) then jumps back into the Run code. Which causes an error because the recordset has been closed.

    Surely this cant be right?

    Am I missing something?

    Thanks
    Mark

  13. Kimball L

    Sep 4 Pre-Release Testers, Xojo Pro Meridian, ID, USA

    I've read through your code and most recent question pretty quickly, so I may have missed something, but I think I know what is going on:

    You call run on your thread from somewhere.
    The thread starts doing its thing.
    Suddenly, the code right after the .run call starts executing, which closes your global DB connection.
    Then the thread suddenly jumps back to try to execute, but with the DB connection closed, bad things happen.

    Do I understand your issue correctly? If so, it is doing exactly what it is designed to do: the whole point of a thread is to allow code to execute more or less in parallel with other code - but the app has to swap back and forth between your thread and your main run loop to make this happen.

    This means that if you call .run from somewhere, you need to understand that the thread will start running, but the rest of the code after the .run call will *also* execute. You posted the code from your thread, but not the code from where you call .run on your thread.

    The Run event exits back to the calling program, runs to the end if the block it is in (which closes the main recordset) then jumps back into the Run code.

    It is important to drive home this distinction: The run event is *not* exiting - but rather the program has started time slicing between the thread and the main event loop, so in the debugger it will jump back and forth.

    TL: DR; Don't touch stuff that a thread needs in your main event loop until after the thread has finished running.

  14. Edited 2 weeks ago

    Hey Kimball

    Thanks for the reply. What you say sounds pretty much spot on about what is happening.

    What is amazing though is that no matter how long I wait while stepping through code (go and get coffee...etc) it ALWAYS drops back to the outside calling code when I hit the exact same line of code in the Thread Run event. If was some kind of time slicing thing surely it would be time based? Or is it down to how many lines of code it executes before returning to the outer code?

    Seems weird

    Whats more, this all seems like a sh** load of work just to get a progress bar to update. I can help thinking there should be an easier way.

    Thanks for the advice. If what you say is the only way forward, that means quite a restructure for my code, sigh...

    Cheers
    Mark

    PS, Another weird thing is that it always drops back to the outer code and runs the exact same number of lines of code, before jumping back to the thread code. Always, no matter how long I wait etc. It runs 5 lines of code then jumps back into the threaded code.

  15. Kem T

    Sep 4 Pre-Release Testers, Xojo Pro, XDC Speakers Connecticut

    Xojo will change context (i.e., switch to a different concurrent Thread or the Main Thread) at set points, and will do it on its own schedule. Those points are, among other things, loops, so it makes sense that you are seeing repeated behavior when you are stepping through the debugger.

    And this behavior may change in a future version, so you should not count on it either, but there is a drop-down menu in the debugger that will let you switch between active Threads so you can see where each is at that point.

    The bottom line is, be aware that a Thread will go off and do its own thing until it's done. You should have your Thread raise an event when it's done so you can clean up or notify the user, and raise events along the way so you can update the UI. That is what the Task class in the example is designed to help you do.

    The "right" way to code this is to set up everything the Thread needs, start the Thread, then react to the events the Thread raises.

  16. Hey Kem and Kimball
    Thanks for that, yes I see what is happening now. I must admit I am a little surprised at the relative complexity of achieving something as simple as updating a progress indicator. I understand the tools are there to do it but I would have thought that the framework would have been a little more supportive of such basic UI requirements without having to resort to timers and/or threads. I used to have to run timers to do this kind of thing in VB6. Ah well, c'est la vie. Xojo is a great tool in so many ways.

    Fortunately I am not too far into this project before the need to do this has come to light. I bit of re-structuring should see it done.

    Thanks
    Mark

  17. Michel B

    Sep 5 Pre-Release Testers, Xojo Pro RubberViews.com
    Edited 2 weeks ago

    There is another, sometimes simpler method for updating the UI.

    The UI cannot be updated while an event has not completed.

    If your code all takes place within the same event handler, you cannot update the progress bar.

    But if you break your code in several event handlers, then you can very simply address the progress bar.

    This can be achieved by placing a portion of the code into a timer Action event handler. Most of the time, the trick is to run code that would have been going into a tight loop into a zero milliseconds multiple timer.

    You then use such a structure

    Static i as integer// The counter
    
    If i < 200 // 200 iterations. You would do For i = 0 to 200 in regular code
        ' do stuff
       ' Update the progress bar
    else
       Sender.Mode = Timer.ModeOff
       Return
    end if
  18. Hey

    Thanks Michel, I think I get what you are saying. Just to run something by you.

    This is an overview of the current flow.

    1. User is presented with a list box of output file types (about 5 options). There is a checkbox in the listbox so they can select all of them or just a couple, free choice.

    2. There is an 'Export' button, which is the main action event and where all the intensive code is. (Actually in a method called by the buttons Action event)

    3. The main export method then loops through the rows in the listbox, running the relevant export routine for each of the selected rows.Each of the exports are fairly intensive and result in the spinning ball and no updates to the progress indicators on the form, when no threading is implemented.

    My first issue with the Task approach is that if I place the core part of the intensive export part in the Run event of a task, when the thread returns to the main export loop (at some random point in time), the main export loop then continues on to the next export selected, when in fact the first one has not actually completed, just dropped out of the Task Run Thread. By returning to the main loop and doing a bit of processing, bad things are going to happen as the next sequential export kicks off when the previous one is not complete. There is a lot of shared data and code for each of the various exports so each one needs to complete before the next is started.

    Bottom line is, I dont want the first thread to drop back to the main loop until it is done. All I want is for it to run through, exporting the data, updating the progress bar as it goes. Then when completed, come out, back to the main loop, go to the next selected export routine, and do the same thing.

    I do not want multiple exports of different kinds happening in a time slice fashion. All I want is to give the app enough breathing space to update a progress bar as it goes so the user can see it is progressing.

    I cant help thinking I must be missing something here as it seems such a basic requirement.

    Thanks
    Mark

  19. Sascha S

    Sep 5 Pre-Release Testers, Xojo Pro Germany, Lower Saxonary
    Edited 2 weeks ago

    Simple but effective:

    1. Let the Thread do it's thing in low Priority (3?).
    2. In this Thread Update a Progress Counter Property of the Window.
    3. Let the Thread start a Timer (Single Mode) in the Window from time to time.
    4. In this Timer, update the Visual Progress Indicator for the User, by using the Counter updated in the Thread in Step 3.
  20. Edited 2 weeks ago

    Hi Sascha
    Thanks for jumping in.
    I am not sure I fully understand what you are saying, sorry.

    Isn't the 'Task' class that is provided in the Xojo examples the combination of Thread/Timer you speak of anyway? That is what I am using. As I said above, when the thread doing the heavy work surrenders back to the main loop, the main loop then carries on and starts another export from the list, making bad things happen.

    Is there not some way of using the ability of the thread/timer Task class to update the UI (which it is doing very nicely) but not exit until it is done with the exporting work? I am not looking for a multi-tasking app here. I want a sequential process of exports, I just want the progress to be updated as it goes.

    Maybe what you describe is the answer, I am probably not understanding.

    Thanks
    Mark

  21. Newer ›

or Sign Up to reply!