Process Execution Question

Hey all,

I’ve got a question on how Xojo processes method calls and so forth when called in a rapid fashion.

I have a situation where I am reading data from an analog control over USB. This analog control provides an output value in the range of 0 to 255 and the values can change basically as fast as the user moves the control. To put it in one real world example, this is an external “slider” that will be used to control the volume of say a TV or some other device.

I have a class that reads the data from from USB via a 1 millisecond timer. This works very well. I get all 256 data points from the control (occasionally a couple get missed but that’s not a huge deal as the timer may not fire exactly every 1 mS). I’m using MBS classes for both the timer and the USB. And when I am talking about moving the control, I’m talking about fairly rapidly, like moving through its range in half a second or less.

Then this is fed into another class that I’ve created that processes the data. I can verify that I am seeing all values of the control over it’s range in the first method. Then that method calls an additional method (let’s call this second method “SendCommand”) that does some further processing with the data and then creates a command and ultimately, this command and data get sent over TCP/IP to some other external device.

If I check at the beginning of this SendCommand method for how many times the method gets called, I get 256 over the range of the control. However, checking at the END of the method for how many times the method runs, I get a value of 98 times! I am not sure why I am getting 256 “starts” to the method but only 98 “finishes.” There’s only one return call before the end of the method, but it’s not being executed as it is a return only on invalid data. I’ve added a breakpoint there and that never gets called. So it’s not like the code is returning prior to the end of the method.

So why would the method only completely run less than half the time? What is Xojo’s process execution for methods? I would think each call would be added to the stack individually so that if you get a second call to the method before the first one is finished, that it wouldn’t interrupt the method, but that does not appear to be the case. It appears that my calls are being interrupted and that I am not processing all the data that I need to process. It’s not a failure in the TCP socket. I have the data being sent to that all buffered so that it gets queued if the socket is in the middle of a write and new data comes in. That’s working well. It’s somewhere else in the framework that my method calls are being interrupted and not finishing.

Is this expected behavior? I’m running everything in the main thread. I could go threaded but in general, it’s not like reading the data and writing it out to a socket takes a long time. It’s just that I have a bunch of it that can be sent in a rapid manner. Think of the worst case situation where a person goes back and forth on this slider control over and over and over again. Lots of data being captured and rapidly.

Anyone have any thoughts on the best way to handle this? I’m thinking that every time I get data, that I put the resultant value into an array and then start a process via a timer, that then loops through each item in the array and sends it on down the line. This seems like a potential way to separate the two events (incoming data vs. outgoing data) and adds a buffer between the two. Not sure If that’s the best way to handle it but I am open to suggestions…

Thanks,

Jon

Can you share some code as either Xojo is very broken or there is something wrong in your code.

I could. There’s several different method calls that get made. Let me copy and paste some code in here for some of them and add some extra comments, clean stuff up, etc.

So if I hear you correctly, multiple rapid calls to a method should end up all being added to the stack with each “train” (for lack of a better term) of calls running to completion before the next one starts. That’s what I thought too…

Yes. In any Thread, including the Main Thread, it’s all sequential. It sounds to me like your code isn’t doing what you think it’s doing.

One way to test that is to add System.DebugLog calls liberally throughout the calls. (Don’t leave them there for production because they are “expensive”.) You can then use the log to narrow down the flow.

1 Like

That’s what I am doing and how I discovered this…

Is there one or more loops in that method?
If so, try:

#Pragma BackgroundTasks False

It could be that you are re-calling the same method multiple times in itself (stack building) but the stack is not overflowing…

Exceptions raised can cause methods to exit before they complete. Is it possible that an exception is being raised, but caught and handled outside this method?

Yes, and have never seen the problem you have described.
What you could try is putting each piece of data you receive into an array and use a thread to process the array.

Or even Workers if it’s a desktop app.

Here’s the entire code chain with a couple small exceptions (like one that converts hex notation like 0x0A to ASCII values). The rest is all there. This is long, but if something is not right, I’d love to know what.

Here is some code from the first method in question. Several sections of it can be ignored as this code was based off some other code and I’m not using some of those sections right now. If you have questions ask.

Public Sub SendCommand(CommandToSend as String)
' Using BarHits3 to count how many times we enter the method
  BarHits3 = BarHits3+1
   
' The is a device on the network that this software controls.  In this particular example
' it is not being used right now but is in here for completeness.
  Dim j as JPDevice  
  If UseL3 Then
    If me.SendToTX Then
      j = SourceDeviceArray(SourceID)
    Else
      j = Screen
    End If
  End If
   
  If Me.DevicePort = 6752 And Not Me.SendToAV Then 
       ' This particular branch is not executing at this time either. 
    Try
      AddHandler j.Connected6752, AddressOf JAPDevConnected6752
    Catch
    End Try
    Try
      AddHandler j.Closing6752, WeakAddressOf CloseJAPDevRS232
    Catch
    End Try
    If UseL3 Then
      SendJAPDevRS232(j, CommandToSend)
    End If
    
  Else  ' Here is where we start executing
    ' First we see if the device we are going to send this command to is using an HTTP command.
    ' AVDeviceDictionary is a Dictionary stored in a module that contains the AV (Audio-Visual) devices
    ' That we connect to and is "keyed" based on the commands we are going to send.
' Again - this branch can be ignored as I am not sending HTTP commands at this time.

    If CommandToSend.Lowercase.IndexOf("http://") > -1 Then
      
      IF AVDeviceDictionary.HasKey(CommandToSend) Then
        Dim OrigCommandCode As String = CommandToSend
        Dim a as AVDevice
        a = AVDeviceDictionary.Value(CommandToSend)
        
        // Check for request header entered into our command code.  Format is as shown below:
        //   http://172.16.100.70 RequestHeader(Cookie:acid=41b7; brlang=0; productID=VNAX01) DELAY500 http://172.16.100.70/api/rec
        
        Dim rx As New RegEx
        rx.SearchPattern = "(?mi-Us)RequestHeader\((.+):(.+)\)\s"
        Dim rxOptions As RegExOptions = rx.Options
        rxOptions.LineEndType = 4
        
        Dim match as RegExMatch = rx.Search(CommandToSend)
        If match <> Nil Then
          Dim p As Pair = match.SubExpressionString(1) : match.SubExpressionString(2)
          a.URLConRequestHeader = p
          CommandToSend = CommandToSend.Replace(match.SubExpressionString(0),"")
        End If
        
        a.AutoWrite(CommandToSend)
      End If
      
    Else
      // OK.  Here we go.  This is where my code executes.  This is TCP/IP based commands.
// An AVDevice is a custom class that contains methods and information about devices I connect to.
      Dim a As AVDevice

      // TheAVDevice is a property of the class.  If it is NIL we look it up in the dictionary. 
      If TheAVDevice = Nil Then
        Dim mKey As String = DeviceIP+" "+Str(DevicePort)
        If AVDeviceDictionary.HasKey(mKey) Then
          a = AVDeviceDictionary.Value(mKey)
        Else
          Return // Nothing else to do   This only happens if the dictionary does not have this device. 
        End If
      Else  // The property isn't NIL so we assign it.
        a = TheAVDevice
      End If
      
      If a IsA TwoWayAVDevice Then  // In this particular case, this is always going to be false.
        TwoWayAVDevice(a).SetButtonStatusHandler(AddressOf AVDeviceUpdateSliderHandler)
      End If
      
      a.MaintainConnection = True
// The below command "TimerWrite" is misleading.  It's not writing via a timer.  It used to but not any 
// more.  Just never changed the method name.  This "TimerWrite" method writes to a TCP socket.
      a.TimerWrite(CommandToSend.ConvertHexStringToAscii)

    End If
  End If

// This is the variable that is NOT adding up to the same as BarHits3 above.  
  BarHits4 = BarHits4+1      
End Sub

OK. Now, here is the code from the “TimerWrite” method:

    Public Sub TimerWrite(a as Auto)  
    //Wrote this when Xojo framework was the rage.  And this was originally part of a Timer.CallLater call
    // So I used an Auto type.  No need to now.
      Dim s As String = a
      
    // First is processing the string to handle some special control commands I may have in there.
    // These are based on a protocol I developed.
    // In this case I do not have anything where I am using these.  So this will be skipped.
    // Basically it allows a user to enter something like a phone number which gets dialed on a VOIP system
    //
    // The rest is all for buffering commands during a TCPSocket negotiation where certain control characters need to be sent.  Or for other cases of holding up commands and buffering them.
// Not using them in this case.
      If s.InStr("\\") <> 0 Then
        
        Dim rx As New RegEx
        rx.SearchPattern = "(?mi-Us)\\\\W(.*)\\\\"
        
        Dim rxOptions As RegExOptions = rx.Options
        rxOptions.LineEndType = 4
        
        Dim match As RegExMatch = rx.Search( s )
        
        If match <> Nil Then
          Dim w As New UserDialog
          Dim sval As String = w.RunDialog(match.SubExpressionString(1))
          
          If sval = "-1Cancelled-1" Then
            Return
          End If
          
          rx.ReplacementPattern = sval
          rxOptions.ReplaceAllMatches = True
          Dim replacedText As String = rx.Replace( s )
          s = replacedText
        Else
          
          If s = "\\b-" Then  //Remove last entry from the buffer
            BufferCommands = BufferCommands.Left(BufferCommands.Len-1)
            DisplayNotifications(BufferCommands)
            Return
          End If
          
          If s.Len >= 3 Then
            // Check to see if it is a buffered command - meaning we want to wait for a whole series of items before sending the full command to the device.
            Dim bufferTest As String = s.Right(3)
            
            // If the last 3 letters = "\\b" then we have a buffered command.  Add it to the buffer and return - we are done for now.
            If bufferTest.Lowercase = "\\b" Then
              BufferCommands = BufferCommands+s.Left(s.Len-3)
              DisplayNotifications(BufferCommands)
              Return
            End If
          End If
          
          If s.InStr("\\b/\") <> 0 Then
            // New command to clear the buffer but not write it out.
            BufferCommands = ""
            Return
          End If
          
          If s.InStr("\\b\\") <> 0 Then
            s = s.ReplaceAll("\\b\\",BufferCommands)
            // Clear out the buffer. - Need to reset it since we've written commands.
            BufferCommands = ""
          End If
          
        End If
      End If
     
    // Here we call our write method. 
      Write(s)
    End Sub

Not sure why the formatting above thinks I missed a quote because I didn’t. OK. now the write method…

Public Sub Write(mytext As string)

  If mytext.Lowercase.InStr("http://") = 0 Then
    
    System.DebugLog("Writing from AVDevice "+m.ToString)
    Dim T as Integer = Ticks
    TCPSock.Address = me.IPAddress
    TCPSock.Port = me.TCPPort
    // This is a custom TCPSocket that will Queue any data sent to it in an array until the socket is 
// connected.  Then when it connects, it pulls each item out of the array.  Additionally, if the socket is
// is in the middle of a write, it will append the data to the array until the send complete is done  Then
// it pulls the next item off the array and so forth.
//
// If there is no sending activity, then the socket just writes out as usual.

    TCPSock.WriteWithQueue(mytext)
    TCPSock.Poll
    
    If TCPSock = Nil Then Break
    
  Else
    Try
      Call HTTPSock.SendSync("GET",mytext,5)
    Catch e As RuntimeException
      If e.Message = "The Internet connection appears to be offline." Then
        MsgBox "The Network Connection appears to be down.  Please make sure you are connected to your network and then try again."
      Elseif e.Message = "A request is already in progress." Then
        MsgBox "The previous command has been sent to the device.  Please wait a little bit before sending another command."
      End If
    End Try
  End If
End Sub

Now here’s code from the TCPSocket. This all seems to be working fine. It queues the data just fine.

Public Sub WriteWithQueue(mtext as String)

// SockConnected is an computed property that uses the socket’s IsConnected property as well as
// Another property to make sure we are actually connected. TCP Sockets can think they are connected
// When they are not. So we do some other checks. This is not Germaine to this…

  If SockConnected Then
    If Sending Then  // We are in the middle of sending data on the socket.  Queue it
      WriteQueue.Append mtext
      System.DebugLog "Sending = True  - Queuing From WritewithQueue - WriteQueue size = "+str(WriteQueue.Ubound+1)
    Else
      System.DebugLog "Sending = False - Writing data- WriteQueue size = "+str(WriteQueue.Ubound+1)
      write(mtext)
    End If
  Else
    System.DebugLog("Not Connected - Queing & connecting - WriteQueue size = "+str(WriteQueue.Ubound+1))
    WriteQueue.Append mtext
    Call Connect
  End If
End Sub

Now the write method of the socket:

Public Sub Write(mtext As String)
  // Part of the Writeable interface.

  System.DebugLog "Writing to "+Me.Address

  If SockConnected Then
    
    If SocketUpdateTimer <> Nil Then
      SocketUpdateTimer.Period = 1
      SocketUpdateTimer.Mode = Timer.ModeMultiple
    End if

    Sending = True
// Looking at NumWrites is how I determined I was not sending every command I thought I was.
    NumWrites = NumWrites+1
    Super.Write(mtext)
    System.DebugLog "Number Writes: "+NumWrites.ToString
    
    If ResetOnLostConnect Then
      If LostConnectionTimer.Mode = Timer.ModeOff Then
        LostConnectionTimer.Mode = Timer.ModeSingle
      End If
    Else
      LostConnectionTimer.Mode = Timer.ModeOff
    End If
  End If
End Sub

That’s EVERYTHING in the whole chain basically…

Not sure what’s wrong…

Have you tried removing TCPSock.Poll in case it is messing up the stack?

1 Like

Oh, this is a good point. Often Poll will call DoEvents behind the scenes, causing Timers to fire again.

OK guys. I’m an idiot. When I posted this, I was looking at the wrong counter variables I was using to see how many times the code was executing. Prior to calling the first method I list above, I have a different method. There was a bug there that was causing me to exit that method sometimes when I shouldn’t have!

I removed the problem code and now all my counter variables are matching up. So the entire chain is executing properly every time.

Sorry to trouble you…

Now to go crawl under a rock some place…

4 Likes

Mark that as the solution please.

Edit: … and I’m glad it worked out.