I have a piece of test equipment that is linked to an internal TCP port through the test equipment app. I am able to send (write) commands to the app flawlessly, but when I make queries (myTCPsocket.Readall), I do not get the expected results. There appears to be a hidden buffer, because the first query gives an empty string, then the subsequent query gives the expected result to the first query. Thereafter, I always get the response one query late. It is not a timing issue, as I have the read in a loop that I have let go as long as 20 seconds (typical response time is 200 milliseconds).
in the DataAvailable event ANY data or NO data may be given. It’s up to your code to see if the data is valid and if everything is received, that you are expecting. This is normal behaviour.
You can read this (frontpage)
http://documentation.xojo.com/api/networking/tcpsocket.html
And use LookAhead (keeps in the buffer) to see if the data satisfies your algorithm.
http://documentation.xojo.com/api/networking/tcpsocket.html#tcpsocket-lookahead
When the data is there, read it using .Read or .ReadAll (removes from the buffer)
Oh and Xojo does not have TCP_NODELAY option, MBS plugins does have this, this disables the pre-buffer algorithm (basicly waits … x ms until x amount of data is received or x ms has passed).
1 Like
Thank you. I did not know about the lookahead function. I have modified my listing to use lookahead rather than readall but the results are similar (after making necessary adjustments). It seems that the only difference between readall and lookahead is that the former clears the receive register and the latter does not.
Below is a complete listing of my thread. I continue to get a response to my previous query, not my most recent one, as if the most recent response were in a separate register and can only be pushed into the receive buffer upon the issuing of another command.
//Thread imports VNAcmd from main where thread.run is initiated
//It issues the query and listens until thread finishes
//Receive data ends up in accumulator and passed back (to calling main)
//VNAsocket is a TCPSocket
//accumulator is a global string variable
//VNAdat is a global string variable (used here locally)
//VNAbusy is a global boolean variable
var ctr as integer
var ctr1 as integer
var trash as string
accumulator = “” //initialize
VNAsocket.Write (VNA_cmd + chr(10)) //send command
ctr = 0 //counter for end of new data
for ctr1 = 1 to 100 //read a number of times
VNAdat = VNAsocket.lookahead //read
if VNAdat <> accumulator then //new dat available
accumulator = VNAdat //update
ctr1 = 1 //reset loop counter on data change
ctr = 1 //reset end of data counter on data change
if ctr > 10 then //no new data for 10 counts:
exit for //all done
end if
ctr = ctr + 1 //incerment end of new data counter
end if
next ctr1
trash = VNAsocket.ReadAll //empties lookahead buffer
VNAsocket.Purge //empties VNAsocket receive buffer
//accumulator is passed to calling sub
VNAbusy = false //thread done: signals main to continue
Not a good idea. Use the DataAvailable event and maintain your current state (“waiting for response to X”).
1 Like
I use the loop to continue to check for more data as I don’t know the length of the incoming string. I keep checking to see that the data is stable. Is there a better way?
Use the data available event to read and concatenate the incoming data into a property.
You then need to look for your message terminator and if one exists split the string (leaving the remainder in the property) and then process the message.
Depending on how frequent the other end sends data you might have more than one message you need to process.
My situation is in some ways much more simple. I am linked to one server, which is the equipment app. I only get messages from that source in response to a specific query Because of that, separating messages is not a problem, other than my main problem, which is that I find the receive buffer always contains a perfectly correct response to my previous query, not my current one. At launch or after a purge, there is no apparent response, however any subsequent query results in the response expected before.
I could just query twice and just accept the second, you might suggest. I can, in fact do that from the UI, but when I wrap up the two query/response code elements into a continuously executing string, it doesn’t work, even with a 5 second sleep between the two. I’ve tried doing it with subsequent calls to the thread and I’ve tried it with both operations in the same thread.
The thing I don’t see being said here is why it’s so bad to process data in the DataAvailable event.
The main issue here is that DataAvailable can be reentrant. That is, the event can fire while you are still processing data from the previous event. As was mentioned above, the only thing you should be doing in the DataAvailable event is adding the internal data to a buffer of some kind and then using Timer.CallLater to fire an event to process the data.
I usually use code that looks something like this:
Buffer.add Self.ReadAll // Buffer is an array of Strings
Timer.CancelCallLater(addressOf myProcessingMethod) // this coalesces data coming in quickly
Timer.CallLater(10, addressOf myProcessingMethod) // 10 ms delay
Then the method could look something like this:
Sub myProcessingMethod(t as Timer)
// I'm writing this code in the Forum
Var terminatingDelimiter as String = chrb(0) // whatever you use to terminate your packets
Static processingData As Boolean
If Not processingData Then
processingData = True
// Find the end chunk
Var lastChunk As Integer = -1
For i As Integer = Buffer.Ubound DownTo 0
If Buffer(i).IndexOf(terminatingDelimiter) > -1 Then
lastChunk = i
Exit For i
End If
Next i
// Bail out if the terminator wasn't found
if lastChunk = -1 Then
processingData = False
Return
End If
// If the terminator *was* found, then put the pieces together
// into a string removing them from the array as we go
Var newData As String
For i As Integer = 0 To lastChunk
newData = newData + Buffer(0)
Buffer.RemoveAt(0)
Next i
// Call a method to process your packet
ProcessPacket(newData)
End If
processingData = False
End Sub
7 Likes
The fact that DataAvailable is reentrant is not noted in the documentation for TCPSocket. Perhaps this should simply be assumed – that all events are reentrant? If not, this would be a glaring hole in the documentation for this event.
1 Like
Another fact is that we have never had re-entry issues, working without timers. We have always used sockets event based on the main thread and they work flawless. Maybe the re-entry is for iOS only? Or other platforms?
1 Like
This is an inherent property of TCP Socket protocol (not just Xojo). DataAvailable, occurs when data is received on the socket, and can arrive in parts/pieces, thus why a termination character is used to determine when the"full string" of data, has arrived. Imagine a faucet with a small pebble somewhere within the line. By running the faucet, you’re similarly reading data to the buffer, and until you see the pebble (term char), the “data” is not ready for processing (so keep reading to the buffer). Sometimes, the pieces may arrive delayed (ie latency), but most generally sequentially and nearly “instantly”… So knowing when your data starts and end, is key (buffer processing).
DataAvailable is a state event of the TCP Socket object, and can only exist once per object instance, but can be invoked infinitely (making it re-entrant). Data is, or is not available; and if it is, then the event state is true (event raised).
1 Like
This is exactly what one has to do with SerialDevice as well. I think the symptoms I experienced when trying to do parsing in SerialDevice.DataAvailable were the same as or similar to those reported by the OP with TCPsocket. You have to use a timer to get the parsing and handling code onto the main thread.
1 Like
I have a thread that processes the data. When it needs data, it calls a method which will look in the thread’s buffer and normally it returns some of that. If that buffer is empty then it looks at BytesAvailable, and if there is some, does a ReadAll into the thread’s buffer, and then returns some of that.
If BytesAvailable is zero, the method will Pause the thread, having started a timer. This allows the waiting-for-data operation to be timed out.
All that the DataAvailable, Error, and Timer events do is to resume the thread.
What could possibly go wrong? Well, for the most part apparently nothing, but one of my remote data sources had an outage recently of some days, and since it’s returned has shown a tendency to generate timeouts. Not many, perhaps three or four a day, but it’s left me wondering whether I’ve got a a race condition somewhere.
Does adding
#pragma DisableBackgroundTasks
to your DataAvailable event code (and any code it calls) prevent re-entrancy? It should, whcih might simplify your logic.
That is reassuring. Perhaps my problem is with the test equipment app and not my Xojo programming. I’m trying to explore all options. Thanks for your help.
I’m using an approach similar to yours, and like you, I’m thinking I have a problem with the app that I’m communicating with. I’m exploring that end with the supplier of the app. Thanks for your take on it.
I’ve done several itterations of the software, some using the DataAvailable event, and some using a direct TCPSocket.readall. All approaches produce similar results. I’m now calling the thread that addresses the TCPSocket and increasing the priority of the thread to speed it up. That is the only background task I have active.
I’m using a SerialDevice in this same program and have no problems with it. I am using a similar approach with the TCPSocket and there’s where the problem occurs. I am becoming more and more convinced that my problem is with my communication protocol with the app that is giving me the problem, rather than with Xojo. I’m working on getting help from the suppliers of the app.
Thanks for your help.
I’ve been reading the DataAvailable until I get a set number of empty strings. Since I’m working with a local host, I know there will not be significant latency in transferring data, and the approach works with no partial responses seen. Is the terminating character inserted by the TCP protocol or is is normally expected to be supplied in the data stream from the sender? If it is supplied by the TCP protocol, I could check for EOF as a quicker way to assure that I have a complete response.
Thanks for the well-annotated bit of code. I appreciate code I can understand.