I’m revising some code of a TCPSocket that processes DataAvailable. If its a full packet, the TCPSocket replies (write) back to the server. At the same time the socket is writing, new data can be coming in. Occasionally there is a slight delay to update my main window and I’m wondering if this is the cause. How does it work?
Thanks
As I understand things, when DataAvailable fires, it provides whatever it could pull from the incoming buffer at the time that the event fired to the body of DataAvailable. Other data may arrive while DataAvailable is running, and the OS will just queue it up in the incoming buffer. After DataAvailable has completed, at some point the socket will either have .poll called on it or the OS will trigger the DataAvailable event again with the new data that arrived.
So, more data can arrive in the incoming socket buffer while you are still handling the prior data that was sucked out of the buffer.
If, as a part of the processing of the dataAvailable you are calling .write() on the socket, all that really does is write the data to the outgoing buffer of the socket that the OS manages. When that data actually goes out over the network is entirely dependent on how the OS decides to do so.
When you say there is a ‘slight delay’, can you quantify that? A few microseconds, milliseconds, or several full seconds?
One thing you could try would be to have the work that DataAvailable is doing handled in a thread, so all that DataAvailable does is spawn the thread that handles the response. This thread can deal with the response and write back to the socket in the background, meanwhile your UI stays responsive.
But, in my experience, since calling .write on a socket really just pushes your data to an outgoing buffer handled by the OS, it is generally nearly instantaneous, and should not be causing a noticeable delay. If you are doing other computationally expensive things in DataAvailable, though, that may cause a delay in your UI.
Sometimes it’s no delay, other times maybe a few seconds.
Doesn’t Xojo handle the problem of the hardware buffers possibly losing data? I thought for this reason, we don’t need to handle our own buffer.
What is considered to be too computationally expensive for the DataAvailable event?
[quote=175592:@Mark Scardingo]Sometimes it’s no delay, other times maybe a few seconds.
Doesn’t Xojo handle the problem of the hardware buffers possibly losing data? I thought for this reason, we don’t need to handle our own buffer.
What is considered to be too computationally expensive for the DataAvailable event?[/quote]
Well, I do a number of RegEx searches in data available events and it works just fine. I’ve never seen an issue with losing data while processing the data available event.
Mostly yes
However if you have a really slow bunch of code in DataAvailable you CAN lose data
[quote=175592:@Mark Scardingo]
What is considered to be too computationally expensive for the DataAvailable event?[/quote]
Checking IF you have a full message and if so processing it through a long sequence of methods calls.
While thats the straight forward & obvious way to do it is CAN also cause you to lose or miss data.
[quote=175670:@Norman Palardy]
Checking IF you have a full message and if so processing it through a long sequence of methods calls.
While thats the straight forward & obvious way to do it is CAN also cause you to lose or miss data.[/quote]
Is this correct?
[code]Sub DataAvailable()
Do
// Attempt to process the packet
dim bytes as Integer
bytes = PacketProcess( me.LookAhead( Encodings.UTF8 ) ) <-- Put this in a thread or timer
if bytes > 0 then
// We've successfully received a full packet
// Remove the data from the RB internal buffer
call me.Read( bytes, Encodings.UTF8 )
end if
loop until me.BytesAvailable <= 0
End Sub[/code]
[code]Class MyTCPSocket
Private Property mBuffer As String
Sub DataAvailable()
mBuffer.Append Self.ReadAll(Encodings.UTF8)
End
[/code]
Put the rest in a timer’s Action event or in a thread. There you get from mBuffer what you need and do the processing.
I have found it very easy and it works well to use the LookAhead function of the socket until the data in the socket has what I am looking for.
So in some code I just spent quite a bit of time working on, I do this and it works well. What I am doing is logging into a device and then sending that device commands and reading back what the response is. I have quite a number of different commands to send to get all the device parameters that I need. So I have a select case statement and an enumeration to keep track of where I am at in my process. Things like GetFIrmwareVersion, GetSerialPortSettings, GetIFConfigData, etc. are all parts of my enumeration. As I step through each step, I set the enumeration to the next step in the process. Then the select case statement in the DatAvailable event directs me to the proper branch for what I want to process.
Once in the branch I want to process, I use a RegEx to verify if the receive buffer has the data I want. Here’s an example of one of those. I’ve sent the command ifconfig to the device I am reading. This is the code to process that…
Case HardwareParameterStep.GetIFConfigData
s = Lookahead(Encodings.ASCII) ' s is a string variable defined earlier in the DataAvailable Event.
dim rx as new RegEx
rx.SearchPattern = _
"(?mi-Us)HWaddr (\\b[[:xdigit:]]{1,2}(:|-)(?:[[:xdigit:]]{1,2}\\g2){4}[[:xdigit:]]{1,2}\\b)\\s*(?(DEFINE)(?'octet'25[0-5]|2[0" + _
"-4]\\d|[01]?\\d{1,2}))(?(DEFINE)(?'IPAddy'\\b(?'octet1'(?&octet))\\.(?'octet2'(?&octet))\\.(?'octet3'(?&octet))\\.(?'octet4'(?" + _
"&octet))\\b))inet addr:(?'IPAddy1'(?&IPAddy))\\s*Bcast:(?'IPAddy2'(?&IPAddy))\\s*Mask:(?'IPAddy3'(?&IPAddy))"
dim rxOptions as RegExOptions = rx.Options
rxOptions.LineEndType = 4
dim match as RegExMatch = rx.Search( s )
If match <> Nil Then
s = ReadAll(Encodings.ASCII) ' I have received all the data I want. So now, read everything in the buffer.
If mMacAddress <> match.SubExpressionString(1) Then ' Set my MacAddress property to the first sub-expression
mMacAddress = match.SubExpressionString(1) ' If it doesn't match what I already have as the MAC address
End If
If IPAddress <> match.SubExpressionString(9) Then ' Set my IP Address property
IPAddress = match.SubExpressionString(9)
End If
If SubnetMask <> match.SubExpressionString(11) Then ' Set my SubnetMask property
SubnetMask = match.SubExpressionString(11)
End If
Else
Return ' The match is nil - so I have not found the data I am looking for in the buffer. Return and wait for next fire of event.
End If
So I know the format that the IFConfig data comes back to me. I look for that format. It’s quick and I’ve not had any issues with losing data.
In my development, I tried using a timer to do all the processing. I fired the timer from the DataAvailable event. My results were not as fast or as good. I don’t understand why people want to use a property to hold the value of the receive buffer and effectively buffer it again. The buffer is there and the lookahead function gives you that ability to look at it. I have one case where I do cache the data in a separate property and look at it later, but that is only because I need to do other stuff with the data that comes in.
Personally, I have found that letting the framework fire the DataAvailable event and checking for your data there tends to work really, really well. Yes, I agree if you end up doing some long computation or something that is CPU intensive, then you may miss data. But once you have all your data, it is all good anyhow.
One thing I have found that you do NOT want to do when using sockets and the data available event is to sleep the main thread. I used to sleep the main thread sometimes for some various reasons and found that sleeping the main thread causes data to get missed in the DataAvailable event. That caused problems. As soon as I stopped sleeping the main thread during any i/o events, it went much better.
Anyhow, YMMV, but this has worked out really well for me like this. It’s fast and works.
Thanks guys.
If going the timer route, how do you manage the timer? Does it stay running for the life of the socket, or do you fire it up in Connected or DataAvailable? Also, what is the recommended timer period for this situation?
[quote=175924:@Mark Scardingo]Thanks guys.
If going the timer route, how do you manage the timer? Does it stay running for the life of the socket, or do you fire it up in Connected or DataAvailable? Also, what is the recommended timer period for this situation?[/quote]
To me, I see no reason trying to “time” anything. Let the socket fire its DataAvailable event as needed and process the socket’s internal buffer at that time. If you have your full message, process it. If you don’t, return from the method and wait for it to fire again.
When I had the timer, I would fire it from the DataAvailable event and its period was 0. But, I found that even with a period of 0, that it could be several milliseconds before the timer would actually execute. I found it was faster to simply let the DataAvailable event handle what it needed to on its own.
And that can, and does, sometimes cause you to end up with “long running” chains of code that can mean you drop data.
That’s the point I was trying to make but …
[quote=175998:@Norman Palardy]And that can, and does, sometimes cause you to end up with “long running” chains of code that can mean you drop data.
That’s the point I was trying to make but …[/quote]
Oh I agree it could. If you get the message you want and then need to do a long bit of work with it, I would absolutely recommend then threading it or putting it in a timer. If it is going to go quick, I bet you would be fine. The socket is asynchronous and it’s buffer should technically hold any incoming data until you pull it out with a read command. So if I am not pulling data out of the buffer until I make sure I have the response I am looking for, I’m not sure how I am going to lose data?
The hardware buffers are of limited size as are the OS allocated buffers
Thats where, if you don’t respond to data available quickly you can lose data - it literally could get overwritten by the next one
Of course, if your transmission size is small and processing is quick, there should be no problem handling it directly in DataAvailable. You won’t overflow the OS buffer. Otherwise, buffer it yourself. I always pull everything into my own buffer out of habit, but you can get away without.
[quote=176006:@Norman Palardy]The hardware buffers are of limited size as are the OS allocated buffers
Thats where, if you don’t respond to data available quickly you can lose data - it literally could get overwritten by the next one[/quote]
How big are the buffers? How many bytes can they hold?
It doesn’t matter. They could be different from platform to platform or OS to OS and you should not be relying on them.
Only use the DataAvailable event to transfer the data to your own buffer and to start a timer which fires your processing method. Just make sure your processing method can handle the situation of bytes being added to the end of your buffer.
Is it faster to use an array as a buffer or is this ok? Thanks
[code]Sub DataAvailable()
PacketProcessor.mBuffer = PacketProcessor.mBuffer + self.ReadAll(Encodings.UTF8)
if PacketProcessor.State = Thread.NotRunning then
PacketProcessor.Run
end if
End Sub
[/code]
[code]Sub Run()
While mBuffer <> “”
dim bytes as Integer = ProcessPacket( mBuffer )
if bytes > 0 then
// We've successfully received a full packet. Remove it from the buffer.
mBuffer = mBuffer.MidB( bytes + 1 )
end if
Wend
End Sub
[/code]
appending strings isn’t the fastest process and has some overhead as you create & destroy temporaries with the assignment
a binary stream backed by a memoryblock might be quicker
ie/
dim mb as new memoryblock(0)
dim bs as new binarystream(mb)
however you also have to worry about the memoryblock growing indefinitely if you do this so you need to rip packets off and shrink the mb by whatever size the packet is
Also note that in this case a MB is much more explicit about things being BYTES - which you’re doing implicitly using the string the way you are. In the new framework you would be required to use a memory block as Text does not have the dual nature that strings do today