Hidden buffer in TCPsocket?

Getting Started
41

Have you wiresharked the data to see if the packet is coming across in the first instance with the expected data in?

1 Like
42

Even if there is another buffer that wonā€™t explain your issue of missing data.
Eighter you are NOT requesting the data or your are cutting away too much data (packet not entirely read, discarded) or your sending wrong data and cause the end device to discard a packet (while the next comes trough good).

Xojo uses the system buffer that you can view using .LookAhead (keeps in the buffer) or .Read, .ReadAll to read from the buffer (clearing it from the system buffer). Now if your end device or OS buffer are set too small you may end up with such issues as (seemingly) loosing data.

43

I assume you mean by ā€œthe next event fireā€ as the DataAvaible event. I have put the thread in a loop looking for more data for up to 5 seconds and get nothing, which is expected, as the equipment app typically has a response time of 200 milliseconds. Iā€™m looking for responses that are not large: from 1 byte to a few hundred. There isnā€™t an overflow problem.

44

You NEED to disable the Nagleā€™s algorithm that is stacking your data to a certain treshold by setting TCP_NODELAY but since xojo doesnā€™t have such option you may want to add FB request and wait forever or use MBS plugins perhaps.

45

I unfortunately donā€™t have hardware monitoring capability for TCP. I do have it for RS-232/485.

46

Actually no, thinking back on it, the next fire of DataAvaible after the loop had 1 piece of data in, then the next fire of DataAvaible after that had a bunch of data in from multiple sends but they did all come through after the loop ended with no intervention needed at the client.

47

Canā€™t install wireshark? Itā€™s a piece of software used to network protocol analyse.

48

I do not process the data in the thread that gathers the data. (Iā€™ve tried the dataavailable event handler and a separate thread at different times, both with identical results. In every case, I pass the data to the main thread for processsing.

49

I will look into it. Thanks.

50

Thatā€™s the Nagleā€™s algorithm doing that stacking so you get MAYBE no data, MAYBE a part, MAYBE ALL data or MAYBE more then you EXPECT. This is how xojo receives itā€™s data, thatā€™s normal. At best you can debug it like so:

.DataAvailable event or your polling thread:

#If DebugBuild
System.debuglog "BUFFER: "+ EncodeHex(me.LookAhead)
// Ditch the EncodeHex if you have already human readable-data.
#End If
51

Could be, might be worth disabling at least to try and see if it makes a difference.

Canā€™t post the link to the other forum so hereā€™s the code I posted there about it:

Declare Function setsockopt Lib "Ws2_32.dll" Alias "setsockopt" ( _
s As Integer, _
level As Int32, _
optname As Int32, _
ByRef optval As Boolean, _
optlen As Int32 _
) As Int32

Const IPPROTO_TCP = 6
Const TCP_NODELAY = &h0001

Dim noDelay As Boolean = True
Dim noDelayLength As Int32 = 1 'byte length of noDelay

Dim ok As Int32 = setsockopt(YourSocketGoesHere.Handle, IPPROTO_TCP, TCP_NODELAY, noDelay, noDelayLength)
1 Like
52

One of the things I tried was to issue a number of queries before reading the buffer. All of the expected responses were there, but NOT ON THE NEXT read, but on the read after a subsequent queryā€¦always delayed by exactly one query.

53

Please show your algorithm of reading the data. Thatā€™s a fair way to help you get more possible solutions. Also try to implement the socket on the main thread in a example project in itā€™s most simplest form, that way you can better debug the info.

Also debug, and print the actual data in the buffer in pre-processing and post-processing that way you exactly know the state of your socket trough the debuglog. (messages pane in the ide)

54

Here it is as an extention method you can put in a global module:

Public Sub NoDelay(Extends sock As TCPSocket, Assigns Value As Boolean)
  #If TargetWindows Then
    
    Declare Function setsockopt Lib "Ws2_32.dll" Alias "setsockopt" ( _
    s As Integer, _
    level As Int32, _
    optname As Int32, _
    ByRef optval As Boolean, _
    optlen As Int32 _
    ) As Int32
    
    Const IPPROTO_TCP = 6
    Const TCP_NODELAY = &h0001
    
    Dim noDelay As Boolean = value
    Dim noDelayLength As Int32 = 1 'byte length of noDelay
    
    Dim ok As Int32 = setsockopt(sock.Handle, IPPROTO_TCP, TCP_NODELAY, noDelay, noDelayLength)
    
  #Else
    
    // No code yet for MacOS and Linux, may be added here:
    #Pragma Warning "There is no NoDelay implemented in MacOS"

  #EndIf
End Sub
55

My queries are now initiated through the UI, so there is no chance that events would be walking on top of each otherā€¦the timing is way off. When I do get a packet, I have never had an instance where the packet was corrupted (parts doubled or parts missing).

56

Iā€™ve done a similar thing. In fact, Iā€™ve used the thread that owns the socket to poll the socket without instantiating the dataavailable event. I have not found that using the dataavailable event handler gives me any benefit, as I always know that data can only be availbable after Iā€™ve issued a query. I have a closed system where there is nothing else on the network. I issue a query, then run the thread. The thread looks long and hard at the input buffer (similar results using readall and lookahead) before returning the complete response to the main thread. I speed up the process by giving the thread a high priority while it is running.

57

Do you ReadALL into your own buffer always or do you check the system buffer before you read anything ?

58

Iā€™ve tried three different methods, all with similar results:

  1. Create a dataavailable event to read the buffer and pass the results to the calling thread.
  2. Poll the buffer from a thread using a socket.readall.
  3. Poll the buffer from a thread using socket.readall.

In each case, I have used the thread to insure that I get all of the response message and have had no problems with incomplete or corrupted responses.

59

You should call .Poll in the thread and see if that helps.

Or you could elimiate the thread entirely and see if that helps you out.

60

I have tried polling using socket.readall as well as socket.lookahead from the main as well as from a thread, all with similar results.

For all who have been following this topic, I have just gotten word back from the test equipment vendor who now believes that there may be a problem with their software. If so, the problem may be cured! Iā€™ll continue to post notices until it is verified. I have spent weeks wrestling with the problem, but it has not been time lostā€“Iā€™ve learned so much from the Xojo community! Thanks to all.

5 Likes