Problem with Serial Port Dropping a single byte - for certain data sets

Targets Linux
1

Hello all.

Wrote a console app primarily developed in Windows but for both Windows and Linux. What the app does is to poll a number of remote devices (also developed in house and are known good) who return either an acknowledgement of sorts or some data after interaction with a person. The remotes only speak when first spoken to by the app. The protocol they use is fairly simple and short.

For testing I have used all the same hardware except for the PC where one is Win10 and the other is Linux Mint.

On windows, everything works perfectly all the time. However, on Linux all does not work perfectly. However the failures are consistent!

I get a string of data returned that looks like this (in hex):
3C 0 1 1 41 0 0 A 55 0 0 D <<=== good complete data
3C 0 1 1 41 0 0 CD EF 0 0 D <<==== good complete data

3C 0 1 1 41 0 0 29 0 0 D <<----- BAD DATA, MISSING THE 9th byte which should be 9C
3C 0 1 1 41 0 0 7 0 0 D <<----- BAD DATA, MISSING THE 9th byte which should be B5

There are more examples, but the ones that get the 9th byte thrown out are consistent. The good ones are also consistent.

Both computers use the same USB to Serial Adapter. And I tested on a Raspberry Pi and it too had exactly the same problem.

The set up is like this for both Linux and Windows

Me.Close
Me.DataTerminalReady = True
Me.Bits = Serial.Bits8    //8  //3
Me.Parity = Serial.ParityNone  //  0  'N
Me.Stop = Serial.StopBits1   //0  '1
Me.Baud=Serial.Baud9600
Me.RequestToSend = False

Me.SerialPort = System.SerialPort(SP)
LastErCode = Me.LastErrorCode
Me.Reset

Looking at the data with a scope, it appears the same on both Linux and Windows too, the remote hardware is sending the complete set of data.

Any ideas what is wrong and how to fix???
Thank you,
Tim

2

Check this out…

https://forum.xojo.com/48526-missing-last-byte-from-serial-readall

Ive fixed it by exchanging the USB serial adapter.

3

Thanks for your reply Roman.

The problem is, the same thing happens on the pi without using an adapter. I’m using the UART direct on the pi.

Tim

4

are you using Serial.LookAhead to see if all the bytes are in before reading them with Serial.Read or Serial.ReadAll ?

Also try using Nil encoding and read it using Chr() and Asc() or using EncodeHex() and Hex(). Encodings on serial are a no-go.

5

Yes I am, and when certain indicators are received we wait until the prescribed amount of data is received before readall is done.

The problem only occurs with certain data; those instances are repeatable and consistent. But other data sets equally long or short are not effected. Makes no sense… and only in Linux.

Tim

6

Checksum problems?

Or is there are configuration for linux to filter / process some characters?

7

No checksum issue, for this test, I am capturing data before it is even processed. There is a checksum error but that is because of the missing byte.

8

just for some more understanding can you show part of the protocol how the packets are setup?

9

They are set like I showed in the my original post - but here it is again mapped out more clearly.
3C 0 1 1 [41 0 0 CD EF ] 0 0 D
< Adress msgs payload[ cmd B0 B1 B2 B3] chk Cr

It is dropping byte9, which is represented by byte B3 above. But only in certain instances! Thats the strange part; and only in Linux. I tried with the hardware UART of the Pi, and the USB adapter on a notebook runing Mint.

Tim

10

Hi I am using the Pi, MAC and win (7 and 10) hardware and VM in a project that has spanned several years of development and updating.

it is a serial port based system working with variable packet sizes up to 50 bytes long at 57600 baud.

involved are any old USB to RS232 and RS485 converters.

in all the time this has been in operation i have never once seen any missing data at all on any of the devices, and they all have alerts in place to capture malformed data, which word fine when forced to be faulty, but have never been reported to have fired when in use.

one thing you do not specify is the hardware version of the Pi.
my tests have been with 2 (both versions) and 3+.
there is a difference with the 3 in that they took away the ‘real’ hardware USART in the Pi and pushed it to the bluetooth leaving the secondary, and less reliable soft usart.

there are ways of changing that (if you do not already know), my project does not use bluetooth so i changed the port to use the hardware port on the Pi directly as you say you are doing.

the method i use to get the data out of the serial port is by using a timeout.

1, wait for ‘data available’ event to fire, in the event read the data into a buffer.
2, start a timer with a 20ms period(so it will run in windows properly) from the ‘data available’ event
3, every time the ‘data available’ event fires ‘real all’ and add the data to the buffer, reset the 20ms timer
4, keep doing this for as long as the data comes in (which may only be once)
4, when the 20ms timer times out (as no more data was received) in the ‘action’ event of the timer, cancel the timer and look at the buffer created in the 'data available event, you should have a complete packet every single time.

one thing i know about serial data is that its never as immediate or linear as one might expect, it will do everything possible to screw you up!

the only down side to this is the need to wait 20ms between messages, but for me this is taken into account within this, and other, projects and it have never been an issue.

increasing baud may also help, there is less time for hardware to miss data, but i found 57600 the highest to be reliable.
also i found that RS485 is WAY more reliable in every way to RS232, especially when over 5m cable length, which is about the maximum i would use RS232, all the same code is used in each case as the only difference is the transport hardware not the data its self.

just because you see the issue on linux does not mean there is an issue with linux, exactly the opposite may be true, linux may be working correctly with the code you have written, and by accident the issue with MAC and win that occurs is that you do get the data when perhaps you should not!

getting an expected result may not be the correct result, haha, how many times have i got caught out by that in code over the years!

11

[quote=410454:@Tim Seyfarth]They are set like I showed in the my original post - but here it is again mapped out more clearly.
3C 0 1 1 [41 0 0 CD EF ] 0 0 D
< Adress msgs payload[ cmd B0 B1 B2 B3] chk Cr

It is dropping byte9, which is represented by byte B3 above. But only in certain instances! Thats the strange part; and only in Linux. I tried with the hardware UART of the Pi, and the USB adapter on a notebook runing Mint.

Tim[/quote]

What encoding did you gave the strings? Nil, ASCII, UTF-8 ?
Try to set it to Nil, and then read it again using the Asc/Chr

are you sure that the packets should be:

3C 0 1 1 41 0 0 29 0 0 D

and not:

3C 01 14 10 02 90 0D

2 hex chars = 1 byte or you are not showing everything?

12

Thank you all for your feedback!

The packet 3C 0 1 1 41 0 0 29 0 0 D is missing the 9th byte which should be 9C - Should be 3C 0 1 1 41 0 0 29 9C 0 0 D

Excellent point! Let me check the recording of the data - it matches what I wrote above.

For this debugging session, I add the following code to show me the hex values of the data:

   Buf = Me.LookAhead
   .....  other code


  If Me.BytesAvailable >7 Then
    Dim t as string 
    For n as integer = 1 to Len(buf)   // 5 
      Dim z as integer = Asc(Mid(buf, n, 1) )
      t = t + "  " + Hex(z)
    next n
    Dim Flds() As string = array("TRACE", CurrentMethodName ,  " Data = " + t )
    Logs.AddFields Flds
    Logs.LogFileAppend( Logs.K_Trace_Poll)
  End If

For debugging purposes, since most of the individual data bytes are not printable chars, reading the packet in hex is easiest.

The packet described is my design. The data is sent as bytes, not as strings. I have control over both ends. The non xojo end has been in the field for 20+ years without issue.

Also, the data is coming over via optically isolated RS485.

The pi is the version 3+
The pi has had its setup altered so that it uses the real hardware UART and not the software UART.

13

You say should be: 3C 0 1 1 41 0 0 29 9C 0 0 D
You probably mean: 3C 00 01 01 41 00 00 29 9C 00 00 D ?

Read hex; a 0 should be 00, a 1 should be 01 etc.
https://nl.wikipedia.org/wiki/Hexadecimaal

Your hardware should be sending that, so you might get some encoding trouble here.

Try this code:

[code] Buf = Me.LookAhead // <-- Keeps data in the buffer
… other code

If Me.BytesAvailable > 7 Then // we will read 12 bytes below

// ResultData = UTF8 as xojo creates it that way.
// Me.Read(12, Nil) says let me read it as binary (may be hex, dec or bin)
Dim ResultData As String = Me.Read(12, Nil)
// This removes the data from the buffer.
// Use lookahead to hold the data in the receive buffer
// No make sure this string is still Nil encoded (binary)

If ResultData.Encoding <> Nil Then
ResultData = ResultData.ConvertEncoding(Encodings.UTF8)

// Try reading the 12 bytes for now with Nil encoding, and then converted to hex for readability
ResultData = EncodeHex(ResultData, True)

Dim Flds() As string = array("TRACE", CurrentMethodName ,  " Data = " + ResultData )
Logs.AddFields Flds
Logs.LogFileAppend( Logs.K_Trace_Poll)

End If
[/code]

The Hex() function showed the hex bytes in a short notation.
For debugging purposes try using EncodeHex(“the string”, True), makes reading easier.

Can you test the above code, and let me know the actual output?

14

Will do, thanks Derk!
Tim

15

Well it looks like the missing data is there!
Data = 3C 00 01 01 41 00 00 29 9C 00 00 0D

So, does that mean it is an encoding issue?

16

[quote=410475:@Tim Seyfarth]Well it looks like the missing data is there!
Data = 3C 00 01 01 41 00 00 29 9C 00 00 0D

So, does that mean it is an encoding issue?[/quote]

Not an encoding issue, but try working/setting it to Nil as all xojo strings have UTF8 encoding by default so

Dim s As String // <-- is UTF8 
Dim s As String = Me.ReadAll(Nil) // <-- is nil encoding
Dim s As String = Me.Read(10, Nil) // <-- is nil encoding
Dim s As String = Me.LookAhead(Nil) // is Nil encoding.

You used the Hex() function, that doesn’t pass the full hexadecimal representation for all hex bytes.
Use EncodeHex() instead, and Chr() and Asc() for checking individual bytes.