Characters lost reading RS232 serial port

General
1

I had been trying to find out why I was losing characters when reading an RS232 serial port:

I had this line at the beginning of the DataAvailable event handler for the serial port (note: B1 is global):

'DataAvailable event handler 1 B1=B1+Serial1.ReadAll //Add new data in the serial buffer to pre-received data '... process B1
This results in characters being lost. (It varies, but typically a string of 10 characters lost in every 1500 characters received)

After much hair pulling and trying different things, I changed it to this, and characters are no longer lost

'DataAvailable event handler 2 SR1 = Serial1.ReadAll B1=B1+SR1 //Add new data in the serial buffer to pre-received data '...process B1
Programatically they should be the same, but characters are sometimes lost in the assignment of version 1.
Any comments please about why this happens…?

2

Sounds like some low-level timing conflict. Good to know.

3

Try to move the “Process B1” away from the data acquisition. Make the DataAvailable event as fast as possible, and fire the processing of the received data in a timer and see what happens.

4

You should set an encoding,

B1 = B1 + Serial1.ReadAll( Nil )
B1 = B1.DefineEncoding( Nil )

// now work with the bytes.
// if you need Encodings.UTF8 you can do that. Mostly it's Nil
// And you'd work with the bytes

Every string defined in Xojo has utf8 encoding by default.
Utf8 may use multiple bytes for some characters.

How did you tell you where missing bytes?
What kind of data is being transferred?

5

To me this has proved foolproof through the years, while the dataavailable can be a bit fragile.

6

In my experience you can’t just miss data, but you can have invisible characters if you where to display them while the encoding is incorrect.

To be sure, output the data as Hex using EncodeHex(data)

7

Thanks for the replies, but still not getting to the heart of the reason why Version 1 doesn’t work and Version 2 does…

What I was asking was:

Why are characters lost when I do this inside the DataAvailable event handler (Version 1):

B1=B1+Serial1.ReadAll

But no characters are lost when I do this (Version 2):

SR1 = Serial1.ReadAll B1=B1+SR1
Both inside the DataAvailable event.
Everything else is the same,

To be clear: When I use Version 2, there are never any dropped or lost characters - ever.

Rick - Yes… I am aware of keeping processing inside the DataAvailable event as fast as possible (which it is in my program).

Derk - here’s answer to your questions:
What kind of data is being transferred?

Here is example data (ASCII data, 4800 baud):

$PEDPT,FAWK01,1198,185100.00,0929.2011,1344,270,17.4,21.4,11.1,77.8,999.9,0,12.4,0,148.3*43

Here is what happens to the above line when data is lost (about 1 line in 200):

$PEDPT,FAWK01,1198,185.2011,1344,270,17.4,21.4,11.1,77.8,999.9,0,12.4,0,148.3*43

(‘100.00,0929’ has been lost from the line)

How did you tell you where missing bytes?

There is a checksum error. In the above example, the ‘43’ at the end is the checksum, It is correct in the first line, and incorrect in the second, because the ‘100.00,0929’ substring has been lost.

I do not thinks it is an encoding error because, when using the same test string, Version 1 loses bytes (see above example), but Version 2 does not.
I run the same test again (using the same strings) and no data is lost when using Version 2 instead of Version 1.

8

Ok let’s begin at the beginning.
What kind of device is this?
Does it require half or full duplex, does it have 8N1 settings?
Does it require flow control?
What’s the baud rate for the device?
What encoding is to be used, probably ASCII ?

When you do ReadAll(Encodings.ASCII) it will remove all the read data from the buffer. You can also do .LookAHead(Encodings.ASCII) to see if all the bytes are there, only then read using .Read or .ReadAll

DataAvailable may or may NOT have All bytes. It can and will sometimes only give a few bytes or sometime more bytes than expected. You need to make sure “all” bytes you expect are actually there before removing it from the buffer. Or handle your own buffer.

Did you set the parity, stopbit and bits correctly?

9

It seems something by this lines. Or some other weird low-level Xojo bug. But I insist that there’s a conceptual design error including data processing task inside the data acquisition event. And this can be causing an unknown side effect.

10

What signs your end of line? &h0A ? &h0D? &h0D&h0A? Other char?

11

Here’s some wild speculation: in V1, Xojo gets a little bit busy performing concatenation of B1 with the string returned by ReadAll and misses bytes (or clears its buffer after additional bytes have been received, losing those bytes), whereas in V2 it only has to do an assignment to SR1, which is faster than concatenation. It may depend on when exactly the Xojo port’s buffer is being cleared by ReadAll.

12

I would try something (not tested, just wrote it) like this.

Decoupling acquisition from processing, assuming LF as EOL:


Class Window1
Inherits Window

Methods

Sub ProcessTheLine(line As  String)

  // process the line we got from the serial device
  
  // Make sure that all com settings are ok, 
  //                   baud rate, data bits, parity, stop bits, and handshake (xon/xoff, CTS, DTR)
  //                   a wrong handshake could cause data loss or data jamming
   
  System.DebugLog line // do something here
  
End Sub


Properties

LinesReceived() As String


Control Serial1: Serial

Sub DataAvailable()

  #Pragma DisableBackgroundTasks

  Static buf As String = ""
  
  Dim anyLineReceived As Boolean = False
  
  Dim rec As String = me.Read(1, Encodings.UTF8) // Get a char, UTF8 mostly works for pure ASCII too 
  Do Until rec = "" // Our readall()
    If rec = Chr(10) Then // let's suppose LF is the line end mark
      LinesReceived.Append buf // save this line in the received lines to process
      buf = "" // Clear buffer
      anyLineReceived = True
    Else
      buf = buf + rec // Add char to the line buffer
    End
    rec = me.Read(1, Encodings.UTF8) // read another char
  Loop
  
  If anyLineReceived Then
    TimerProcessLines.Period = 10
    TimerProcessLines.Mode = Timer.ModeSingle
    TimerProcessLines.Enabled = True // Fire the processing and keep receiving data
  End

End Sub

Control TimerProcessLines: Timer
Sub Action()

  If LinesReceived.Ubound<0 Then Return // Nothing to process. Stop and wait acquisition fire it again
  
  ProcessTheLine(LinesReceived(0)) // Do whatever you need to with the oldest line in the line buffer
  LinesReceived.Remove(0) // remove it from the pile
  
  TimerProcessLines.Period = 10
  TimerProcessLines.Mode = Timer.ModeSingle
  TimerProcessLines.Enabled = True // Fire another line processing event and but keep receiving data
  
End Sub

End Class
13

hi All
Thanks again for the responses.

Here’s just some updates…

//Reading data from serial ports // This loses data (Version 1 DataAvailable event handler): B1=B1+Serial1.ReadAll

//But no characters are lost when I do this (Version 2 DataAvailable event handler): SR1 = Serial1.ReadAll B1=B1+SR1

I have quadrupled the data rate (19200 baud test mode instead of 4800 baud, which is used in the final version).
Have also set up 4 serial port objects, working simultaneously, at the higher baud rate.
Version 2 never loses data, Version 1 does.

In regard to the suggestions:

  1. Encoding: All the data is RS232, 4800 baud, 1-byte ASCII, so I don’t think encoding is an issue.
    In any case I have kept the everything the same (including the test data strings) and Version 1 always drops some data, Version 2 doesn’t. Trying different encoding options makes no difference.

  2. Thanks Rick for the timer based code sample, but as mentioned I have never seen any data lost with Version 2, so I don’t think performing minimal processing in the DataAvailable event handler is a problem, so will stick with that approach.

FWIW, I think something along the lines of what Julia Truchsess suggested is likely the problem - there is some data lost during the assignments in version 1.

Anyway I don’t have time to investigate further. I would like to know why Version 1 loses bytes, but will go with Version 2 since after testing for several days continuously at rates much higher than required, I have never seen it lose data.
It would be good to know what’s wrong with Version 1, but will use Version 2.

14

[quote=469790:@Gary Quinn]//Reading data from serial ports
// This loses data (Version 1 DataAvailable event handler):
B1=B1+Serial1.ReadAll

//But no characters are lost when I do this (Version 2 DataAvailable event handler):
SR1 = Serial1.ReadAll
B1=B1+SR1

I have quadrupled the data rate (19200 baud test mode instead of 4800 baud, which is used in the final version).
Have also set up 4 serial port objects, working simultaneously, at the higher baud rate.
Version 2 never loses data, Version 1 does.[/quote]

I would love the engineers investigating this issue.
Seems worrying and at the same time easy to find what’s the difference between both and fix it, or explain why it occured and say “don’t do that” in the docs.

Can you open a feedback? We can’t lose data like this, this can be a serious issue.