Receiving Variable-length Serial Message

Getting Started
1

I’m from the microcontroller world where sending and receiving serial data is pretty straightforward. I’m looking for guidance on converting a process I use in a microcontroller to work in Xogo.

This is for a serial connection with an XBee radio device using the API-1 mode. Sending a message is not a problem. Where things can get tricky is receiving messages, as they can vary in length and don’t have a specific terminating character. A message could look like this:

7E 00 06 88 02 41 50 00 01 E3

In the micro controller world I run a method that does this:

  1. Is first byte 7E? (if no, abort)
  2. is second byte 00? (if no, abort)
  3. This byte defines the length of the rest of the message minus one (note that 7 bytes follow the 06)
  4. Grab the other bytes and set a “hasMessage” flag to be dealt with elsewhere

Again, I’m new to Xojo (a couple weeks in), but I have always found doing something meaningful beats running through demos that do no apply to what I’m doing. Unfortunately, the only official serial demo is about receiving terminated strings – nothing byte-by-byte.

Any advice or guidance is appreciated.

2

you should be able to use the “read” method for the serial class
the first parameter is the number of byte(s) to read.

https://documentation.xojo.com/api/deprecated/serial.html

https://documentation.xojo.com/api/language/readable.html#readable-read

3

That 7th byte in the payload is a checksum if I am right, so you need to read it, it’s part of the message.

One thing you could do is receive characters in a FIFO buffer. This way if he sender send more than one message at a time, you don’t loose characters. You would do that in a thread. In a second thread you read the characters one a a time: while you get the characters and everything is doing fine, you create a message in an array or class as you wish.

Char #1 is 7E, copy it to the new message
Char #2 is 00, copy it at the end of the new message
Char #3 is the number of characters in the payload, then you know how many characters you need to get to complete the message.
Once you get the last character, the check sum then you have a complete message,.
Compute the checksum and compare it to the one received. It they match, then act on the message. If not, act accordingly.
If something fails during analysing of the message read from the FIFO, erase the message you are creating and then wait for 7E to start a new message.

Does that help ? If you do not want to use threads, then simply set the hasMessage flag and process the message just received.

Do you have a good documentation about the XBee ?

4

Use ReadAll to get the data, then examine the bytes. Pull what you need from the string, store the rest in a buffer, then add onto the buffer on the next pass.

Something like this:

thisData = Buffer + ReadAll
Buffer = ""

do until thisData = ""
  if thisData.Bytes < 5 then
    // not enough data yet
    Buffer = thisData
    return
  end if

  if thisData.LeftBytes( 1 ) <> String.ChrByte( &h7E ) then
    thisData = thisData.MiddleBytes( 1 )
  
  else if thisData.MiddleBytes( 1, 1 ) <>  String.ChrByte( 0 ) then
    thisData = thisData.MiddleBytes( 2 )

  else
    var dataLength as integer = thisData.MiddleBytes( 2, 1 ).AscByte
    if thisData.Bytes < ( 4 + dataLength + 1 ) then
      // not enough
      Buffer = thisData
      return
    end if

    var chunk as string = thisData.MiddleBytes( 2, dataLength + 1 )
    thisData = thisData.MiddleBytes( 3 + dataLength + 1 )

   // process chunk
  end if
loop

Or something like that. I haven’t tested this so forgive typos or logic errors.

5

I think my biggest problem is adapting to the event-driven environment of Xojo. In the Propeller microcontroller, my code is a timed loop that runs a state-machine – at the top of that is a call to check for an XBee message. If a valid message is available a handler is called.

I’ll have a look at threading. My initial inclination is to setup a timer to look for a message every 25ms (same rate as my microcontroller code). I’m guessing that’s not the best long-term solution, though.

FWIW, my microcontroller code is running on 1000s of laser tag weapons in use across the country and overseas, too. Dealing with XBee isn’t difficult, I’m just struggling with a new platform.

Thanks for your feedback.

6

I need to look at this a bit to understand your thinking. Thanks for the feedback.

7

Getting used to event-driven, object-oriented programming is a challenge, especially when you’re used to doing things in a certain way. It’s something we’ve all been through, but one day it will be like a light switch got flicked on and your world will change. We’ve all been there too.

The example code above would go into the SerialConnection.DataReceived event. In fact, the best way to do it would be to create a subclass of SerialConnection called, for example, XbeeConnection, put the code in it’s DataReceived event, then raise additional events depending on the instructions given. You could then drag your XbeeConnection to, say, a Window and implement the new events you created to deal with the data coming in from the device. This lets you encapsulate the low-level code in your subclass, and deal with the high-level instructions in an easily understandable and maintainable way.

This is the difference between asking if there is data to consider (i.e., using a Timer to poll the connection) and being told when data becomes available (i.e., dealing with an event as it arises). It’s a different paradigm, but one you’ll grow to love.

But baby steps. :slight_smile: First put that code (or code like it) into the right event and go from there.

8

In DataReceived, you will get some bytes. It may be a single message. It may be part of a message. It may be the rest of a message you already recieved part of. Or it may be the end of one message and the beginning of another. Don’t assume anything. That is why most people recommend reading all the available bytes received and appending them to a buffer, then evaluating what you have.

ReadAll into a string.
Append to a buffer string.
Is first byte 7E? if not abort
Is second byte 00? if not abort
MessageLength = byte 3
Message = buffer.MiddleBytes(3, MessageLength)
buffer = buffer.MiddleBytes(MessageLength+3) // remove the message from the buffer

1 Like
9

Here’s what I do for talking to XBees with API mode. You want all data handling to be done on the main thread, not on the Serial Connection’s thread, so use a timer to move it to the main thread. I make a subclass of SerialConnection and call it “XBeeDevice” or some such. I give it a timer property and initialize the timer in the class constructor:

// Polltimer is a one-shot, fired by DataReceived, and serves simply to get the buffer parsing code out of the
// DataReceived event.

PollTimer = new Timer
PollTimer.Period = PollTime // 2ms typ
PollTimer.Mode = timer.ModeSingle
PollTimer.Enabled = True
AddHandler PollTimer.Action, WeakAddressOf ParseRX

The DataReceived event handler is simply:

RXBuffer = RXBuffer + Me.ReadAll(Encodings.ASCII)
PollTimer.Reset // PollTimer.Action is handled by ParseRX

I have a class APIpacket which encapsulates all the properties of an API packet, with a constructor that takes a string parameter. The ParseRX method of the SerialConnection class looks like this:

// This is the handler for PollTimer, which is a one-shot fired by our DataReceived event.

// AT Response Packet
// <SOX><Len MSB><Len LSB><FrameType 0x88><FrameID><Cmd1><Cmd2><Status><data><Checksum>

// Receive Packet Packet
// <SOX><Len MSB><Len LSB><FrameType 0x90><Source MAC><Source Net Addr><ReceiveOptions><Payload><Checksum>

Dim PacketStart, PacketLen As Integer
Dim ResponseStatus As Integer
Dim Packet As APIpacket

// Align buffer with SOX at start
PacketStart = RXBuffer.IndexOfBytes(0,ChrB(SOX))
RXBuffer = RXBuffer.MiddleBytes(PacketStart)

If  LenB(RXBuffer) >= 3 Then // We have enough length to try parsing, i.e. we have the Length field.
  
  PacketLen = AscB(MidB(RXBuffer,3,1)) // Ignore MSByte, we don't get messages > 255 bytes.
  
  If LenB(RXBuffer) >= PacketLen + 4 Then // We have all the bytes of the packet as specified by its length field
    
    Packet = New APIpacket(RXBuffer)
    
    If Packet.Checksum = 0 Then // Got a packet, but note that it could be a ZigBee Status message
      
      RXBuffer = RXBuffer.MidB(PacketLen+5) // Remove the handled packet from the buffer, leaving anything that might have come in since.

If Packet.FrameType = FrameType_ATresponse Or (Packet.FrameType = FrameType_ReceivePacket and Packet.Command = CMD_WirelessAT) Or Packet.FrameType = FrameType_RemoteATresponse Then // Not a ZigBee message, we need to handle it
  
  // Handle the message
     RaiseEvent PacketAvailable(Packet)

In my system, all received messages are in response to commands transmitted from the host Xojo app, and RXBuffer is cleared every time a new message is sent, because I never send another message to the device until I get a response to the previous one. The above code will need some tweaking if you expect to receive messages at random from the XBee, as it looks like it assumes a cleared RXBuffer.

1 Like
10

@Julia_Truchsess you shouldn’t set the encoding on the ReadAll because this is binary data and encodings only apply to strings that are meant to be human-readable. In this case, a Nil encoding is preferable since the String is just acting like a bucket of bytes.

Unless I’m mistaken, the SerialConnection.DataReceived event is on the main thread, but your idea of using a Timer to keep the data flowing is a good one.

I do see a potential bug. Let’s say RXBuffer contains two packets. You process the first one and leave the second in RXBuffer. What happens if no more data comes in from the device? Since the Timer is in Single mode and is only started when data is received, that second packet would be left unattended, no?

11

Thanks, @Kem_Tekinay, the encoding setting was probably a vestige of some random debugging attempts at a point when things weren’t working. Good to know it can/should be removed.

It’s a mystery. All I know is that doing a lot of parsing or handling work in the DataReceived event invariably leads to lost bytes.

Definitely, that’s why I added the final paragraph to my post about it needing tweaking if random incoming messages are expected. My system only ever expects a response after sending a request for one, so it clears the buffer before every request. Ideally I guess you’d have a While-Wend to keep handling packets remaining in the buffer until it’s empty, but in my system that’s unnecessary.

1 Like
12

I think there was/is a bug where DataAvailable wouldn’t fire if you’re already in DataAvailable (or DataReceived), so the Timer lets your app deal with it during otherwise idle time.

Or I’m wrong and it’s something else entirely, but either way, I like and endorse the idea.

1 Like
13

Thank you for the feedback and guidance. And, yes, in my project, an external XBee (one of many) may send a spontaneous message to the master (that is connected to Xojo).

14

So then you’d probably want a loop in ParseRX that keeps handling packets and clearing them until there’s nothing left in the buffer, or just clear the buffer in ParseRX after handling each packet, since XBee packets generally come in at relatively long intervals.

15

Agreed on the timing – generally lots of breathing room between packets. One of the things we want to do is an ND scan to find our devices (we use the NI field to specify our device) so that we can inventory what’s in the field, and use 64-bit transmit messages.

Again, thanks for your feedback. I’ve been dealing with XBees using a microcontroller (Propeller P1/P2) for a while and everything works nicely. Moving to the desktop environment is presenting new challenges.

16

Same here - my primary work with XBees has been with microcontrollers in endpoints, but I’ve also written a PC utility in Xojo for device configuration.

17

Jon, if most of your work has been in the microcontroller world, then it may help to think of Xojo’s event driven code as being structured very similar to microcontroller interrupt driven code.

1 Like
18

i usually collect the input data.
then its searching for a start and end, checksum test.
cut this part and remove from buffer.
if chesksum is ok process it direct or memory it in a list.
for a own system with important data you would send a receipt feedback that nothing get lost. instead of microcontroller sending nonstop data maybe send a request if possible.
because faulty serial transfer,
your bundle can missing the start or end
have wrong data
have wrong checksum.

what kind of device is this where xojo is running?