Best way to handle serial packets

General
1

We are interfacing with a microcontroller, and the engineer who is writing the firmware has proposed the following setup for sending and receiving instructions and responses from the micro:

command:length:parameter

Where command is the command to or from the microcontroller, length is the length of the current packet, and parameter is any applicable parameters (probably itself separated by some delimiter if we have to do multiple params.

In the past I’ve done this differently when using serial to communicate with an Arduino, though that was back in the RB days. In that case, we inserted a terminator into the string, to indicate the end of a packet. Something like Command|Parameter; where the semicolon is the terminator.

This is some code someone here gave me:

// the thorn in RB.
// DO NOT ALLOW RE-ENTRANCY
// it will mess with your buffers
static IsProcessing as Boolean = false
if IsProcessing then
  Return
end if

//this will be our input frame
dim message as string

//the lookahead finds the last character in the first message
dim pos As integer = me.LookAhead(encodings.ASCII).InStr(TERMINATOR)
while pos > 0
  
  //engage re-entrancy protection
  IsProcessing = True
  
  //read only up to the frame character
  //  ensures we only process complete messages
  message = me.Read(pos, encodings.ASCII)
  
  //remove the terminator from the end of the string
  message = message.Left(message.Len-1)
  
  //do something with the framed message
  processArduinoOutput(message)
  
  //are there any more complete messages in the serial buffer
  pos = me.LookAhead(encodings.ASCII).InStr(TERMINATOR)
wend

//when we got this far, there are no more complete messages
//a partial message may be in the serial buffer
// but we will get to that when the framing character is delivered
// and the eventhandler is raised again

//disengage re-entrancy protection
IsProcessing = false

This worked well for me in all the tests I tried. Looking at what the firmware developer has suggested, I’m wondering if one method is more reliable than the other. It seems to me to be a lot simpler to just look for a terminator, no?

2

Suppose that for some reason some characters are lost between the first character and the terminator (and you receive the terminator). How would you know something is wrong ?

With the length you know exactly how much character you will receive after the length. Then the length is a good idea. You need to design a strong protocol that will catch most, if not all, the errors. For example, you may state that all commands are 4 characters, upper case, and that all start with AT (AT has been used with modem for a long time.

If you wish to see a good protocol, look for XBee modules. The most interesting one is the API mode protocol. You may create your own protocol after this one.

3

so a hybrid then - where the length acts like a checksum to verify the whole command is there but still using a terminator to indicate the end of the packet. I like that.

1 Like
4

You need to have the length on top of the checksum, if not you don’t know where the checksum is ending - that supposes the checksum is always the same length and the parameter may be of varying length. The terminator may not be a good thing. If the parameter (payload) contains the terminator, you will need to escape it.

Believe me, your protocol needs to be solid.

5

Obviously, this is what we want. I almost wonder if it would be easier to just use fixed sized packets. The data being passed around isn’t a lot so we could come up with some packet size that’s larger than we’d expect we’ll need, and keep everything under that limit.

As for terminators slipping into the packets, it won’t happen. This is a pretty controlled setup with a fairly limited set of commands and return messages, not a general purpose serial communication tool. We just don’t use the semicolons in our messages, and we’re good.

6

You could use 0x0a as the terminator. And possibly TAB 0x09 as a separator.

7

My wife prefers to use scissors, but I am too lazy, so I grab them near the top (at opposing sides) and pull the packet open… Oh wait, you said serial packets, not cereal packets…

3 Likes
8

For wired serial communications (which I think is what you are after), the MODBUS protocol is pretty solid (either RTU or ASCII). What you propose looks good; I would allow for variable size data (otherwise you may outgrow your limit or you may overdesign the limit).

[EDIT]
I wanted to add that you may not want to add the complexity of making your hardware device MODBUS compliant; but their packet structure is certainly worth a look:

Start
Address
Function
Data
CRC
End

9

So in talking about this last night, it seems that TCP and UDP are options on the hardware controller we’re using as well. What, if any, advantages would we see if we were to do our communication over ethernet vs serial?

All communication would be wired, by the way.

10

Perhaps you would get a more stable connection, but this is not accurate, you need to test everything in action and look for the best options.

11

I’ve used both approaches: sending the length without a terminator, and sending a terminator without the length. You can also use length and a terminator, if you’re not trying to save milliseconds. I always use some kind of checksum, and a timeout in case termination requirements are not met. A terminator generally makes it a little easier to parse.

1 Like