Serial Best Practices

Hello all.

I have code that works most all of the time, especially when receiving relatively small amounts of data. However when the data expected is larger, it fails to work with bytesavailable not increasing over time.

My question is maybe my process is wrong.

What I am doing is sending a poll to a remote device from a method. That remote device responds which is picked up by the dataavailable event. Within the data available event, so long as a minimum number of bytes have been received, that short string is decoded to find out how many bytes to expect. This is done due to the protocol that the remote devices use - there is a carriage return but no end of line.

Within the DataAvailable event code I am looking and checking BytesAvailable. I am looking for the BytesAvailable to be equal to what I am expecting, but it does not increase. I am wondering if this is because I am in the DataAvailable code and need to exit for more data to become available?

Should I really be doing much code at all inside of the DataAvailable event? Or should I be treating this like a micro controller and keep code in these events to a minimum? What is best practice?

Currently I am not reading the data to a local buffer until after all of the data has been received. Is this correct or is there a better practice?

Has anyone got a sample they would be willing to share? The Xojo sample is very simplistic and really not representative of what I need to do.

Thanks,
Tim

Hello,
maybe it helps, if the data you receive begins and ends with a special character, so your app knows where a chunk of data begins and ends. For example you can use “{” for the start and “}” for the end. So for example your data received would look like this: {345.78/12.57/123.34}
When I am receiving data in this format from a microcontroller I have this code in the “DataAvailable” event of the SerialController:

If InStr(Me.LookAhead(Encodings.ASCII), "}") > 0 Then dim TempString as String = Me.ReadAll(Encodings.ASCII) ParseData(TempString) End If

The ParseData method then parses the string. The string sent by the microcontroller ends with a curly bracket “}”. The ParseData method is only called when the SerialController received “}” so it is sure that the whole string has arrived.

Hi Tim

if you are not seeing the bytes available increase then that would suggest that the data is not getting to the computer for whatever reason.

please forgive me if the following is already known to you.

I do not use API2 as for some unknown reason the serial port has be renamed and I am not interested in changing everything I know about API1, event names are API1 only in the text below.

one mistake I made at the start was expecting the DataAvailable event to know (a) when the data I want starts and (b) it would know when the data I want stops.
well believe me, its no idea about your data at all!

the event will fire when it has received data and the event is magically told to fire by the OS, thats not likely to be when your data has stopped arriving, it is just as likely to be in the stop bit phase of a random point in your data.
the event is not fired when your data stops sending, how could the event know that?

the method I use recently is to have a timer set at 100ms which simply has Serial.Poll in the Action event.
this will make the DataAvailable event fire every 100ms where you can then test if all the data is in there.
as suggested above, if you are able, use an end of packet identifier and search for it with LookAhead, if found then ReadAll the data and the buffer is cleared which should be the data you want (this is what I do).

if you want to count bytes then you need to be sure there are none in there that you do not know about or you may see issues that are not actually there, noise on the line is just as valid data to the computer as expected data you send.

Read and ReadAll empty the buffer of data, LookAhead looks at the buffer data but leaves it in there.

my question would be more about the transport system between sender and receiver.
serial 232 in single ended and as such is very prone to electrical noise messing the data up, if you are running over greater than a couple of meters then this could be an issue if you are not using high quality screened cable, above 5m then a standard cable is much more likely to have issues.

with genuine serial data (such as a MAX232 chip) that is ±12v, a 24v differential, you might bet 10 or 15m reliably.
the standard USB to serial are usually running at 0 to 5v at best, only intended to work over the lending of cable supplied.

then there is the question of baud rate, the faster you go the shorter the reliable distance.
if I were testing your system I would try at 9600 baud first to see if the data is getting through.
if you are using 57600 and above then I am afraid its going to be far more unreliable.
those speeds may be ok over a very short path.

supposing all that is ok and you still have problems then I would download CoolTerm (which was made with Xojo) and test the hardware using that very good application.

if it works on that then its your code, if it does not its the hardware.

the Xojo serial class is used in all my projects in the last few years and it is as reliable as any class I ever used, any issues have either been me being a bit stupid in code or, bad hardware feeding the data to the receiver.

good luck, I have no doubt you will get it working in no time.

Mark

Hello Mark,
Where do you get coolterm?

The 485 is optically isolated and runs at 9600 bps. There is no issue with communication, or noise, especially in the test setup where the cable is less than 1 meter. Further the system works great using older code created in VB6. This is not a communication issue but a process problem with how I implemented in Xojo. BTW, this scheme works perfectly with shorter data streams. It only got messy with longer streams.

@Chiristian - I cannot change the protocol of the remote devices. The initial 7 bytes includes a message length so that I know when the last char has arrive, assuming of course no noise on the line.

I’ll try your timer suggestion and see how that works out.

Thanks,
Tim

You might want to check this thread. I put some code of mine from actual project: https://forum.xojo.com/56034-serial-readall-does-not-read-all

Bottom line is that DataAvailable event doesn’t guarantee that everything has been received, nor if you have multiple messages received so you have to do some work to figure that out.

ok, well you answered all the questions as to where the problem might be and it certainly looks like your Xojo implementation.

a quick google gives this for CoolTerm:-

https://freeware.the-meiers.org

you mention 485, am I right in thinking you are using RS485?
I use 485 in many projects using a single pair for both TX/RX data which obviously needs manual control of the bus, this is where I had one issue, not leaving the bus in TX long enough for the data to physically get down the wires before going back to RX.

the ‘issue’ you mention about longer packets is that you are not managing the incoming data properly.

a short packet will allow you not to worry about where and when the data starts and stops.
if the protocol has no start or stop delimiter then you are stuck with counting bytes and hoping there is no corruption along the way(which seems unlikely but always possible).

as Bob shows, that thread will help in sorting out what you want to do.

if you are counting bytes and have this 7 byte header to work with then I think I would use LookAhead to determine when that portion has been seen and then decode the length value, then look at BytesAvailable and when that value exceeds the number set in the length use ‘MyRXString=Read(length value)’ to remove the whole packet from the buffer, any left in the buffer may be from the next message which you can come back later and test for.

if you use the Poll method then you do not simply wait until the OS fires the DataAvailable event.
I don’t know how big the packets might be, but if its a byte length then you are limited to 255 (plus the 7 in the header) so at 9600 you will need about 266 milliseconds to transmit a whole message (supposing 1 start/stop bit) which would mean if you polled the serial port every 500ms you would be very likely to get all the data from a packet in the serial buffer.
that might be a start point to fine tune what you actually need.

it is really simple to do when you realize the data can not manage its self and you must do so.

Thanks all.
@Bob Keeny - where in your project is the code you provided

while oSerial.BytesAvailable > 0
  
  dim s as string = oSerial.ReadAll
  sSerialBuffer = sSerialBuffer + s
  oSerial.poll
wend

//Now do something with data

Is it in the DataAvailable Event or elsewhere? Timer?

I guess one good question is where to loop looking at BytesAvailable? I was depending on the event firing multiple times - as necessary - until the full packet arrived. Obviously it does not know what a full packet it, but the assumption was that it would stop firing when all data had been received. I also use a Timeout timer so my code does not get stuck waiting. But alas these methods do not seem to work properly with bigger incoming data…

Or -is it better to use the .poll method as Bob uses in his code? If so, where to put it?

Finally, is it best practice to do work while in the event? Or should that be left pretty much as a notifier?

@Mark, the most time was calculated at almost 500mS to receive the largest amount of data. Considerable time over the more standard data streams of only 10 bytes +/-.

Tim

DataAvailable Event.

You have to assume that the event could fire multiple times or just once. It really depends on how big the data is. In our app the packets are small and they have a very specific ending byte pattern so I check to see if I have multiple packets of information when I’m all said and done. If I have something left over I keep it around until either timeout or I get some more data.

Hi Bob

Thanks for your response -

There should never be more than one packet at a time, since the remotes are polled and they only respond when asked to. The ending byte is a carriage return and is not the best to use since data within the packet can be the same as a cr. But, the protocol allows for knowing how many parts are in the packet and how long they are. That part is easy.

The harder part right now is getting the data available.
I think I am getting closer - finally!

Thanks all,
Tim

the only difference between Bob code and how I do it (from a logic point of view) is that I have a timer of 100ms period which calls Serial.Poll.

once the DataAvailable event knows the data is complete (my case with an end of packet character, yours with byte length) then call a method to process the data.

can you post your code to see if there are any little things that might be slightly off?

Hi Mark,

I’m getting closer… once I get further, I can post - others can use it too for future reference.
Tim