I have an application (compiling for Intel Mac, Linux ARM, and Intel Windows) that sends a serial command to an external piece of hardware and receives data via serial.dataReceived. The code in dataReceived is simple and size ranges from a few hundred bytes to 1 million bytes with a baud rate of 921600 in an 8N1 configuration.
rawdata = rawdata + me.ReadAll(Encodings.ASCII)
Once the data is received, I clear out rawdata and loop. The issue is that on occasion, the expected number of bytes will not be received during the loop. This happens regardless of platform and tends to be worse if I interact with the gui (clicking, dragging, keyboard pecks, etc.). As a band-aid, I watch the number of bytes and if < expected, I quickly close the serial port, re-open, and ask for the bytes again. This works, but I would like to build a more robust solution.
I would expect for serial.dataRecevied to have a very high priority, but maybe it does not.
I have a ‘GETDATA’ command on the hardware, when I send the command, it should dump the entire contents over, which is sitting in a buffer on the hardware. It normally works, but sometimes it stops midway. For example, I might expect 160000 bytes and it sends 100000 and ends. However, if I close and re-open the serial port, it continues just fine with the last saved contents piping over. If I use a terminal app, it seems to always come over, but in Xojo, the data seems to be disrupted on occasion. Thanks
The hardware sends a few bytes when the data is ready to be read. Once I send ‘GETDATA’, it is supposed to stream the entire content. The hardware retains the data in physical memory, therefore, when it fails to read the entire string and I close/open the port, it starts working on the next GETDATA cmd that I send. Not sure if you would classify this stream of data as a ‘flush’ or not in serial-speak, but it would seem logical. In terms of communications, the unit is very basic and without sophisticated handshaking, checksums, etc. I only know the expected number of bytes. I typically do not receive errors via serial.error. Thanks
It does seem to me that there’s no guarantee what you are trying to do will work. If the receiving side has no means to slow down the sender, then you may lose data and not always know it.
What about the transfer rate - can you slow that down and see if that fixes the issue (or reduces its likelihood)?
I wonder if the DataReceived event is guaranteed to fire again for any new data that is received on the serial port during the time your code is already running inside the DataRecieved event?
As a test, you might loop inside the DataRecieved event until BytesAvailable = 0. Something like:
While me.BytesAvailable > 0
rawdata = rawdata + me.readall()
wend
Tim, I can’t change the baud rate if that is what you mean. It is fixed at 921600. The good news is I am not losing any data, if I don’t receive the correct number of prescribed bytes, then I am either a) asking for the data again, or b) closing and reopening the port and asking for the data again. In case b, the data is still there because it is stored in physical memory on the hardware. So, what I have in place in terms of a reset method works, but not elegant in terms of a consistent cadence of drawing data in my app. I am struggling to understand if this is an issue on the hardware, Xojo, me, combination, or other. I FEEL the hardware is OK since the issue is made worse by interacting with the gui. Thanks-
Chay, I tried your variation and had the same result. I did add a flag to measure how many times dataAvailable fires during an acquisition. For a 160KB transfer, dataAvailable fires 124 times (~1296 per firing).
It sounds like there is flow control but Xojo isn’t using the right one. I can’t believe that there is no flow control and you are transferring an entire megabyte over the link. I doubt the serial port buffer runs into the megabyte range.
You cannot loop - your loop will monopolize the CPU and the Serial Port won’t be able to receive data. You have to make it all event-driven. This works best for me:
DataAvailable (or whatever it’s called now) event appends to buffer and fires a one-shot timer (or maybe Timer.CallLater)
Timer Action event handler parses what’s in the buffer.
A recipe for not working reliably, or not at all, in my experience.
The dataReceived event is not guaranteed to send all bytes, even if the hardware device did send every byte in one write.
You should buffer to your buffer using ReadAll (this removes the data from the class buffer) the call a method* in a timer** (low period say 10ms) that checks if you have a full packet. If not then call Poll on the class. If you have your packet, remove it from the bufffer you created.
in this method try (top of the method)
Pragma BackgroundTasks False
**Alternatively you can do Timer.CallLater(0, AddressOf ASharedMethod)
This calls the ASharedMethod the next time main loop is run, so it does this ASAP.
serial.dataAvailable is the built-in event handler that should (in my mind) be the most efficient way to collect serial data. With regards to the loop, understood and I am not looping to get the data per se, I am reading all of the available data, then requesting another set of data be presented and looping that routine.
Hi Derk, 921600; this is a relatively common rate for higher serial speeds. To reiterate the issue:
~98% of the time… rawdata = rawdata + me.readall() in Serial.DataAvailable works fine for collecting the expected data, which can be between 100 and 1M bytes.
~2% of the time… readall() stops before the expected number of bytes have been collected. I have to either ask for the data again, or, close the port and re-open and ask for the data again. The data is never lost on the hardware, it’s a matter of porting to the cpu with high accuracy.
Your suggestions are very close to what I am doing except for using poll. Also thanks for sharing ‘Pragma BackgroundTasks False’. I have been looking for this! UNFORTUNATELY, it has no impact.
rawdata = rawdata + me.ReadAll(Encodings.ASCII) 'in serial.dataAvailable; appends new bytes to the public string rawdata
A 50ms timer measures the length of rawdata so when it’s size = expected, then we move on. The timer is not turned on by serial.dataAvailable, but ahead of time.
I am very intrigued regarding your comment: The dataReceived event is not guaranteed to send all bytes, even if the hardware device did send every byte in one write.
Why might this be? I think understanding this might be helpful in creating a solution.