You might want to put this process into a thread, turn off background events so it’s not interrupted, and Poll the port in a tight loop until you get size = expected, then fire your timer to process it and exit. When you need the next round of data, Run the thread again.
The reason I suggest this is it sounds like the main event loop - processing the timer and user intereaction - is interfering with receiving the data.
You have already dealt with this. What Dirk means is that the machine may write 160KB in one go, but you won’t receive it in one DataAvailable event. You mentioned earlier that you receive 160KB in 124 DataAvailable events. So this isn’t anything new to you.
Yes, DataAvailable is where you collect the data, i.e. append it to your buffer. But you want to spend as little time in that event as possible, hence the timer that both I and @DerkJ have recommended to offload the data parsing to code that’s not in the event handler. I (and seemingly no one else) cannot tell you exactly why this is so, but if you put too much code in the event handler you will miss bytes. I have done dozens of projects using Xojo serial ports (albeit not at 921600 baud).
If you mean that you request data, return to the main event loop, and eventually respond to an event that tells you that you’ve received a message, then fine. If you mean that you literally sit in a loop waiting for data then you will miss bytes.
Don’t do it that way.
Socket DataAvailable Events (including Serial) always fire on the main thread and will interrupt anything else that’s running. Using the pragma for disabling background tasks is a great way to actually miss incoming data events.
I do have another suggestion for the incoming data however that will be just a bit faster…
First, I suggest creating a subclass of Serial and add two private properties:
mBuffer() as String
mProcessTimer as Timer
Add a Private method called ProcessData(t as Timer) which contains this code at the top
Dim data as string = join(mBuffer, "")
And then check the length and process as necessary.
If you need an event of some sort, add a new Event Definition (let’s call it DataReceived) and once you’re done processing, raise the event:
RaiseEvent DataReceived
In the DataAvailable event do this:
Buffer.Append me.ReadAll
// initialize the timer if necessary
If mProcessTimer = Nil then
mProcessTimer = new Timer
mProcessTimer.RunMode = Timer.RunModes.Single
mProcessTimer.Period = 10
AddHandler mProcessTimer.Action, AddressOf ProcessData
End If
// reset the timer back to zero
mProcessTimer.Reset
As long as data is coming in, it should prevent that timer from firing. You may need to increase the period by a little bit if it does fire too many times for your liking or if there are documented delays.
Is that so? I’ve actually never lost data. Could it be that the buffer has the data anyway the next dataReceived event? Maybe it’s about the speed of transmission… so perhaps don’t use the pragma.
What is hard to understand actually is, this kind of topic comes up every x weeks or so. Some issue different customer. Why doesn’t xojo have a solution for this build-in? It just makes me wonder.
Back on topic:
So to learn from this, in Xojo dataReeceived or dataAvailable events may be an interrupting event so it will stop other code when this this is called. This causes issues if handled incorrectly that’s what the timer is used for.
Try to not loop too long in your data process method, try to check for a packet, extact the packet and remove the packet from your buffer, check for next packet. If there are no packets, exit the looping and return. If done well, this keeps your buffer and fills it using the event data (see @Greg_O 's comment with the class). Basicly you want to have as little as possible loops in your data processing method called by the timer.
I stand corrected about using a thread and blocking. I was obviously thinking of some other scenario.
There are no “interrupting events”. Events are called in sequence when the event loop executes. One event will not interrupt another unless you use DoEvents. That’s why a loop in one event can block other events.
Absolutely… assuming there is a next DataAvailable event.
You’re right of course. I explain this technique quite often as it was explained to me by other ex-Xojo engineers.
This isn’t the solution for every case. This technique, while stable, is rather complicated and because of the timer and how sockets work makes it feel very unpredictable and complicated for beginning users.
There was a trio of classes however, EasyTCPSocket, EasyUDPSocket and AutoDiscovery which attempted to hide some of this from the users in the old framework. They were a pain to maintain and as someone who used and abandoned them, they’re just not scalable. Basically, great for demos but not for long term use.
To be clear, the reason that events can “interrupt” others is that DoEvents isn’t the only time that things can switch. It also occurs on loop boundaries, so the idea of using a for-next or a while loop inside a DataAvailable event is particularly bad because you get what’s called reentrancy.
Basically, because you can’t know when the DataAvailable event fires whether or not you have all of the incoming data, the tendency is to have a property meant for buffering the progress or output of the processing. If a second DataAvailable event were to fire while the current one were still in progress, the two events could be writing to the output buffer at the same time, interleaving or clobbering the data from the other without knowing it. The more events that fire, the worse it gets.
Trust me when I say that I come from a place of experience with this. We had a similar problem in web 1 which couldn’t be fixed until web 2 when the backend was rewritten for http/1.1 compatibility.
Which is what I suggested 10 posts up from yours - append data in DataReceived and use a timer to call the parsing code.
Thanks Everyone for your feedback and suggestions.
Julia, to be clear- I am not engaging in any post processing activity at present other than collecting the data out of the buffer. The issue arises during the DataReceived event using a single line of code:
Serial1.DataReceived()
rawdata = rawdata + me.ReadAll(Encodings.ASCII)
End
Only after the string reaches an expected length do I attempt to process the data. For now, I simply have a button that signals the hardware to send the data and the single line above does the collection. That is it- all post-processing is turned off so there are no timers for now. To obtain 160KB, it takes about 2 seconds for a successful collection.
Greg, I followed your suggestion. I typically get an array of ~125 lines averaging roughly 1296B each… and fails in the same manner. Most of the time, it works and then it will not read the entire buffer. Close, then re-open the port and then it works again.
Thanks, Sean
Have you seen any pattern in the incoming data? Like… is there a NULL byte chr(0) right where it stops every time?
Good question, but I don’t think it is. I’ll check closer though.
Hi Greg, I don’t see any patterns in the data where it stops reading.
One interesting test though is running the same hardware and similar code on RB 2012r1.1 without failure thus far. I’ve been running for over 30 minutes without anything coming down. I’m going to run it until it overflows- which should occur within about 2 hours.
Thanks, Sean
@Sean_Pons , are you seeing this particular issue on all platforms you are building for, or just one particular one? Does your hardware include a built-in USB/RS232 converter, or is the adapter a separate piece of hardware that you could exchange for something different? Over the years, I have found that the hardware/driver combination can be iffy, particularly with Windows, but there have been occasions where specific adapter misbehaved on macOS. Swapping over to a different adapter with a different USB/Serial chipset or updating the driver has resolved such issues in almost all cases.
CoolTerm is written in Xojo. You might want to check your hardware with that to see if you see the same issue, if you do, then it’s probably an issue with your hardware rather than your Xojo code.
A few tips for working with serial ports in Xojo (and some if not most of this might be a repeat of the various responses above):
Roger, yes, the issue is present on Mac, Linux, and Windows and it is based on the FTDI 2xx serial port to USB converter. I’ll check CoolTerm and see if I can learn anything.
I only have one line of code in DataReceived and move all incoming data to string (not a memoryblock). Is a memory block more suitable?
how is a 100ms timer different than DataReceived? Do you think it is more reliable?
Thanks, Sean
Doing the heavy code lifting in a timer event instead of the DataReceived event prevents the DataReceived event from being occupied unnecessarily since it can be reentrant.
I have my local buffer set up as a circular buffer with a configurable size. A memory block is much more suitable for something like this. I also found it to be faster when dealing with larger amounts of data. My use case might be different than yours. I keep the data in the buffer so that I can refresh the display with what’s in the buffer. If your local buffer only needs to store relatively small amounts of data, then a string should be fine.