I am trying to get messages (call details records) from a PBX system
For that, I am testin a serial port monitor I’ve found in this forum, with this simeple code on DataAvailable’s event of a Serial object
Dim s as String
s=Serial1.ReadAll()
TextAreaRs232Rx.AppendText(s)
The problem is the last (or last two, not sure) character(s) of the returned data are missing.
I am sure at least one byte is missing because the last column should be a time formatted column and I get something like HH:MM:S
I don’t know if besides that I am also missing an end of line (which probably should be present as well).
I tried some more complex alternatives of code that I’ve seen in the forum with the same results…
So, in my line of thought there are two alternatives
It’s a buffering issue. Xojo is not capturing all the data in the buffer or the connected PBX is simply not sending all the data at one
it’s an issue with the serial port configuration. (Currently is 8 Data bita, No parity, 1 stop bit, XON, CTS, DTR) ?
Any Ideas where should I start looking ? It’s hard to track for me…
You are not guaranteed when or how many bytes are in the buffer at the time the DataAvailable event when it fires. It’s possible the event is firing at first character and you ReadAll the characters before the last character arrives.
If you know you are getting a fixed-length message the solution can be corrected by reading the buffer in a loop until you reach the number of characters expected. You can also use LookAhead to read the quantity of bytes in the buffer prior to doing the ReadAll. But, I like the first method better because once you readall it will retrigger the DataAvailable event if new stuff arrives. It’s easy to keep all the code to detect complete messages that are received in the DataAvailable event. If you receive a complete message you do something, if not, store the data and append following messages.
If they are not fixed length and you are making the request for data just make sure you delay your next request until all of the characters have been received from prior request.
@ : I’ll try your suggestion and let you know how it goes…
@Kem Tekinay : The Serial instance was dropped in the main window, so it should not go out of scope…
@Joseph Evert : I have no idea of the length of the data. By "just make sure you delay your next request "… you mean something like fire a timer ? (oh… I just realized you said ‘your next request’. but I am not requesting data… I am just catching an event handler)
If you have no idea of the length of the data, do you mean there may be several messages to be read, or that different messages have different lengths?
When I was reading call detail records from a PBX (this was in the 80s), using a microVAX, I configured the PBX to terminate each record with a special character (a control char, DC1 or DC3, can’t remember which now). That will help you know when a record is complete. But perhaps you already have a well-defined line-end character.
In my app I need to read lines of data from a remote host, and I have a method that returns one line or times out. I do this with readAll, and if I don’t get the defined line terminator then I start a timer for the timeout period (30 secs in my case) and then suspend the thread. The event handlers for the timer and DataAvailable resume the thread so I can tell whether there’s data available or not. I use a thread because I don’t want the app’s GUI to lock up.
Regarding the data request; are you sending a string to the device requesting it send data back? Is the communication bi-directional?
Or, is it just streaming data at all times? If it is this, you will need some way to delineate the messages like Tim is discussing above. That should be your indicator when a full message has been received.
Typically my method of choice is just to readall the data every time the event fires appending it to a string with the proper scope so it retains between reads. Then after I append, I string search the string to look for complete messages, when I find one I pull it out, leaving the balance (because that may be the beginning of a new partial message) and process the complete message. If there is no complete message I simply do nothing waiting for the next DataAvailable event to fire again.
@Tim Streater : It’s just a stream that’s being output from the device every couple of minutes (when a call is made). No idea of the length
@Joseph Evert : I am not sending data… data is spit from the hardware… I am trying to collect it (and parse it)
Looking at the output from the PBX looks like the PBX is configured to terminate each row with a 0D (chr (13))… So I might try adding this before the rest of the code, so it will not get any unterminated data from the buffer… could it work ?
[code]Dim buffer as String = me.LookAhead(Encodings.ASCII)
Dim LastByte as integer = AscB(RightB(buffer,1 ))
if LastByte <>13 then // chr(13)
Return
end[/code]
That looks like it might work, but you need to then grab the buffer once you have detected the Chr(13), but only from the Chr(13) backwards, realizing more characters may have come in - ReadAll would take the few characters from the new message after the Chr(13) if two calls were made/terminated within milli-seconds or at the same time as each other.
Are you sure the messages are not structured? It seams like they would have to be if your reading specific parameters. Can you show us what these messages actually look like? That would be helpful.
@Joseph E : “but you need to then grab the buffer once you have detected the Chr(13), but only from the Chr(13) backwards”. That is correct but what if I keep skipping the read until I find a 0D at the end ? (would it not just keep coming in the buffer until i ReadAll ?)
Here’s how the message looks like… (I don’t know WHEN do headers come… ) In this I manually added the missing character and the missing endofline.
Date Time Ext CO Dial Number Ring Duration Acc code CD
Yes, that is correct. The concern is if the NEXT message shows up in the buffer and you detect the line ending CHR(13) For Example, consider the buffer contents look like this:
This is one complete message, with another partial message coming in AT THE time the DataAvailable event fires. You want to take from the CHR(13) back, leave everything after the CHR13 as it is part of the next incomplete message. If you do a readall you will wipe out the entire string in the buffer, your next message will be incomplete and lost.
28/06/18 09:55AM 100 03 46191090 00:00’19
So it does look like the message is fixed length, 41 Bytes + the CHR(13)? That’s good! Know you can find the CHR 13 then take just the preceding 41 bytes and that is a complete message. You can do this using Read(42) then strip off the CHR13. I personally would create a string property on the window then when I see DataAvailable I append it to that string and parse out the messages from there. keeping parts you want to keep. Remember that DataAvailable can fire at any time and doesn’t care when or what is in the buffer - it’s saying there is “some” data available.
@Joseph Evert : ‘If you do a readall you will wipe out the entire string in the buffer, your next message will be incomplete and lost.’.
I would ONLY do a ReadAll IF the last byte in the buffer is a CHR13, which is not… Until I get a complete message with a CHR(13) at the end, I don’t read anything…
Looking at the message in the binary tab in Xojo looks like every line (except the ‘header’) is 81 bytes, which makes some sense…
So it would be actually 'finde the CHR 13, and then the previous 80 bytes… but that only if it’s not the header… Is it not easier ‘Find the CHR 13, and get everything before it, leaving the rest in the buffer’ ?
But how do you know you have a message with a chr(13) waiting to be read? You should do a readAll if you get a DataAvailable event. Then scan what you have, and if there is a chr(13) then you have a message you can process. Any bytes you have left after the complete message, you have to store until another DataAvailable event arrives, which will give you more of the next message.
Exactly what Tim said. I would not assume the Chr(13) will always be the last character in the buffer. In most cases it might be, but its possible you may get stuck in an endless loop waiting for it to be the LAST character. Consider the following sequence:
the following data arrives and the event fires, you look at the buffer and see this:
28/06/18 09:55AM 100 03 46
it’s a PARTIAL message, the Chr(13) hasn’t arrived yet. You do nothing on that event. Now the rest of the FIRST message comes along with a new message - your buffer looks like this:
Its the rest of the FIRST message and you get a Chr(13) and part of the next message (so chr 13 is not the last byte). If you do a READALL, you get your first message and wipe out the first bytes of the second. now you get the balance of the incomplete second message and the Chr(13).
/18 09:55AM 100 03 46191090 00:00’19Chr(13)
Now consider waiting until the Chr(13) is the last character, you wont do anything on the first or second message because Chr(13) is not the last message. But on the third event, you would now have two messages in the buffer and will need to process both.
You do have to do some parsing and string manipulation to get this right and more importantly, reliable. Otherwise you’ll be chasing “bugs” and unreliable data logging.
First of all, I want to thank you all. And leave something clear: If you notice I insist on something it’s not I am stubborn, underestimating your help or trying to be right. I am probably not understanding what you say or not understanding the mechanics of serial data transfer, as I am very inexperienced with serial data transfer.
Now, with that said…
I am confused…
At this point what I was looking for is to either get a full message (or pack of messages) with its terminator at the end OR reject the full buffer (just like Wayne suggested in its link), leaving the data in the buffer without reading it, until the endofline is received. (Not sure this is good)
@Tim Streater : I am not assuming I have a chr(13) at the end of every message (in fact that’s exactly the byte it’s missing), that’s why I thought I could just return from the event handler without any processing in case the last byte is NOT a line terminator, and wait for it.
@Joseph Evert ‘ts possible you may get stuck in an endless loop waiting for it to be the LAST character’. I am not looping any code. I am just not reading-wiping the buffer, and returning nothing from the event handler.
@Brian O’Brien : I am now running a terminal app in the server, connected to the PBX to check exactly what data does the terminal receive. I will double check the line terminator. I am not sure how to check for odd/even length buffer (?)
After all, ANYWAY I am still getting the messages with the last byte missing. I am now running a terminal app to see if it happens here, and exactly how does each line end.
I’ll post the literal results as soon as someone makes a call there
@Roman Varas - If your results don’t yield a different result - Could you post your code on the serial buffer missing the 1 byte? I am a Cisco Voice CCIE with a lot of previous Nortel / Avaya PBX experience perhaps I can help?
