ive started have a very difficult to duplicate problem since Ive moved my code to Cocoa. It seems the cocoa main thread is re-entrant in a way that it wasnt before. I have an app that runs applescripts in response to data that comes in over sockets. If that script is calling out to an external program and waiting for a response I am now getting data available events while that is outstanding. This should not be happening, and did not happen prior to cocoa. This app is not multi-threaded. The script is run from the main thread and the data available events MUST be firing into the main thread, at least they always have done so before.
This is alarming as it turns a nice steady linear program into a mess of queues and semaphores when there are no threads.
Am I crazy in thinking this is happening? Or has cocoa changed things on such a basic level? Has anybody noticed anything similar or know of any apple docs that talk about this new cocoa feature of being able to reenter the main thread for events while an apple event is waiting?
I dont think this is a Xojo bug, I think its a change in philosophy in cocoa but Im not sure whats going on.
The app doesnt work so well when you receive another command while its literally in the middle of handling the last one. Those socket events should wait till my other code returns shouldnt they? I am not calling doEvents or thread sleep or anything like that anywhere in the app.
It’s not a bug. In your DataAvailable event, all you should be doing is copying data to a buffer. Then use a timer or other mechanism to pull data from the buffer.
So its a known factor that the data available event can run out of proper place of the rest of the program? ive been developing in xojo since it was CrossBasic, 1994 or so And Ive never heard that. Socket events fire into the main thread is what its always been. Is this new since cocoa then? I can certainly rework things to do that, but wow. Thats a major hidden gotcha that needs to be somewhere more obvious in the docs.
and if thats the case, whats to keep the data available even from firing again while Im still appending data to the buffer in the last event? This seems like a significant detriment to getting anything truly deterministic and linear done with the data? I might end up with things out of order in my own buffer.
and whats to keep another event from adding to the buffer just as my timer callback is removing a command from the front of the buffer? Can I wrap the access in semaphores? Or since its the same main thread only in some strange re-entrant manner would that blow things up? Im a little bit baffled by this the more I think about it. My main thread can get interrupted at any time with more data now? But then even more processing of the buffer can get interrupted.
This is giving me a serious headache. Can you explain more about what exactly this means and how I can reliably and properly handle it the way I used to?
The timer has to be on the same thread of course - in the socket subclass. Then in the timer’s action event you pull out as much as you need from the buffer and process it.
and why on earth do I have to create ANOTHER event with the data? Isnt that what the xojo framework should do? It should buffer my data and then present it to me on the main thread as it has always done. Why am I having to manage another buffer layer on top of all the other ones under there?
Eli, timers should always fire into the main thread, there isnt any way to make them fire into any other as far as I know. You can create a new thread from the action event but the action event happens in the main thread.
Which is how the data available events always worked up until cocoa too. But there is absolutely nothing to say that the main thread wont be interrupted just as Im starting to access the buffer with another data available event in this context. If the events can happen at any time at all yet somehow in the main thread while interrupting it, then there is absolutely no way to protect the buffer that I am using from extra accesses to it. If thats actually true, and please tell me if its not and WHY its not, then this is a huge blow to making anything long term reliable that deals with sockets isnt it? OK, got a buffer set a timer, start to remove bytes from the front of the buffer, OH, but heres some more data to stick onto the end, all your offsets are now potentially wrong as the 2 accesses to the same string resource run all over each other.
James… With really kind attitude, your message content is perfect and I agree DataAvailable Event needs more info from Xojo. Just when hit a bug or a change ( a barrier ) … In spanish there is a say: Don’t kill the messenger. Eli is helping ( he’s a top notch forum member ) and not Xojo. I am beginning to have a nice collection of broken keyboards thrown
As usual could not edit my post… (glad I’m on a laptop so keyboard will be safe )
Could you please explain me a bit more in-depth what reentrant code can affect Xojo. I studies it in the 90s, with MS-DOS but was something I never understood correctly. Would appreciate if you describe more in depth your worries. Just to add to my programming notes
Truly I am not unhappy with Eli! He is terrific and I appreciate the help! My paragraph to him above wasnt negative I hope, I am sorry if it was, I did not mean for it to be. All the other paragraphs were angry, but they were not at him, they were just me yelling at a system function I used to think I understood which is now very unclear.
I am inferring a lot from Gregs short answer to the question I know. But if its necessary to treat the data available event as an interrupt now, that has all sorts of implications doesnt it? If thats the case then the solution cannot be as simple as simply pushing my processing of the buffer into a timer event because that could get interrupted too. Either that or I simply do not understand what is actually happening which is the most likely answer
I understand your angriness were not forwarded to Eli. I’m actually doing with ServerSocket/TCPSockets and seeing the same behavior I did not expect, after having most of my basement code written and tested.
In my case I will be using Linux, but developing Xojo on a Linux VM … and having to test with remote debugger is frankly, exhausting. I wonder if this problem can affect Linux as I am developing on my mac and hoping it will work as it should (DataAvailable event) on Linux. You think this can be only a Cocoa issue or have done some Linux tests?
I see there was more from Eli that I didnt answer as I was too busy ranting I said:
“So its a known factor that the data available event can run out of proper place of the rest of the program?”
My observation of the problem Im having is this. I receive a packet in the data available event and from the event, if I have a whole packet, I start a potentially lengthy response to it. There can be a lot of code triggered from the reception of a packet. What Im seeing is that I get another data available event in the middle of that previous code, interrupting it. When that new packet is complete control returns to my original code which finishes, but now events have happened out of order. The second received packet that was received in the middle of processing the first one has finished first even though it was received second. Not to mention the chaos that is caused by all the variables setup for the first packet are no longer correct half way through processing it. My initial thought that it was related to sending apple events I dont believe has anything to do with it given Gregs answer.
This is what makes me think the code can run out of the proper place previously in xojo I would get another data available event only once my own code from handling the last one had returned and was finished. But now my code can be interrupted at any point with another event. Interrupted, put on pause and the main thread or whatever re-enters with the new data.
then I said “and if thats the case, whats to keep the data available even from firing again while Im still appending data to the buffer in the last event?”
And it may be the same thread, but it doesnt wait for the currently running code to complete. It interrupts me in the middle of my code. If I was programming on an arduino with interrupts I would expect this behavior and its behavior is documented enough that I can still get a deterministic output while processing interrupts. But if xojo can just stop the execution of my code and call back into the data available event at any time, then it can also call back into the processing I might delay by use of a timer as Greg suggested. I can no longer treat that buffer string I would create as he suggests as immutable while Im using it. What happens if Im pulling a packet off the front of the string and re-assigning whats left to a new string to put in the buffer at the moment Im interrupted. Where does the new data go? I would lose it in that case as it would be appended to the old string which I already took some packet off of and when it was done I would re-assign what was left to a new string, loosing what had been added in the meantime by the interrupt. Thats an overly complex example, but its exactly the same as if 2 threads were working on an unprotected resource at the same time. You cant do that without semaphores and critical sections. But if this is re-entering the same main thread then I cant use those because I cant pause the main thread until the main thread completes Thats just silly.
All that lengthy, what if, chatting Ive sat here and inferred from Gregs 2 sentences telling me how to handle it. I need more info than just move your processing to a timer because I dont believe thats the fix without more information. Please explain why its different on cocoa and why my fears on why thats not a fix dont apply Then I will be happy. Need more than 2 sentences.
You’re going to find this behavior on Linux too. One way to handle this is to use a binary stream backed by a memoryblock. When DataAvailable fires, simply call:
bs.write me.ReadAll
If you want to take data off the front, just remember the position, copy the data you’re keeping in memory to another memoryblock and hook up the binary stream again, with the position set to the old position - the number of bytes you removed.
I’ve also done this with arrays of strings, but you’ve got to be very careful about encodings that way.
nope, I still dont understand. Please explain why this is suddenly necessary. Ive been treating data available events in this same program as every other callback into the main thread since OSX version 10.1 and the first bug report Ive had of this causing any trouble was day before yesterday. I can now duplicate this problem, I cannot duplicate it with the pre-cocoa version. This is something new. we need examples on how to properly manage this.
If I move to a binary stream reading from a memory block for the buffer, how do I properly clear the buffer or reset it without being in danger of having a re-entrant event put more data into the old one at the moment Im switching references?
Reading your first comment again, it strikes me that another thing that may have changed (I’m just hypothesizing here) is that apple events may simply not block sockets in Cocoa.
I’ve evaluated a lot of DataAvailable bug reports since I started here and it’s not uncommon to see people trying to do some sort of time consuming processing of the data coming in through a socket and getting themselves into trouble like this.