in cocoa socket events are firing while apple events are outstanding?

HI folks,

i’ve started have a very difficult to duplicate problem since I’ve moved my code to Cocoa. It seems the cocoa main thread is re-entrant in a way that it wasn’t before. I have an app that runs applescripts in response to data that comes in over sockets. If that script is calling out to an external program and waiting for a response I am now getting data available events while that is outstanding. This should not be happening, and did not happen prior to cocoa. This app is not multi-threaded. The script is run from the main thread and the data available events MUST be firing into the main thread, at least they always have done so before.

This is alarming as it turns a nice steady linear program into a mess of queues and semaphores when there are no threads.

Am I crazy in thinking this is happening? Or has cocoa changed things on such a basic level? Has anybody noticed anything similar or know of any apple docs that talk about this new cocoa feature of being able to reenter the main thread for events while an apple event is waiting?

I dont think this is a Xojo bug, I think it’s a change in philosophy in cocoa but I’m not sure whats going on.

The app doesn’t work so well when you receive another command while it’s literally in the middle of handling the last one. Those socket events should wait till my other code returns shouldn’t they? I am not calling doEvents or thread sleep or anything like that anywhere in the app.

It’s not a bug. In your DataAvailable event, all you should be doing is copying data to a buffer. Then use a timer or other mechanism to pull data from the buffer.

This is something I stumbled upon a few years ago - and it took me some time to find this kind of information.

IMHO this should be mentioned in the docs of DataAvailable event.

So it’s a known factor that the data available event can run out of proper place of the rest of the program? i’ve been developing in xojo since it was CrossBasic, 1994 or so :wink: And I’ve never heard that. Socket events fire into the main thread is what it’s always been. Is this new since cocoa then? I can certainly rework things to do that, but wow. Thats a major hidden gotcha that needs to be somewhere more obvious in the docs.

and if thats the case, whats to keep the data available even from firing again while I’m still appending data to the buffer in the last event? This seems like a significant detriment to getting anything truly deterministic and linear done with the data? I might end up with things out of order in my own buffer.

No, it doesn’t. Why do you think so?

No, it doesn’t. This is just not possible, since the event is always raised in the same thread.

and whats to keep another event from adding to the buffer just as my timer callback is removing a command from the front of the buffer? Can I wrap the access in semaphores? Or since it’s the same main thread only in some strange re-entrant manner would that blow things up? I’m a little bit baffled by this the more I think about it. My main thread can get interrupted at any time with more data now? But then even more processing of the buffer can get interrupted.

This is giving me a serious headache. Can you explain more about what exactly this means and how I can reliably and properly handle it the way I used to?

The timer has to be on the same thread of course - in the socket subclass. Then in the timer’s action event you pull out as much as you need from the buffer and process it.

and why on earth do I have to create ANOTHER event with the data? Isn’t that what the xojo framework should do? It should buffer my data and then present it to me on the main thread as it has always done. Why am I having to manage another buffer layer on top of all the other ones under there?

Eli, timers should always fire into the main thread, there isn’t any way to make them fire into any other as far as I know. You can create a new thread from the action event but the action event happens in the main thread.

Which is how the data available events always worked up until cocoa too. But there is absolutely nothing to say that the main thread wont be interrupted just as I’m starting to access the buffer with another data available event in this context. If the events can happen at any time at all yet somehow in the main thread while interrupting it, then there is absolutely no way to protect the buffer that I am using from extra accesses to it. If thats actually true, and please tell me if it’s not and WHY it’s not, then this is a huge blow to making anything long term reliable that deals with sockets isn’t it? OK, got a buffer set a timer, start to remove bytes from the front of the buffer, OH, but here’s some more data to stick onto the end, all your offsets are now potentially wrong as the 2 accesses to the same string resource run all over each other.

Please don’t yell at me.

:slight_smile: not yelling at you, you’re just standing in front of me while I wail at the injustice inherent in the universe :slight_smile:

James… With really kind attitude, your message content is perfect and I agree DataAvailable Event needs more info from Xojo. Just when hit a bug or a change ( a barrier ) … In spanish there is a say: Don’t kill the messenger. Eli is helping ( he’s a top notch forum member ) and not Xojo. I am beginning to have a nice collection of broken keyboards thrown :stuck_out_tongue:

As usual could not edit my post… (glad I’m on a laptop so keyboard will be safe :wink: )

Could you please explain me a bit more in-depth what reentrant code can affect Xojo. I studies it in the 90s, with MS-DOS but was something I never understood correctly. Would appreciate if you describe more in depth your worries. Just to add to my programming notes :slight_smile:

Truly I am not unhappy with Eli! He is terrific and I appreciate the help! My paragraph to him above wasn’t negative I hope, I am sorry if it was, I did not mean for it to be. All the other paragraphs were angry, but they were not at him, they were just me yelling at a system function I used to think I understood which is now very unclear.

I am inferring a lot from Greg’s short answer to the question I know. But if it’s necessary to treat the data available event as an interrupt now, that has all sorts of implications doesn’t it? If thats the case then the solution cannot be as simple as simply pushing my processing of the buffer into a timer event because that could get interrupted too. Either that or I simply do not understand what is actually happening which is the most likely answer :slight_smile:

I understand your angriness were not forwarded to Eli. I’m actually doing with ServerSocket/TCPSockets and seeing the same behavior I did not expect, after having most of my basement code written and tested.

In my case I will be using Linux, but developing Xojo on a Linux VM :frowning: … and having to test with remote debugger is frankly, exhausting. I wonder if this problem can affect Linux as I am developing on my mac and hoping it will work as it should (DataAvailable event) on Linux. You think this can be only a Cocoa issue or have done some Linux tests?

I see there was more from Eli that I didn’t answer as I was too busy ranting :wink: I said:

“So it’s a known factor that the data available event can run out of proper place of the rest of the program?”

My observation of the problem I’m having is this. I receive a packet in the data available event and from the event, if I have a whole packet, I start a potentially lengthy response to it. There can be a lot of code triggered from the reception of a packet. What I’m seeing is that I get another data available event in the middle of that previous code, interrupting it. When that new packet is complete control returns to my original code which finishes, but now events have happened out of order. The second received packet that was received in the middle of processing the first one has finished first even though it was received second. Not to mention the chaos that is caused by all the variables setup for the first packet are no longer correct half way through processing it. My initial thought that it was related to sending apple events I dont believe has anything to do with it given Gregs answer.

This is what makes me think the code can “run out of the proper place” previously in xojo I would get another data available event only once my own code from handling the last one had returned and was finished. But now my code can be interrupted at any point with another event. Interrupted, put on pause and the main thread or whatever re-enters with the new data.

then I said “and if thats the case, whats to keep the data available even from firing again while I’m still appending data to the buffer in the last event?”

And it may be the same thread, but it doesn’t wait for the currently running code to complete. It interrupts me in the middle of my code. If I was programming on an arduino with interrupts I would expect this behavior and it’s behavior is documented enough that I can still get a deterministic output while processing interrupts. But if xojo can just stop the execution of my code and call back into the data available event at any time, then it can also call back into the processing I might delay by use of a timer as Greg suggested. I can no longer treat that buffer string I would create as he suggests as immutable while I”m using it. What happens if I’m pulling a packet off the front of the string and re-assigning whats left to a new string to put in the buffer at the moment I’m interrupted. Where does the new data go? I would lose it in that case as it would be appended to the old string which I already took some packet off of and when it was done I would re-assign what was left to a new string, loosing what had been added in the meantime by the interrupt. Thats an overly complex example, but it’s exactly the same as if 2 threads were working on an unprotected resource at the same time. You can’t do that without semaphores and critical sections. But if this is re-entering the same main thread then I can’t use those because I can’t pause the main thread until the main thread completes :wink: Thats just silly.

All that lengthy, what if, chatting I’ve sat here and inferred from Gregs 2 sentences telling me how to handle it. I need more info than just move your processing to a timer because I dont believe thats the fix without more information. Please explain why it’s different on cocoa and why my fears on why thats not a fix dont apply :slight_smile: Then I will be happy. Need more than 2 sentences.

You’re going to find this behavior on Linux too. One way to handle this is to use a binary stream backed by a memoryblock. When DataAvailable fires, simply call:

bs.write me.ReadAll

If you want to take data off the front, just remember the position, copy the data you’re keeping in memory to another memoryblock and hook up the binary stream again, with the position set to the old position - the number of bytes you removed.

I’ve also done this with arrays of strings, but you’ve got to be very careful about encodings that way.

nope, I still dont understand. Please explain why this is suddenly necessary. I’ve been treating data available events in this same program as every other callback into the main thread since OSX version 10.1 and the first bug report I’ve had of this causing any trouble was day before yesterday. I can now duplicate this problem, I cannot duplicate it with the pre-cocoa version. This is something new. we need examples on how to properly manage this.

If I move to a binary stream reading from a memory block for the buffer, how do I properly clear the buffer or reset it without being in danger of having a re-entrant event put more data into the old one at the moment I’m switching references?

Do you have an example project I could take a look at?

Reading your first comment again, it strikes me that another thing that may have changed (I’m just hypothesizing here) is that apple events may simply not block sockets in Cocoa.

I’ve evaluated a lot of DataAvailable bug reports since I started here and it’s not uncommon to see people trying to do some sort of time consuming processing of the data coming in through a socket and getting themselves into trouble like this.