Threads and 2016r3

General
1

2016r3 is giving me problems with threads that connect to sockets and wait for data. I spied on the wire using PacketPeeper, and I see some of the traffic I expect to see, but the threads make little progress. Reverting to 2016r1.1 fixes all this.

Essentially when a thread connects or needs data, it suspends itself, and then when a connect or gotdata event occurs, those event handlers resume the thread. That seems fairly straightforward to me. The behaviour I’m seeing is as if the resumes are sometimes not being done or are ineffective.

I had a look through the release notes and can’t find anything that might account for this. I did see a reference to case 44683 but that would not appear to be relevant here. If no one has an obvious or instant suggestion I’ll start trying to find out what the threads are doing.

2

Using synchronous sockets with threads will indeed do that. You should try asynchronous sockets without threads.

3

I suspect he already does since he’s referencing the connect and DataAvailable events.

Look in the release notes for r2 and 2.1 and I think you will find that Threads got some love.

Keep in mind that the events in sockets fire on the main thread, regardless of whether they were started in a thread, so the gains may not be what you think they are. If you still want to use them, I suggest adding a timer to your thread class to periodically poll the socket to see if there’s more data.

It would be helpful to know why you’re using Threads though.

4

[quote=299119:@Greg O’Lone]Keep in mind that the events in sockets fire on the main thread, regardless of whether they were started in a thread, so the gains may not be what you think they are. If you still want to use them, I suggest adding a timer to your thread class to periodically poll the socket to see if there’s more data.

It would be helpful to know why you’re using Threads though.[/quote]
My handlers for the socket events are very short - basically they just resume the thread. I do have a timeout timer as well, since the sequence is, I send a command to the remote end, it responds with data. So the thread ought to resume either if there is data or if the timer (typically 20 seconds or so) expires. Obviously these handlers have control blocks with a thread pointer so they resume the right thread.

It would seem natural to use threads so that the main thread stays responsive to other user interaction.

5

Then I frankly don’t see what he expects from threads on top of an asynchronous socket.

Since the socket is asynchronous and does all its job in the background, it is just the same as a thread already. Only events will happen on the main thread, which unless you are moving huge amounts data will take very little time. You should not have any slowdown of the UI.

6

Has anything been done to the way threads are scheduled? I have quite a lot of logging enabled at the minute, and I’m seeing a thread, when putting out a series of messages, suddenly having a difference of 3 minutes in successive timestamps, and another where 4 minutes separates the timestamps for messages separated by a few lines of code in the thread.

Or perhaps there are now issues related to using SQLite in a thread.

7

There’s a bug I worked on yesterday where a thread switch will fail to happen sometimes if the main thread is running, a second thread is suspended, and the second thread is them resumed.

8

It’s not actually. The events themselves are, but if interpreting the incoming data is processor intensive, using a thread is absolutely appropriate. [quote=299126:@Tim Streater]Has anything been done to the way threads are scheduled? I have quite a lot of logging enabled at the minute, and I’m seeing a thread, when putting out a series of messages, suddenly having a difference of 3 minutes in successive timestamps, and another where 4 minutes separates the timestamps for messages separated by a few lines of code in the thread.[/quote]
Yes, threads got some love in r2 as I mentioned above.

Here’s what I suggest you try (and this will work going back as well):

Subclass Thread and create a pair of Start/Stop methods which sets a boolean property named something like “isRunning as Boolean”. In the Run event of the thread have code like this:

[code]While True

if Not IsRunning then
    self.Sleep(1000,false)
    Continue
End If

.... Your Code Here....

Wend[/code]

That way the thread is always “running”, but only actually trying to do something when you want it to.

The other way to do this is to have the thread code simply look and see if there’s more data to be processed from some sort of queue. Perhaps an array of strings which you populate in the DataAvailable event with:

dataQueue.Append Self.ReadAll

Then all you’d need to do in the run event is:

If ubound(DataQueue) = -1 then self.sleep(1000,false) Continue End if

9

@Greg O’Lone I’ve a Feedback about this with a sample project (it’s a private feedback) <https://xojo.com/issue/45248>

It has been verified.
The thread (randomly I.E. not always) will starve and in a random point.

10

It varies.

Bit tongue-in-cheek there, eh?

[quote]
Here’s what I suggest you try (and this will work going back as well):

Subclass Thread and create a pair of Start/Stop methods which sets a boolean property named something like “isRunning as Boolean”. In the Run event of the thread have code like this:

[code]While True

if Not IsRunning then
    self.Sleep(1000,false)
    Continue
End If

.... Your Code Here....

Wend[/code]

That way the thread is always “running”, but only actually trying to do something when you want it to.

The other way to do this is to have the thread code simply look and see if there’s more data to be processed from some sort of queue. Perhaps an array of strings which you populate in the DataAvailable event with:

dataQueue.Append Self.ReadAll

Then all you’d need to do in the run event is:

If ubound(DataQueue) = -1 then self.sleep(1000,false) Continue End if[/quote]

Thanks. I think I can see how to use that. As it happens, I do get all the data there is and then feed it to the app in lines one by one. Only when there is none left or only a partial line do I do another read and suspend.

Meanwhile a bit more testing with my app, this time with just the one thread reading from remote. It’s now looking like an issue with CPU starvation for threads: largely, the main thread has been idle while I’ve been looking at what’s going on. In my most recent test, I forced the main thread to do some stuff, and every time I did, the other thread suddenly moved on rapidly for a short while before becoming comatose again. It eventually completed, correctly, the work it was expected to do. It just took 17 minutes instead of about two.

I’ve got case 45925 open for this issue.

11

I’ve added a simple test project to case 45925 which demonstrates that having a thread start a timer, then suspend itself, and having the timer’s action resume the thread, does not work in 2016r3.

The thread gets status “Running” but does not run.

12

And what happens if the timer suspends the thread?

13

Is this already fixed @Joe Ranieri? It looks like I’m getting this in a CGI whereas it seems fine on my Mac.

14

It should be in 2016r4.

15

Thanks! Here’s my experience…

Disclaimer: this is my first Xojo (and OO) project. So I might be off with some of the concepts or language. Also, this is only in the CGI build, on my Mac it’s fine…

Anyway, what I’ve noticed is that when I do this in the App.Open

// Setup main thread // MT = New masterThread MT.Run

the thread starts out fine and the Open event ends, but after an async Xojo.Net.HTTPSocket ‘GET’ and suspending itself it appeared the complete app suspended. E.g. an (update) timer from the main thread also stopped working. And my browser never got a launch page.

Later I noticed the masterThread might have crashed because instead of ‘Resume’ my code did a fall back to ‘Run’.

[code]requests.Append request

Select Case Me.State
Case Thread.NotRunning // Just in case
Me.Run
Case Thread.Suspended
Me.Resume
Case Thread.Sleeping
// Me.Resume disabled because of API request limit
End Select[/code]

To make a very long story short: after I changed the ‘MT.Run’ in App.Open to a method and put it inside a ‘Xojo.Core.Timer.CallLater’ all is well.

I also had a feeling it might have to do with Xojo (or maybe Apache?) not being able to establish a connection to the browser, and stumbling with other things as a result.

I’m still trying to figure out what was going on though as in my mind the separate masterThread shouldn’t be able to block all other comms or startup of an App. At the same time, giving the App some breathing room solved the problem.

16

App.open is complicated in CGI. At that point, the app is still starting up and getting the protocol between the app itself and the cgi script established and any delay in that process can prevent the connection from starting up properly. Your solution of using a timer is fine or just have your thread sleep for a moment when it first starts up in the Run event, Self.Sleep(1000) should do the trick.

17

Thanks for your input Greg!

I tried that before I came to this solution, but it didn’t work. The starting of the thread had to be after the finishing of the Open event.

Sleeping ended up to be more or less the same; timers, threads or comms stopped working.

18

I’d be curious to know it the app was consuming any CPU when it was stuck.

19

Dunno if I can help you there as I’m only uploading the CGI to @Phillip Zedalis but if you want I can send you the project…

20

Not necessary, it was just a curiosity.