Web app hangs randomly (100% CPU)

Targets Web
22

The Socket is a subclassed TCPSocket that is created in a TCPServerSocket AddSocket event.

Nope, the only methods that I use are:

self.Write(“Some data”+Chr(10))
self.Flush

and

Result = self.Read(256)
self.Purge (I think this is redundant as close will flush and purge the socket)
self.Close

23

Ok, well I’d like to suggest that you either add a timer to your subclasses socket and call Poll once in a while, or make an external mechanism that polls the open sockets. You could easily do this in the ServerSocket itself. Keep in mind that calling Poll will fire DataAvailable events.

Just out of curiosity… Is this code in a DataAvailable event?

Result = self.Read(256) self.Purge (I think this is redundant as close will flush and purge the socket) self.Close

If it is, calling Purge may also be triggering another DataAvailable event resulting in an endless tight loop.

24

It’s not in the DataAvailable event, but it is in a method in the DataAvailable event, so that could be it. Would just calling close be better?

In regards to polling open sockets, do I want to poll all open sockets, just the serversocket or just the subclassed tcpsockets.

Thanks

25

Just the ones you’re having trouble with

26

Hmm, I don’t know what one is giving me trouble, and it is quite random. I guess I will just try all TCPSockets.

27

So I was making a desktop program to test some the tcpsocket (by sending different forms of data to the socket) and I had some unexpected results. When I clicked the button to send the test data it froze. Upon looking at the CPU it was at 100%. I can imagine something close or exactly the same could be happening in my main program.

TestCode:

//TCPSocket1 is on Window1

TCPSocket1.Address = "<Not_Important>"
TCPSocket1.Port = "<Not_Important>"
TCPSocket1.Write(chr(10)) //Just random data
TCPSocket1.Flush //This line right here is the cause
TCPSocket1.Close

Result:

I have similar code in the main application:

In the Connected event of a TCPSocket:

self.Write("<about 10 char long string>"+Chr(10))
self.Flush

Then in the DataAvailable event:

dim Result as string = self.Read(256)
//Do some things that take time
self.close

if you take out the code in the middle, they are almost exactly the same, I will have to do some more testing but I hope I can put that last nail in this coffin as well as help any others that come along this in the future.

28

Just some more observations…

  if app.Prefs.Get("LoginID","NO_KEY") <> "NO_KEY" and Session.Cookies.Value("LoginID") = app.Prefs.Get("LoginID") then
    Session.LoggedIn = True
    
    if Width < 768 then MobilePage.Show else MainPage.Show
  elseif app.Prefs.Get("SubLoginID","NO_KEY") <> "NO_KEY" and Session.Cookies.Value("SubLoginID") = app.Prefs.Get("SubLoginID") then
    Session.LoggedIn = True
    SubPage.Show
  end if

With this code I get a Segmentation fault

SubPage.Show

Without it, the program continues as normal, It never actually gets the that line of code. I put a break point in there and it never dropped into the debugger, it just Seg faulted.

29 
add a timer to your subclasses socket and call Poll once in a while, or make an external mechanism that polls the open sockets

I can’t seem to catch one actively open. With my (non Web) console application running on Ubuntu 12.04.2 LTS within a HyperV virtual environment. My daemonized console application is simple, create a SocketServer on port 7878 which listens for a connection, if connection is made, immediately sent an HTTP formatted response with a single jpeg image, once sendcomplete is fired, disconnect. I’ve now setup logging at all events, errors, unhandledexceptions. There are no socket errors being reported, no unhandledexceptions. Just at random it goes into some loop. But every loop I have (even my do while for my DoEvents() first writes a log entry. But all logging stops, all sockets stop responding, 100% CPU, 0.4% memory used (according to htop) overall system memory still shows 165Mb of 2016Mb available.

In trying to debug I’ve also noticed some other odd issues. DoEvents(1000) or any specified millisecond time also causes the same issue, but immediately XoJo goes into space, and sockets stop running (rather than using DoEvents() with no parameter, then it goes into space after thousands of socket connections). So maybe this isn’t quite a TCP Socket issue, but more DoEvents hang.

Also, I’ve tried limiting the number of sockets with SocketServer.MaximumSocketsConnected = 5, it still fires off 10 adding socket calls upon startup. I wish there was an AllConnections method rather than just ActiveConnections so you could get a list of all the sockets fired up. My ActiveConnections always returns a Ubound of -1 even when my timer is firing at 400 milliseconds, I just can’t catch an open socket with this app.

I’m stress tesing it with 70-80 connections per second also. But even when I slow it down to 10 sockets per second, it will go into space within an hour. At this point, I need another console app as a WatchDog to kill off this one and fire up another instance.

30

Also notable. I’ve left one of these run for over 24 hours in 100% CPU mode. Shutting down all test calls so nothing will try making a new TCP connection. Thinking eventually, something will timeout or it will come out of its funk, but same deal, it remains stuck in some non-XoJo code (internal XoJo kernal) loop.

31

You’ll have to provide us with a sample then, because we can’t reproduce this in our own apps.

32

Hi Greg,
Sample submitted as case 41753.

33

Since the 30th of september when I made the changes to my project I haven’t seen any hangs. (running on more the 25 servers each with a different setup, some serving 100’s requests a day)

In the connected event of the TCPSocket, I only write the data, I don’t call flush (I think this was the main culprit)
In the DataAvailable event, I read 256 bytes then call TCPSocket.close at the end of the method.

The whole process from connect to close is less than 100ms.

I’m not sure what I did fixed it, but it has been more than 5 times the maximum amount of time it took for the app to hang before I made these changes.

34

Wow Alex, you might be on to something there. I too have removed the “Flush” command after the socket write and that may have been the issue. I’ve been stress testing the application for a day now and have not had it fault or hang.

35

Great to hear that!