Web Server TCP LoopBack Problem

Targets Web
1

I have a Web App that runs on a Windows Server 2008 Standard running in VMWare ESXi. It has 8GB of RAM and 4 virtual processors. I am using the Abyss Webserver and the app is accessed securely (HTTPS). The database is MS-SQL Server 2008. I have had intermittent problems since it has been deployed. I added a second Web App and today it began to get some use and it started having problems. The app was sluggish and would simply drop off line from time to time. It is so problematical it cannot really be used.

It was built with 2014r3.1 but I also tried 2014r2.1 with the same results.

As I dug into things with Resource Monitor (which can be run from the Performance Tab of the Task Manager in Server) I saw many TCP Connections listed as “IPv4 loopback”. If the app is not being accessed (no active browser connections) the TCP connections drops to just a few and none are loopback. When the first connection comes up to the login page the app spikes to at least 50 TCP connections that are mostly these “IPv4 loopback”. These show in the resource monitor with no Image Name (program) and no PID (Process ID). The local port is listed as a high number port (50,000 range) and all seem to increment by one for each. The remote port is the same for every one of these TCP loopback connections generally in the 25,000 range.

If I launch the login page and do nothing it seems to create about 50 connections and more than 40 are these loopback connections. If I type the User ID and tab to the password field the TCP connections spikes to more than 100 then in a minute or so starts to settle back to the 50 level. I do have a keypress event that looks for the ENTER key for the UserID and PW (the only keypress events in the app).

If I go ahead and login to the first main page and do nothing I can see the connections spike to well over 250 and slowly rise to maybe 300 even if I do nothing. This is all with no other users in the system. If I logout and close the browser it settles back to only a few connections and no loopback connections. If I remain on a page in the app the loopback connections start to drop. I have a 15 minute timeout and by the time that is reached there are about 25 connections.

I have another totally different virtual server in a different datacenter that is a very similar configuration except it uses MySQL. It does seem to spike some loopback processes but not so many and the app generally works very well. The app on this server is much more complex.

Is this normal behavior?

Where else should I look?

2

Standalone or cgi?

3

This is a CGI App and it uses the Abyss Webserver. If it makes a difference it is also HTTPS.

This morning it seems to be working better but it still has lots of loopback connections.

I also compiled a standalone version on port 8080 and it seems to be a bit quicker. Because I have active users on the CGI version I cannot measure the loopback count for the standalone version until all users are out of the CGI version and I can shut it down.

4

I have the same problem :frowning:
But with a Standalone (Abyss webserver Reverse proxy, Windows Server 2012 R2 Standard).
Without SSL, there has fewer connections, but there is still too much.

5

Olivier,

Did you attribute this to Abyss?

I created a standalone app and put it on the same server. I connected non secure to port 8080 (no reverse proxy) and I see the same spike but less than via Abyss. Similar to you.

Thanks.

6

It is possible. I had a detailed and rapid response of the Abyss support (excellent support!), I’m doing tests, I will then result here.

7

(sorry for my bad english)

Apparently, in reverse proxy, each request performed causes a second.

Example: I display the web app in my browser. This causes multiple requests to the proxy (Abyss). There is a request (TCP connexion) for each resource: each image, css, javascript each file, etc.

The proxy then relays the requests to the web app (the loopbacks). So, all requests are doubled.

So we have:

  • A web app, so an application which by nature already generates a lot of requests to the server

  • A Xojo web app, 100% server, which by nature generates more requests

  • A proxy (this may be similar with CGI?) which doubles requests

  • In the end, a lot of requests …

Abyss uses HTTP / 1.0 reverse proxy, so each request uses a TCP connection (after, this connection is not used, so it is grayed out in the resource monitor). It does not help to reduce the number of TCP connections…

With the HTTP / 1.1, the server can reuse connections faster apparently.

Aprelium will therefore apparently do an update to use HTTP1.1 in reverse proxy.

Meanwhile, it is possible to update the timeout of the TCP connection. The default is 240 seconds. By dropping to 30 seconds (the minimum value apparently), it already reduced significantly the number of connections. But lowering it as much, I do not know if it will not cause other problems …

To change:

http://msdn.microsoft.com/en-us/library/aa560610.aspx

To understand the problem:

http://stackoverflow.com/questions/337115/setting-time-wait-tcp

Thank you to the support of Aprelium (Abyss)!

I hope this will be enough to reduce the number of connections. This is an important problem.

8

Lower the timeout is actually very effective.

With a timeout of 30 seconds, for my app:

  • 12 client web apps connected : 200 connections
  • 16 : 250
  • 20 : 320
    about 15/16 connections per client

With a timeout of 50 seconds:
about 25 connections per client

With a timeout of 100 seconds:
about 50 connections per client

is proportional …

With a timeout of 30 seconds, that’s ok. Especially apparently Windows can handle millions of TCP connections (to check, I’m not sure).

It’s still not good news, because there may be limitations on the number of connections at the host, firewall, etc.

And if we install 30 applications on a server, and each has 30 connections, the server will load?

Far Abyss was perfect, but there we have perhaps a weakness. I hope they will move quickly reverse proxy in HTTP1.1

9

Olivier,

WOW — Thanks for all of your work. EXCELLENT!

I have set the delay to 30 seconds but I cannot reboot until later because of active users.

I will also log a support call to Aprelium that I have the same problem.

After I reboot I will post my results here also for others to see.

Thanks again.

Mark

PS – Your english is very good. My French is zero. ;-))
PS2 – My condolences for the recent attacks in Paris.

10

Thank you Mark.
This is Aprelium me well informed.

I noticed that it was possible to put a timeout less than 30 seconds. 30 seconds, it seems to me much already. TCP packets may arrive 30 seconds later? But apparently Microsoft advises not to go under 30 seconds. There must be reasons …
However, if you still too TCP connections, you can try to put a lower value.
Thank you for the results after boot.

11

All these TCP connections are not due to the need that Xojo whenever closes the TCP connection (and so then opens new) instead of the reuse?

See this thread:
https://forum.xojo.com/3132-help-with-apache-proxy-and-virtual-hosts