How do I tell?
Oh. I vaguely remember we talked about that before. You must be using ps aux through SSH.
You may want to look at the server logs. They list each request made for each object with the originating IP. A Grep should allow you to see only the requests by Grep yourexecutablename.cgi.
This will display only the lines containing the name of your app. Useful with long log files :
grep -n 'yourapp.cgi' logfilenameCreate a new file with the results
grep -n 'yourapp.cgi' logfilename >requestslist.txtFile a bug report about that. The .cgi files should not be taking up that kind of space, and they certainly shouldn’t be hanging around.
Just out of curiosity, do you have a place in your app which downloads a 30-40MB file to the user or possibly a WebFileUploader in your app which your users use regularly?
[quote=115647:@Michel Bujardet]This will display only the lines containing the name of your app. Useful with long log files :
grep -n 'yourapp.cgi' logfilenameCreate a new file with the results
grep -n 'yourapp.cgi' logfilename >requestslist.txtThanks Michel - I was able to find a bunch of different logs, and just browsed them with the cat command. My user crashed again last night. These comments are leading up to the time of reported crash: From the access_log I see all the commands (button push, scroll, text change, …) and yes they are all coming from the same IP. I also see the error_log which has many regular occurrences of “Script timed out before returning headers:” (errors) as well as “Timeout waiting for output from CGI script” (warnings). Nothing else extraordinary - so not sure if they are anything to be concerned about.
[quote=115680:@Greg O’Lone]File a bug report about that. The .cgi files should not be taking up that kind of space, and they certainly shouldn’t be hanging around.
Just out of curiosity, do you have a place in your app which downloads a 30-40MB file to the user or possibly a WebFileUploader in your app which your users use regularly?[/quote]
Sorry Greg - I may have done the math wrong - but still have an issue. Each myapp.cgi occupies about 0.4% of memory (available memory = 1GB) - so each myapp.cgi (check the math) consumes 4MB. However, each myapp.cgi appears to carry the burden of an httpd command, and each of those consumes 2%, or 20MB of memory. So it still stacks up. Last night there was 140 tasks running on my server, 139 of them were sleeping (the bulk of them being the commands mentioned above). They do seem to go away eventually - when I logged in this morning they were indeed gone, but they were still alive after 2-3 hours last night before I went to bed (and I did not restart the server or shut down the app) - but other automated processes like virus checks, backups, etc take place in the wee hours.
To answer your other question, I do not have a WebFileUploader in this app (I do have a downloader routine - and each of those files are about 300kB, and only a few a day are downloaded). I also use MBS Dyna_PDF for generating PDFs, which are what the users download.
Do you still think I should file a bug report?
As you mentioned about 40 or so requests to your app cgi lingering around, if they all originate from the same IP, it looks awfully suspicious. As if someone was flooding your site with requests.
If you continuously get so many requests from the same IP and it is not yours or an identified customer, you may want to block it out from your server. I believe Apache has such feature.
It is my guess that they are all stemming from the single user of my app. So I am not leaning toward suspicious activity yet - but always open to consider the possibilities.
Well that’s interesting. That error means that Apache timed out while waiting for a response from the .cgi file. That could also mean that the .cgi file is still waiting for a response from your app. Do you have any long running processes in your app which could take longer than 3 minutes?
Whew! That’s more in line with what I’d expect. Do you happen to know if your server is running the mod_perl module? This can make a huge difference memory usage and response time.
There shouldn’t be a one-to-one relationship between the number of .cgi instances and the number of Apache (httpd) instances. Apache spins up a certain number of instances so it can stay responsive, but each instance can handle hundreds to thousands of connections. The number of instances and number of connections per instance are configurable in the Apache config files. That said, it’s certainly possible that Apache hasn’t been tuned for the size of your server. That’s usually not done automatically for you.
So these PDFs, how big are they? how long do they take to generate? how often are they requested?
Not yet. I’d like to get answers to more of this first.
Not that I or my users are aware of - but when I get some time I will take a close look to see if something in the background is going on - though I doubt it.
Looks like it is: mod_perl 2.0.4-10.el6
Agreed - probably not 1 to 1, but observations do show similar number of instances of each.
When I get a chance to find that I will let you know. I assume there is some “.cfg-like” file I need to find and edit. Any hints on what the config line should look like?
About 300kB each. I have to check on time to create, but they are generally 1 page and take seconds to generate. Some days not at all, some days 3 to 5 would be typical.
One setting I have is “PHP Support (Run as: Apache Module, FastCGI application, or CGI application)”. I am currently set for FastCGI Application - does that sound like the best option for this?
I don’t think something named “PHP Support” is going to affect this, but you certainly should NOT be using FastCGI. I’d try Apache Module first and if that doesn’t work, switch to CGI Application.
Hi guys,
I have also been having “issues” with phantom sessions refusing to die away, but I seem to have fixed it, but unfortunately do not know how / why! (sorry, not very helpful). 2014 r2 cgi web app.
I have a cgi app deployed on 1701. You can try this for yourselves… browse to www.speedyspares.co.uk
After the splash screen has been replaced by the main homepage, scroll to the bottom and you will see “Active = x” where x is the number of open sessions. If you open a new tab, and browse again to the above URL you will see that both sessions will report an increase of 1 to the value of x (as you would expect). If you now close one of the tabs, you will observe that the remaining session page will report a decrease of 1 to the value of x after approx 3 minutes (as you would expect). However, previously (before I somehow fixed it), the session count would immediately rise by 2 at this point! Giving the impression that a dying session was sporning 2 new phantom sessions which would then hang around for hours. (I should point out that this is a live website, and random visitors could browse / leave at any time during the above experiments, but… I tried it many times with the same results).
So what did I do to fix it? I have no idea if the following info is relevant, and I appologize if this info turns out to be a false trail…
At the time this issue was occurring, I was attempting to solve a UI problem. When a user clicked on a Vehicle List (left margin of home page), there is a delay while the list is being displayed into a listbox at screen centre. With Google Chrome the delay is very brief (<1sec), but with IE the delay can be up to 10secs. So I decided to implement a popup with a ProgessWheel and a webtimer (permanently set to modeMultiple) which checks for the completion of listbox, and then closes itself when listbox has flagged ready. This is when I first noticed the “phantom” sessions issue.
Yesterday, I deleted this popup and its associated methods & timers, and instead implemented a webtimer within the mainpage object, and simply made a ProgressWheel visible / not visible according to the listbox flags. I also set this webtimer to modeOff by default, only changing to modeMultiple while displaying the PW, and resetting it to modeOff once the list is displayed.
Since making the above changes, the phantom sessions problem has gone away! I do not know if this is coincidence, or related - but thought I would share in case it was helpful.
By “popup” do you mean WebDialog?
Sorry, yes it was a webdialogue (type: modal)
Interesting. I do have a few places where webdialogs are used, and I have suspected a potential relationship to those. Will look into it.
However, the very simlpe test program I wrote for the feedback case (34438 - WebSession ‘ghosts’ chewing up memory) didn’t have a WebDialog, and was able to demonstrate the issue. Nevertheless - all things to be considered until we can conclude a resolution.
Thanks for the additional data points.
You’re welcome Mark.
Phillip, at 1701 has suggested to me that he configures my server so that I can run my apps in Standalone mode, instead of cgi mode. He believes this mode does not suffer from the ghost session syndrome - so that may be another workaround to consider.
Cheers,
Tony
I have thought about that as well - there was a healthy debate on that topic in this forum (another thread). I am currently educating myself on Apache tuning - perhaps I will discover something there to work with also.
OK - Now I have read up on configuring Apache, and understand some of the options. However, I am reluctant to make any changes yet, because my server is used for multiple purposes. I have some websites, some wordpress sites, and some Xojo-built CGI apps. At the moment, only the CGI apps seem to be creating problems, so I am reluctant to change server settings, just to try to address memory leaks caused by the CGI apps. I would rather see if we can identify the source of the memory leaks. So, back to my code and the Xojo team for insights.
To help me with the Apache nomenclature, I wonder if someone can translate between it a what I am observing on my server…
Apache documentation references “servers”; e.g.: MaxSpareServers 10
Also referenced are “clients”; e.g.: MaxClients number
And also there are “requests”; e.g.: MaxRequestsPerChild 10000
The issues I have been describing deal with large numbers of myapp.cgi commands (which I might think are “servers”???) and httpd commands (which I might think are “requests”???) - but I really don’t know if those terms are correlated. Any experts out there to educate on this?
Oh - and by the way, Greg…
I have changed to Apache Module, and everything seems to be working. I am running some tests right now, but as you suspected, it is not clear that this will resolve the issue I am trying to address. I am curious though, if you have the time, why was it so obvious to you that FastCGI was NOT the correct choice?