I have a weird situation here. I deployed a web application this morning and no matter what I did, I get 500 errors in the browser. In my Apache Error Log, I see:
[Fri May 16 06:44:59 2014] [error] [client W.X.Y.Z] Can't use an undefined value as a symbol reference at /var/www/portal6/portal.cgi line 118.
[Fri May 16 06:44:59 2014] [error] [client W.X.Y.Z] Premature end of script headers: portal.cgi
This is not the first web app I have deployed. I have a few others running as well w/o issue. I believe permissions are fine:
drwxr-x--- 4 jcowgar www-data 4.0K 2014-05-16 06:35 .
drwxrwxr-x 53 www-data www-data 4.0K 2014-05-16 00:31 ..
-rw-rw---- 1 jcowgar www-data 54 2014-05-16 06:11 config.cfg
-rw-rw---- 1 www-data www-data 0 2014-05-16 05:56 errors.log
-rwxr-x-- 1 jcowgar www-data 6.3M 2014-05-16 06:35 portal
-rwxr-x-- 1 jcowgar www-data 6.9K 2014-04-10 10:53 portal.cgi
drwxr-x-- 2 jcowgar www-data 4.0K 2014-05-16 06:35 portal Libs
drwxr-x-- 3 jcowgar www-data 4.0K 2014-05-16 06:35 Resources
Those are the same permissions I run on my other web apps w/o issue.
So, I began playing around locally and ran ./portal6.cgi from the console. That works fine (as well as can be expected that is, the “Your browser is not supported” page comes up. Anyway, once I run it from the command line, it then works great from the browser! If at the console I killall portal then I will begin getting 500 errors again. Run ./portal6.cgi from the console and all works great again.
Any thoughts on what is going on or how I can go about debugging this?
OK, I found another clue. If I run the portal binary as www-data, it exits with the exit code 246. I do not do that anywhere in my app, so it must be the framework exiting with that code.
Any idea what exit code 246 means? Or maybe that isn’t anything at all, maybe it just means it was not called correctly from portal.cgi
If you can launch the app from the command line and it works, the problem is probably caused by the web server user not having permission to launch the app ( or write to the tempdirectory) on your behalf.
What temp directory is it trying to write to? /tmp is rwx,rwx,rwx. All other permissions seem fine.
That should be fine.
As far as just running it goes, in cgi mode you’re going to have limited success because the cgi script and the app need to communicate for everything to work properly.
Have you tried launching the app with ldd to see if any dependencies are missing?
The app works great, as long as I run portal.cgi first from the command line. Running ldd portal shows all dependencies linked.
Is there a specific file or directory it maybe creates in /tmp that could have been created by me calling portal6.cgi directly as one user and it now not being able to access it as www-data?
I’ve seen this problem before. It’s been fairly random. For whatever reason an app will stop launching from the CGI. It worked previously and nothing changed. I launch it from command line to make sure everything is okay and suddenly it starts working. Never is a problem again.
For me, if the app terminates then I have to launch it from the command line again to make it work again. Surely seems like a permissions problem, I just can’t figure out permissions to what. It’s not anywhere in the deployed directory… I did a chmod -R a=rwx www-root/ and it still fails in the same way. I have obviously since reset permissions to their proper and secure permissions, but just as a test I did the above.
What’s the permissions of the app folder itself inside the cgi-bin? Do you have suexec enabled? It won’t let it run if the app permissions are fine but the host folder is 777. I would check your Apache error log and your suexec log. You should be able to track down the issue.
Phillip, suexec is not enabled. The host folder is 770. The apache log says nothing more than what was in my original post:
[Fri May 16 06:44:59 2014] [error] [client W.X.Y.Z] Can't use an undefined value as a symbol reference at /var/www/portal6/portal.cgi line 118.
[Fri May 16 06:44:59 2014] [error] [client W.X.Y.Z] Premature end of script headers: portal.cgi
I think I have it solved. I noticed in /tmp that there were files named com.*.portal and a few directories also. I did a rm -rf /tmp/portal* and everything is now working as it should. I’m thinking a file got created in the /tmp directory as me when I tried running from the command line to make sure everything worked. Then subsequent runs as the the web user failed because it couldn’t deal with the file in /tmp. That’s just a guess, but sure looks like it’s working now!
That’s the mutex file. Essentially it is there to make sure you don’t get two of the same app running at the same time… And you are right, running from the command line created a mutex that was owned by the user you logged in with, and when it crashed, the mutex was left behind. When the app quits normally, that file gets removed. Then when your apache user tried to launch it, it was refused because for all it knows, another user has a copy running on the system