Apache SSL Mutex
I had a crazy problem to day when I decided to clear out some junk from my server. I was busy doing it that I did not notice suddenly my webserver died mysteriously. The even crazier thing still is that after that I just could not restart it. Each time I attempt to start up the httpd, I get this message in the Apache error log:
[Wed Jul 11 22:11:03 2007] [notice] Digest: generating secret for digest authentication ...
[Wed Jul 11 22:11:03 2007] [notice] Digest: done
[Wed Jul 11 22:11:03 2007] [notice] FastCGI: process manager initialized (pid 3810)
[Wed Jul 11 22:11:03 2007] [emerg] (2)No such file or directory: Couldn't create accept lock
[Wed Jul 11 22:11:03 2007] [alert] (2)No such file or directory: FastCGI: read() from pipe failed (0)
[Wed Jul 11 22:11:03 2007] [alert] (2)No such file or directory: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye
So there is a problem, and the carnal rule of server problem is the outage is always somehow related to the most recent thing you did. This means something I have just removed must have upset it, but what? The error message helpfully tells me Apache was looking for a file but could not find it, but which file? (Don't you just hate code that do that? Why display the system error without the associated context, namely the name of the file it was looking for?). The other thing is even though the log seems to mention FastCGI, I discovered once I pinpointed the problem that it had nothing to do with FastCGI.
Note that restarting the machine does not help solve this problem (I already tried that).
OK, the next thing I did before that, a few weeks ago after upgrading Apache, was to reconfigure the SSL layer. Somehow this layer just stopped working with the OS upgrade. The error was another cryptic one, which was added to the log file each time I tried to access the site with https instead of http protocol:
[Thu Jul 05 01:46:28 2007] [error] (2)No such file or directory: Cannot create SSLMutex with file `/var/cache/apache2-mod_ssl/ssl_mutex.5601'
And yes I did check, the path /var/cache/apache2-mod_ssl does exist, is writeable etc.
After Googling for it a bit, I found that the only way to get the SSL layer to work was to add the following line to the SSL configuration (why? I'd never know):
Bingo! The /other path is the one I have been 'cleaning up'...
So, I recreated the /other/cache directory and Apache happily started up again. Did you notice that even though the problem lies in the SSL configuration layer, there was no mention of it at all in the log, with the red herring being FastCGI (which in fact happened to be the result, not the cause) ?
The moral of the story is,
- do not remove things without knowing what they do, and
- backtrack your path when there is a problem, as the changes you have just made are most likely to be the culprits