[plug] Logrotate Killing Apache (Again)

[Timothy White]

Ok. Finally got around to trying to fix this problem were apache won't
restart when logrotate tries, cause ppp is started (via pon) from a
php script, so gain's the file handle's to the apache log files, which
means when apache trys to start, the log locks are still present.

On 24/10/05, Cameron Patrick <cameron at patrick.wattle.id.au> wrote:
> Bernard Blackham wrote:>
> > Yep. Ideally there'd be some way to close all FDs, but I don't know
> > one yet and haven't looked hard.
>
> Can you be evil with /proc/self/fd?  e.g.
>
> for x in /proc/self/fd/*; do
>     if [  "$(basename $x)" -gt 2 ]; then
>         eval "exec $x>&-"
>     fi
> done

I tried this method first (I edited pon, I know, bad boy, but it was
easiest). Unfortunately file handler 255 seems to get into the list
somehow, and seeing as it doesn't exists, annoys the heck out of it.
Some changes later, and I find out that 3 also has problems when it
tries to be closed. In the end, I used Bernard's code to just close 6
and 7, as they appeared to be the apache log file handles.
Unfortunately, apache still doesn't restart.

A look at the fd for the current ppp process shows...(6 and 7 were
closed in pon, so they didn't get passed to ppp)
$ sudo ls -l /proc/20483/fd/
total 16
lrwx------  1 root root 64 2005-12-18 16:54 0 -> /dev/null
lrwx------  1 root root 64 2005-12-18 16:59 1 -> /dev/null
lrwx------  1 root root 64 2005-12-18 16:59 10 -> socket:[35824802]
lrwx------  1 root root 64 2005-12-18 16:59 11 -> socket:[35824803]
lrwx------  1 root root 64 2005-12-18 16:59 12 -> /var/run/pppd.tdb
lrwx------  1 root root 64 2005-12-18 16:59 13 -> /dev/ttyS0
lrwx------  1 root root 64 2005-12-18 16:59 14 -> /dev/ppp
lrwx------  1 root root 64 2005-12-18 16:59 15 -> /dev/ppp
lrwx------  1 root root 64 2005-12-18 16:59 2 -> /dev/null
lrwx------  1 root root 64 2005-12-18 16:59 3 -> socket:[35435256]
lr-x------  1 root root 64 2005-12-18 16:59 4 -> pipe:[35435260]
l-wx------  1 root root 64 2005-12-18 16:59 5 -> pipe:[35435260]
lrwx------  1 root root 64 2005-12-18 16:59 6 -> socket:[35824798]
lrwx------  1 root root 64 2005-12-18 16:59 7 -> /dev/null
lrwx------  1 root root 64 2005-12-18 16:59 8 -> socket:[35820051]
l-wx------  1 root root 64 2005-12-18 16:59 9 -> /var/log/ppp.debug

Nothing in apache logs from what I can see. lsof tells me the
following processors have the following files open in
/var/log/apache2/
$ sudo lsof +D /var/log/apache2/
COMMAND   PID     USER   FD   TYPE DEVICE    SIZE  NODE NAME
apache2 29832     root    2w   REG  254,2     731 16791
/var/log/apache2/error.log
apache2 29832     root    6w   REG  254,2     731 16791
/var/log/apache2/error.log
apache2 29832     root    7w   REG  254,2 2614515 16813
/var/log/apache2/access.log
apache2 29834 www-data    2w   REG  254,2     731 16791
/var/log/apache2/error.log
apache2 29834 www-data    6w   REG  254,2     731 16791
/var/log/apache2/error.log
apache2 29834 www-data    7w   REG  254,2 2614515 16813
/var/log/apache2/access.log
apache2 29835 www-data    2w   REG  254,2     731 16791
/var/log/apache2/error.log
apache2 29835 www-data    6w   REG  254,2     731 16791
/var/log/apache2/error.log
apache2 29835 www-data    7w   REG  254,2 2614515 16813
/var/log/apache2/access.log

The process file descriptors are as follows
$ sudo ls -l /proc/29832/fd/
total 8
lr-x------  1 root root 64 2005-12-18 17:01 0 -> /dev/null
l-wx------  1 root root 64 2005-12-18 17:01 1 -> /dev/null
l-wx------  1 root root 64 2005-12-18 17:01 2 -> /var/log/apache2/error.log
lrwx------  1 root root 64 2005-12-18 17:01 3 -> socket:[35435256]
lr-x------  1 root root 64 2005-12-18 17:01 4 -> pipe:[35435260]
l-wx------  1 root root 64 2005-12-18 17:01 5 -> pipe:[35435260]
l-wx------  1 root root 64 2005-12-18 17:01 6 -> /var/log/apache2/error.log
l-wx------  1 root root 64 2005-12-18 17:01 7 -> /var/log/apache2/access.log
$ sudo ls -l /proc/29834/fd/
total 8
lr-x------  1 root root 64 2005-12-18 17:01 0 -> /dev/null
l-wx------  1 root root 64 2005-12-18 17:01 1 -> /dev/null
l-wx------  1 root root 64 2005-12-18 17:01 2 -> /var/log/apache2/error.log
lrwx------  1 root root 64 2005-12-18 17:01 3 -> socket:[35435256]
lr-x------  1 root root 64 2005-12-18 17:01 4 -> pipe:[35435260]
l-wx------  1 root root 64 2005-12-18 17:01 5 -> pipe:[35435260]
l-wx------  1 root root 64 2005-12-18 17:01 6 -> /var/log/apache2/error.log
l-wx------  1 root root 64 2005-12-18 17:01 7 -> /var/log/apache2/access.log
$ sudo ls -l /proc/29835/fd/
total 9
lr-x------  1 root root 64 2005-12-18 17:01 0 -> /dev/null
l-wx------  1 root root 64 2005-12-18 17:01 1 -> /dev/null
l-wx------  1 root root 64 2005-12-18 17:01 2 -> /var/log/apache2/error.log
lrwx------  1 root root 64 2005-12-18 17:01 3 -> socket:[35435256]
lr-x------  1 root root 64 2005-12-18 17:01 4 -> pipe:[35435260]
l-wx------  1 root root 64 2005-12-18 17:01 5 -> pipe:[35435260]
l-wx------  1 root root 64 2005-12-18 17:01 6 -> /var/log/apache2/error.log
l-wx------  1 root root 64 2005-12-18 17:01 7 -> /var/log/apache2/access.log
lrwx------  1 root root 64 2005-12-18 17:03 8 -> socket:[35829210]

Doing a quick cat of the fd output's, sort and uniq -d on it (to show
me the duplicate lines), I see the only fd's ppp and apache have in
common are
/dev/null
pipe:[35435260]
socket:[35435256]

So should I be closing them? (the pipe and socket, obviously not dev
null as i think that's where stdout and stderr are pointing)
So should I close fd's 3 and 4? Also, in pon, should I close fd 4
first, then 3, so that the numbers don't change, or will they stay the
same? I assume they stay the same from Bernard's script.

Tim
p.s. For any read who has gotten this far, you are lucky. If you are
running a wordpress blog, and want to try a autosave technique for
comments, then buzz me. I have taken a autosave plugin for the admin
panel from twilight? and modified it to work for comments! Although,
you need to make a small template hack as well.
--
Linux Counter user #273956