[plug] Remote syslog trap

[Brad Campbell]

G'day all,

Just another one of my 'lessons learned' mails.

My main server is a vanilla Debian install. 2 Days ago I decided I 
wanted to enable remote syslog to be able to capture the logs from my 
various remote devices dotted around the place. I put "-r" in 
/etc/default/syslogd and wooo it worked.

Last night I noticed my entire world had pretty much stopped. The server 
had ground to a halt, and anything I tried took forever (like a bind 
restart took 25 minutes!)

This machine started life a lot of years and a number of Debian releases 
ago when sysklogd was the default syslog daemon. Debian has since 
switched to rsyslog however. As this machine has always been upgraded 
rather than re-installed I never switched (or knew it was a good idea 
and I should).

Turns out my reverse zone for my home network was broken and syslog was 
stalling trying to reverse lookup my wireless router. This caused a 30 
second pause for *every* syslog entry, stalling every other process 
trying to log anything and paralyzing the machine.

After a lot of very slow straces, I finally figured out it was syslog 
causing the problem and disabled remote logging. Job done.

This morning, Mr Google and I found Debian suggests rsyslog, so I 
installed that and also fixed my broken reverse zone.

As the system was slowed trying to resolve my wireless router, bind 
ground to a halt and caused every other lookup to fail compounding the 
problem.

The symptoms on the server were very similar to those I had years ago 
when I had bad blocks in my swap partition, and it took me _ages_ to 
figure out what was really going on.

Regards,
Brad