[plug] server load spikes

[Craig Ringer]

I think I found the source of those server load spikes, and it's
something I think some others may find useful. Many a veteran sysadmin
may laugh at me, but sometimes it's not obvious what's causing a problem
and if you've never seen it before, it can be hard to figure out.

I eventually found something that appeared to coincide with the load
spikes - heavy incoming email traffic. After tweaking and tuning things
in ClamAV, MimeDefang, Sendmail, and Cyrus without success, I noticed an
oddity:

# du -h /var/log/debug.log
debug.log 325M

Hmm... _lots_ of logging activity going on there.

It turned out that the load spikes were caused by syslog, which was
writing out absurd amounts of data to the debug log and tty12, and was
synching after every write while doing so. Wham! Magic melting server
goodness.

Pulling back the debug-level logging and telling syslog not to sync
after every write (prepend a '-' to the log target in syslog.conf)
appears to have solved the issue. Hopefully, the next time I see a load
average of 45+ will be after a disk failure or other truly nasty event,
not a particularly heavy wave of spam.

So ... if you're seeing weird load spikes that don't appear to be
associated with heavy disk I/O, CPU use, or memory use, look at your
syslog.conf and other processes (databases, etc) that might be syncing
after every write.

Craig Ringer