[plug] handling failed non-redundant storage in a server

[James Devenish]

In message <1076555369.1835.52.camel at bucket.localnet>
on Thu, Feb 12, 2004 at 11:09:29AM +0800, Craig Ringer wrote:
> Our core server at the POST has just had a small accident (it's been
> one of those weeks)

Sounds so cute :-)

> For some reason the load average is at something stupid like 12 (this
> machine's norm being < 1, usually about 0.3), but it doesn't appear to
> reflect real load. Perhaps it's the two sync processes that are hung.

Since the load number basically indicates processes that are queued (not
sure what it truly indicates under Linux), it is possible to have a high
load average without noticeably slow performance, if those processes
don't have long-lasting I/O or computation requirements. I guess, in
your case, the hung processes are 'in the middle of something' but each
time the kernel looks at those processes, it decides to ignore them and
try again later.

> I was wondering if there's any way to deal with this - to remove the
> processes I know will never recover, unmount the dead volume without
> causing any harm to other parts of the system, etc.

Sometimes I wish it were possible to remove a broken, non-in-use volume
from the userspace mount tab without actually making the kernel attempt
to sync/unmount it (i.e. just "hide" it). Would prefer to have
programmes die "cleanly" with 'file not found' than hang.

> Being able to convince GNOME not to try to talk to the residual nautlus,
> but rather start a new one, would do too. Is there some lock or socket I
> can remove to force this?

I've never had any luck with Linux's "uninterruptable processes", so
perhaps the above is your best bet? I don't know much about GNOME's
interprocess communication, but presumably it would be capable of
dealing with processes that terminate abnormally. Perhaps, then, there
is a way of triggering "deregistering" manually without the co-operation
of the broken processes.

> it's loud enough already.

You clearly work in a noisy environment! Do you have your own streaming
audio server that we can connect to, in order to listen in on your
workplace? ;-)