[plug] server failing with bizarre disk errors

[Craig Ringer]

>>Have you enabled S.M.A.R.T and queried your drives (if possible)?
> 
> I think S.M.A.R.T is enabled - will have to read up on how to query them.

Its a bit of a black art from what I can gather. The S.M.A.R.T standard 
attributes don't seem too useful, and the vendor-specific ones are never 
documented anywhere (*sigh*). Still, you can tell that many spin 
retries, for example, suggests a dying spindle motor, and reallocated 
sectors don't bode well for platter condition.

# smartctl -v /dev/hda

Vendor Specific SMART Attributes with Thresholds:
Revision Number: 16
Attribute                    Flag     Value Worst Threshold Raw Value
(  1)Raw Read Error Rate     0x000b   200   200   051       0
(  3)Spin Up Time            0x0007   105   100   021       3808
(  4)Start Stop Count        0x0032   100   100   040       67
(  5)Reallocated Sector Ct   0x0033   200   200   140       0
(  7)Seek Error Rate         0x000b   200   200   051       0
(  9)Power On Hours          0x0032   097   097   000       2372
( 10)Spin Retry Count        0x0013   100   253   051       0
( 11)Calibration Retry Count 0x0013   100   253   051       0
( 12)Power Cycle Count       0x0032   100   100   000       64
(196)Reallocated Event Count 0x0032   200   200   000       0
(197)Current Pending Sector  0x0012   200   200   000       0
(198)Offline Uncorrectable   0x0012   200   200   000       0
(199)UDMA CRC Error Count    0x000a   200   253   000       0
(200)Unknown Attribute       0x0009   200   200   051       0
SMART Error Log:
SMART Error Logging Version: 1
No Errors Logged