[plug] Ubuntu woes

[Daniel Pittman]

Steve Baker <steve at iinet.net.au> writes:
> Daniel Pittman wrote:
>> Steve Baker <steve at iinet.net.au> writes:
>>   
>>> Nothing out of the ordinary in the kern.log, messages, syslog,
>>> daemon.log, or the usual suspects in /var/log.
>>>
>>> All disks are in a hardware RAID - 8-port adaptec SATA-II raid card, 2
>>> discs in RAID-1 and 6 in RAID-5.  The two RAID-1 drives are 12 months
>>> or so old, the other 6 are all new.  My next plan is to find the
>>> afatools kit from Adaptec and run the afacli command to check the
>>> SMART status of the discs and then scrub (check) the arrays.
>>>
>>> What is confusing is some tasks are quite quick and others really
>>> slow.  Unpacking a tgz archive was quick, aptitude safe-upgrade takes
>>> a long time to do steps like read state information, build the tag
>>> database, etc. but the download was quick, then it took even longer to
>>> do the post-installation steps.
>>
>> It sounds, to me, very much like your system is delivering good write
>> bandwidth[1] but terrible read bandwidth or latency.
>>
>> Unpacking the tar is fast iff the content is in memory, but the update
>> is slow when it reads databases or faults in code from disk.  Network
>> runs at full speed, and the post-install was slow only where it had to
>> fault in code, etc...
>   
> Sounds like a good theory - I am not sure of the best way to test this
> though.  

Well, the easy way would be to time reading a file that had not been
read before, then time re-reading it.  That should show up as slow the
first time around, fast the second.

You should also be able to echo ... 3, IIRC, to /proc/sys/vm/drop_caches
in order to have the kernel flush all the caching.  Ah, there we go:

http://linux-mm.org/Drop_Caches

> I know of hdparm/sdparm but I don't know if they are appropriate for
> RAID volumes.  I can time large file copies and such (eg dd
> if=/dev/urandom of=/some/file bs=1048576 count=1024 or whatever) but I
> have nothing to compare results to.  And how do I stop caching from
> upsetting the results?

Well, my expectation would be that y'all could stop other activity and
time the commands 'dd if=/dev/zero of=whatever bs=1M count=100; sync'
together, which should give you a view of the time to write data out.

>> At a guess.  Oh, are you running a vendor driver for the Adaptec card,
>> or is it all open source drivers?
>   
> I'm running the default Ubuntu aacraid driver.  When I ran the
> installer it found that driver automatically, so I just used that.  

That seems reasonable, and if the driver picks it up then it is real
hardware RAID and all. 

> In my experience the aacraid drivers have been pretty stable and
> reliable, and I believe that Adaptec have always been quite open about
> providing specs for their hardware.  

Sadly, these days they sell a pile of "fakeRAID" cards, for which only
binary drivers exist.  Great, isn't it?  Anyhow, that is why I ask.

[...]

> Something I forgot to mention is that it's also very slow to boot,
> which could indicate a hardware level issue with the RAID card (or
> maybe it slows down after the kernel loads the raid driver).  I think
> it's the most recent firmware revision but I'll check on that too.

I would almost guess that your RAID array was degraded, and that is
killing performance.

Actually, a question: you have a RAID1 and a RAID5, right?

Is the terrible performance the same on both?

Regards,
        Daniel