[plug] evil performance on large directories.

[Craig Ringer]

On Tue, 2005-04-05 at 12:49 +0800, Shayne O'Neill wrote:
> The server I run at the murdoch guild has a couple of monster directories.
> The shared student drive where our lovely student reps have promptly
> dumped 10 million cubic tonnes of debris all into the root , and the www
> directory (ugh. webdesigners... We now have a cms that puts an end to
> that, but the debris is all there still)
> 
> Regardless, any operations that require a listing seem to take ages. ls
> can pause for up to two minutes. connecting via appletalk takes ages, and
> the macs which habitually ask for full directory listings over and over
> again can really get stuck on this. But I dont think its an appletalk
> problem.

MacOS/9 or MacOS/X ? NetATalk 1.6.x or NetATalk 2.0 ?

I've never had particularly fantastic results with large directories and
NetATalk, but never had any serious problems either. Then again, my idea
of a "large directory" is 300 files/directories.

First, make certain all macs have the 'count items in subfolders' or
whatever it's called option *off*. It'll slow some things to a crawl.

If you have MacOS/X clients (or preferably, just anyway), upgrade to
NetATalk 2.0 if you're not there already. In my experience it's vastly
more reliable, faster, and behaves much better with OS/X. This is doubly
true if built with Berkley DB transaction support (but Debian's BDB
doesn't seem to enable this so you'd need to build a copy private to
NetATalk). Note that converting to NetATalk is painless if done
carefully, but should be preceeded by a FULL backup of all your NetATalk
volumes - the format has changed and must be converted.

> The drive runs ext3 with a journal , 7200rpm , snappy etc.

I'm assuming you're using ext3, since you haven't specified otherwise
and that's the general default. Recent ext3 work has apparently sped up
large directories significantly, but even so it might be worth doing
some trials with reiserfs and possibly other file systems. The file
system can make a significant difference.

Also remember that for NetATalk access, NetATalk has to look up the CNID
for each file when it sends it to the client. Because of that, you want
to ensure your Berkley DB configuration is up to the task (perhaps
increase the cache, etc).

> Any idea what might be the cause of this? It really causes alot of
> problems.

More info. Is it just AppleTalk listings, or is it slow over SMB and/or
local file listings too? What filesytem? etc.

-- 
Craig Ringer