[plug] evil performance on large directories.

[Shayne O'Neill]

Appletalk is uselesss. ls on the server is useless (a few minutes to
return on large directories). etc.

ext3. by large directory I mean 3-400 files.

--
 I wish a robot would get elected president. That way, when he came to
town, we could all take a shot at him and not feel too bad.
- Jack Handey (And now, Deep thoughts)

On Tue, 5 Apr 2005, Craig Ringer wrote:

> On Tue, 2005-04-05 at 12:49 +0800, Shayne O'Neill wrote:
> > The server I run at the murdoch guild has a couple of monster directories.
> > The shared student drive where our lovely student reps have promptly
> > dumped 10 million cubic tonnes of debris all into the root , and the www
> > directory (ugh. webdesigners... We now have a cms that puts an end to
> > that, but the debris is all there still)
> >
> > Regardless, any operations that require a listing seem to take ages. ls
> > can pause for up to two minutes. connecting via appletalk takes ages, and
> > the macs which habitually ask for full directory listings over and over
> > again can really get stuck on this. But I dont think its an appletalk
> > problem.
>
> MacOS/9 or MacOS/X ? NetATalk 1.6.x or NetATalk 2.0 ?
>
> I've never had particularly fantastic results with large directories and
> NetATalk, but never had any serious problems either. Then again, my idea
> of a "large directory" is 300 files/directories.
>
> First, make certain all macs have the 'count items in subfolders' or
> whatever it's called option *off*. It'll slow some things to a crawl.
>
> If you have MacOS/X clients (or preferably, just anyway), upgrade to
> NetATalk 2.0 if you're not there already. In my experience it's vastly
> more reliable, faster, and behaves much better with OS/X. This is doubly
> true if built with Berkley DB transaction support (but Debian's BDB
> doesn't seem to enable this so you'd need to build a copy private to
> NetATalk). Note that converting to NetATalk is painless if done
> carefully, but should be preceeded by a FULL backup of all your NetATalk
> volumes - the format has changed and must be converted.
>
> > The drive runs ext3 with a journal , 7200rpm , snappy etc.
>
> I'm assuming you're using ext3, since you haven't specified otherwise
> and that's the general default. Recent ext3 work has apparently sped up
> large directories significantly, but even so it might be worth doing
> some trials with reiserfs and possibly other file systems. The file
> system can make a significant difference.
>
> Also remember that for NetATalk access, NetATalk has to look up the CNID
> for each file when it sends it to the client. Because of that, you want
> to ensure your Berkley DB configuration is up to the task (perhaps
> increase the cache, etc).
>
> > Any idea what might be the cause of this? It really causes alot of
> > problems.
>
> More info. Is it just AppleTalk listings, or is it slow over SMB and/or
> local file listings too? What filesytem? etc.
>
> --
> Craig Ringer
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://www.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.linux.org.au
>