[plug] evil performance on large directories. [RANT]

[Shayne O'Neill]

Odd development. Sometime last night I had a fairly catastrophic hd crash.
After a few rounds of e2fsck I got her back up and shes running again
quite smoothly.

And oddly enough the directory listings are running reasonably fast again.
Its entirely possible that perhaps the hd was fragmented to hell and the
e2fsck fixed that (it took like about 1 1/2 hours on an 80gig drive!).

Either way, I'm thinking the drive might be overdue for a replacement to
something alot bigger, this time with reiser or something.

I'm still holding out for sarge for a full rebuild & latest 2.6 kernel.

--
 I wish a robot would get elected president. That way, when he came to
town, we could all take a shot at him and not feel too bad.
- Jack Handey (And now, Deep thoughts)

On Wed, 6 Apr 2005, Nick Bannon wrote:

> On Tue, Apr 05, 2005 at 09:27:33PM +0800, Cameron Patrick wrote:
> [ext3 dir_index/htree]
>
> Yep, with Linux 2.6, ext3 can have fast, large directories too.
>
> The feature was developed on Linux 2.4 and you can patch it in, but it's
> not in there by default. The cunning thing is that indexed directories
> won't confuse an old kernel (but they won't be any faster, and they'll
> lose their indexing if the old kernel changes them).
>
> Unfortunately, if a program like "ls -l" has to stat(2) every file AND
> the data isn't cached AND it doesn't happen to do it in inode order,
> yes, it'll seek everywhere and be slow. (Or if you're opening a qmail
> Maildir and you read the headers from every message file)
> ext3+dir_index listings won't be sorted by inode, ReiserFS or other
> FSes might be for a freshly filled directory, but that changes as files
> get deleted or moved.
>
> The program, or a LD_PRELOAD library, can take the time and memory
> to deliberately sort the directory listings, the kernel can't,
> unfortunately.
>
> Check out:
> https://listman.redhat.com/archives/ext3-users/2004-September/msg00026.html
> http://www.linuxarkivet.se/mlists/ext3-users/0409/msg00025.html
>
> If "ls -ifl" shows mostly increasing inode numbers in the lefthand
> column, it will have been fast to stat(2) the files in that order. Try
> compiling that library and compare it to:
> LD_PRELOAD=./spd_readdir.so ls -ifl
>
> [...]
> > where /dev/hda3 should be replaced with the appropriate device.  It'll
> > only take effect to any directories which you create _after_ adding
> > the dir_index option.
>
> ...plus, fsck -D can go through and index all the old directories.
>
> Shayne: Try this. If it's your root partition or something, it might
> be easier from a Linux 2.6 live CD.
>
> # umount /dev/xyz
> # tune2fs -O dir_index /dev/xyz
> # e2fsck -fD /dev/xyz
> # mount /dev/xyz
>
> Nick.
>
> --
>    Nick Bannon   | "I made this letter longer than usual because
> nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://www.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.linux.org.au
>