[plug] evil performance on large directories. [RANT]

[Nick Bannon]

On Tue, Apr 05, 2005 at 09:27:33PM +0800, Cameron Patrick wrote:
[ext3 dir_index/htree]

Yep, with Linux 2.6, ext3 can have fast, large directories too.

The feature was developed on Linux 2.4 and you can patch it in, but it's
not in there by default. The cunning thing is that indexed directories
won't confuse an old kernel (but they won't be any faster, and they'll
lose their indexing if the old kernel changes them).

Unfortunately, if a program like "ls -l" has to stat(2) every file AND
the data isn't cached AND it doesn't happen to do it in inode order,
yes, it'll seek everywhere and be slow. (Or if you're opening a qmail
Maildir and you read the headers from every message file)
ext3+dir_index listings won't be sorted by inode, ReiserFS or other
FSes might be for a freshly filled directory, but that changes as files
get deleted or moved.

The program, or a LD_PRELOAD library, can take the time and memory
to deliberately sort the directory listings, the kernel can't,
unfortunately.

Check out:
https://listman.redhat.com/archives/ext3-users/2004-September/msg00026.html
http://www.linuxarkivet.se/mlists/ext3-users/0409/msg00025.html

If "ls -ifl" shows mostly increasing inode numbers in the lefthand
column, it will have been fast to stat(2) the files in that order. Try
compiling that library and compare it to:
LD_PRELOAD=./spd_readdir.so ls -ifl

[...]
> where /dev/hda3 should be replaced with the appropriate device.  It'll
> only take effect to any directories which you create _after_ adding
> the dir_index option.

...plus, fsck -D can go through and index all the old directories.

Shayne: Try this. If it's your root partition or something, it might
be easier from a Linux 2.6 live CD.

# umount /dev/xyz
# tune2fs -O dir_index /dev/xyz
# e2fsck -fD /dev/xyz
# mount /dev/xyz
 
Nick.

-- 
   Nick Bannon   | "I made this letter longer than usual because
nick-sig at rcpt.to | I lack the time to make it shorter." - Pascal