[plug] What is the maximum file size allowed under Unix

[Grahame Bowland]

On 01 Aug 2001 19:29:10 +0800, James Bromberger wrote:

> I once had a bad kernel create a similar number of ksymoops files in 
> /var/log/ksymoops. Commands such as "ls", "rm", etc, would fail to 
> be able to show or remove the contents. The only idea I had that it was 
> ull in here was that the *directory* /var/lib/ksypoops had a size of several 
> hundred meg (ie, not the contents). The only solution that worked was
> 	find /var/lib/ksymoops -name ksym\* -exec rm {} \;
> 
> Since `rm *` gave a "too many files" on the expansion (globbing) of * in 
> the shell. *sigh* Indeed, a large number of files in an ext2 FS cna be slow. 
> I ran a test a while back with my Debian system: exact same number of files 
> in /usr/share/doc (1200 or so). One FS was ext2, one was (is) resierFS. Ext2 
> took around 5 seconds to show an ls. resier was instantaneous. There have 
> since been several patches proposed to ext2 code, but I dont think any of them 
> have been blessed into the Linux tree yet (though I am not sure of this).

The reason it's slow is that ext2 does a linear scan of the directory
for 
for the file you're looking for. So the average number of comparisons 
required to find your file is (FILES_IN_DIRECTORY / 2) - O(n). This
isn't 
particularly scalable when you have many thousands of files.

We had a kernel patch to the 2.0 series so that the kernel 
translated references to /var/spool/mail/username to
/var/spool/.u/[a-z]/username
and then all of the mailboxes stored in these subdirectories. Now that 
we've gone over to 2.2 we're using ReiserFS and all of the mailboxes
(15929 of them) in one directory. It works with better performance than
ext2 and the 
subdirectory hack.

Cheers,
Grahame

--
Grahame Bowland, Network and Systems Administrator
University Communications Services, The University of Western Australia
Phone: +61 8 9380 1175