[plug] ext4 fragmentation

[Brad Campbell]

G'day all,

I'm approaching my 6 yearly HD replacement on my bulk storage array. 
Current array is 14 x 2TB drives in a RAID6 with an ext4 filesystem.
This array was re-built with a fresh format in 2013, so it has about 4 
years heavy use on the filesystem.

I can stream direct from the md block device at about 1.2GB/s, but I was 
finding when copying files across the network I was getting about 
~150MB/s (tar and netcat). I tried streaming some larger files (~1TB) to 
/dev/null and saw the same transfer rates. So 1.2GB/s from the array and 
150MB/s from the filesystem. What gives?!?

Fragmentation. You know, that thing _everyone_ says just isn't a problem 
on Linux? Twaddle.

I ran e4defrag over a directory of video files (~1.5TB). This took 2 
days, but afterwards the average transfer rate (tar to /dev/null) went 
from about 150MB/s to 500MB/s. The 1TB file mentioned above went from 
~150MB/s to ~600MB/s.

Having said that, the 1TB file copied across to the replacement array 
was giving me ~900MB/s until I used e4defrag on it. Now it's down to 
500MB/s. So things are not perfect in defrag land. None the less, 
fragmentation does become an issue on heavily used, mostly full 
filesystems over a long period of time, and e4defrag does have tangible 
benefit in helping clean this up.

Now I wish someone had a good offline defragger that planned ahead and 
really reorganized the drives like they do for FAT and NTFS. Even as a 
once every couple of year thing. On large arrays you just can't copy the 
files somewhere else, then copy them back to defragment them.

Just more rambling as I come across stuff that I think is worth mentioning.