[plug] speed: find vs ls

[Onno Benschop]

Interesting observation.

The glob with ls will only have to check 170 files (or as many as match the
glob if there are more directories), where as you say find does not. It
would be interesting to see if using *find . -type f | fgrep
bkb.rhash.crc32 *gives a quicker response.

That said, I've had the exact opposite experience with a massive directory
with log files where find . -type f -print0 gave me a file list MUCH faster
than ls, to the point where bash ran out of resources.

The point being that finding a solution is often specific to each unique
problem.

Thank you for sharing.

We should do more of that here.
--
finger painting on glass is an inexact art - apologies for any errors in
this scra^Hibble

()/)/)() ..ASCII for Onno..

On Thu, 28 July 2022, 17:22 Brad Campbell, <brad at fnarfbargle.com> wrote:

> G'day all,
>
> An observation while I'm still playing with my sizeable set of backup
> directories.
> I've been adding a bit that creates a file of crc32s of the updated files,
> and then toying around with a script to crawl the drive and check them all.
>
> I started using find to give me a list of dirs that contain the files. It
> was spending a *lot* of time just creating the list. In fact it spent more
> time looking for the files than the subsequent iteration and check of each
> one.
> I must qualify that with the fact, I'm about 10 days into creating the
> crcs and most directories already have ~800 days worth of backups.
>
> The script run with :
> for j in `find . -maxdepth 2 -type f -name bkb.rhash.crc32 -printf "%h\n"`
> ; do
>
> Checked 170 directories with 0 errors in 0:00:34:58
>
> stracing find, it's dropping into each directory and performing a stat on
> every file. Some dirs have a *lot* of files.
>
> I thought about trying a bit of globbing with ls instead, and blow me down
> if it wasn't "a bit faster".
>
> The script run with :
> for j in `ls ??????-????/bkb.rhash.crc32 2>/dev/null` ; do j=($dirname $j)
>
> Checked 170 directories with 0 errors in 0:00:09:49
>
> I know premature optimisation is the root of all evil, but this one might
> have been a case of "using the right tool".
>
> Regards,
> Brad
> --
> An expert is a person who has found out by his own painful
> experience all the mistakes that one can make in a very
> narrow field. - Niels Bohr
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20220728/63a20fdc/attachment.html>