[plug] mirroring and updating a remote http directory

[James Devenish]

In message <41130233.7040206 at jensz.id.au>
on Fri, Aug 06, 2004 at 11:59:47AM +0800, Ben Jensz wrote:
> http://www.vet.com.au/files/libad/sigs/ is the URL.

Hmm...I wonder if VET are intentionally trying to be unhelpful, or
whether their server configuration is based on some policy against
unofficial mirrors. Their headers include:

    Cache-Control: no-cache, must-revalidate
    Connection: close
    Pragma: no-cache

(And no Last-Modified.)

> I figured [it's] probably the easiest way of figuring out which is the
> latest def, have the files locally first.  Rather than trying to grep
> out the latest filename out of the HTML directory listing.

Will you *ever* need to download a file twice? If you can effectively
presume "no", then you need not worry about timestamps: you need only
worry about existence. Then, wget has an -nc option which (if I'm right)
will make it skip over filenames that already exist locally.