[plug] ftp scripting?

[Russell Steicke]

On Wed, May 21, 2003 at 04:00:12PM +0800, Brad Hill wrote:
> Can anyone think of a nice easy way to download files that are sequentially
> numbered and only get those that you haven't already got?

A medium complexity shell script using wget could do this.  For the list
of files, use

  wget -O - --quiet ftp://a.b.c/dir/

and pipe that to some awk magic to extract the names.


> As an added complication, i'll be processing the files as they get
> downloaded and removing them from the download folder so i can't just do a
> simple wget --no-clobber... have to keep track of the number it's up to
> somehow.

Have a directory in which you touch a file of the same name as the file
you've processed.  When you do the listing again, if a file is in the
touch directory, but not in the listing, delete the touch file so you
know you have to process it again when it next appears.


> The files are basically 00000000.txt through to 99999999.txt (i'm guessing
> they'll loop back around to 0 once they get too high)
> 
> Also i really should allow for if a file gets skipped on the server (ie 45
> isn't there but 46 is), or if those skipped files appear at a later date
> (ie, i've retreaved up to number 50, but 45 now becomes available).

The touch file thing will cater for this.


> As an added confusion, i don't really want partially downloaded files
> appearing in my download directory, or at least if they are partially
> downloaded, lock them in some way so my processing programm can't get at
> them untill they're unlocked.

Download them to a tmp directory.  When wget exits without error, mv
(not cp) the file to the real directory.  If the directories are on the
same filesystem, this is an atomic operation.





-- 
Russell Steicke

-- Fortune says:
Take an astronaut to launch.