[plug] ftp scripting?

[Cameron Patrick]

On Wed, May 21, 2003 at 04:00:12PM +0800, Brad Hill wrote:
| Can anyone think of a nice easy way to download files that are sequentially
| numbered and only get those that you haven't already got?
| 
| As an added complication, i'll be processing the files as they get
| downloaded and removing them from the download folder so i can't just do a
| simple wget --no-clobber... have to keep track of the number it's up to
| somehow.
| 
| The files are basically 00000000.txt through to 99999999.txt (i'm guessing
| they'll loop back around to 0 once they get too high)

This shell script should do the trick, modulo any typos.  man seq may
help explain what's going on...

(for N in `seq -f%08d 0 99999`; do
  if [ ! -f "${N}.txt.done" ]; then
    wget ftp://server/path/${N}.txt || exit 1
    random_program ${N}.txt || exit 1
    mv ${N}.txt ${N}.txt.done
  fi
done)

| Also i really should allow for if a file gets skipped on the server (ie 45
| isn't there but 46 is), or if those skipped files appear at a later date
| (ie, i've retreaved up to number 50, but 45 now becomes available).

Maybe try changing the inside of the for loop to check whether the file
downloaded properly...

if wget ftp://server/path/${N}.txt; then
  random_program ${N}.txt || exit 1
else
  echo $N >>missing_files_list
fi

| As an added confusion, i don't really want partially downloaded files
| appearing in my download directory, or at least if they are partially
| downloaded, lock them in some way so my processing programm can't get at
| them untill they're unlocked.

Wget them to a separate directory, then mv them somewhere when they're
finished?

Cameron.