SirSpanky.com - The Secret Diary of James Pearce Aged 20-Something

Anyone who has used the SFTP command line client to do bulk transfers has probably run into the problem that it will not copy or get files recursively. That means, it won’t spider into directories. It just gets files in the current (or specified) directory.

The normal solution is to use scp -r instead, since they both us the SSH protocol. However, I’ve run into a problem with Gippy hosting where they only allow SFTP and not scp. Of course, I could use a GUI client. But GFTP crashes when copying bulk files (common error in Ubuntu Edgy / Feisty / Gutsy), and kftpgrabber refuses to connect.

Enter lftp. It doesn’t actually recursively copy as much as mirror. But it mirrors the directories you tell it, which is effectively a recursive copy.

Usage:

lftp sftp://user@host
mirror remote-dir local-dir

A full usage list of the mirror options is:

mirror [OPTS] [source [target]]

   Mirror specified source directory to local target directory.   If target directory ends
with a slash, the source base  name   is  appended  to target directory name. Source and/or
target   can be URLs pointing to directories.

-c, --continue      continue a mirror job if possible
-e, --delete        delete files not present at remote site
--delete-first       delete old files before transferring new ones
--depth-first        descend into subdirectories before transferring files
-s, --allow-suid         set suid/sgid bits according to remote site
--allow-chown   try to set owner and group on files
--ascii         use ascii mode transfers (implies --ignore-size)
--ignore-time        ignore time when deciding whether to download
--ignore-size        ignore size when deciding whether to download
--only-missing  download only missing files
--only-existing download only files already existing at target
-n, --only-newer    download only newer files (-c won't work)
--no-empty-dirs don't create empty directories (implies --depth-first)
-r, --no-recursion  don't go to subdirectories
--no-symlinks   don't create symbolic links
-p, --no-perms      don't set file permissions
--no-umask      don't apply umask to file modes
-R, --reverse       reverse mirror (put files)
-L, --dereference   download symbolic links as files
-N, --newer-than=SPEC    download only files newer than specified time
--on-change=CMD      execute the command if anything has been changed
--older-than=SPEC    download only files older than specified time
--size-range=RANGE   download only files with size in specified range
-P, --parallel[=N]  download N files in parallel
--use-pget[-n=N]     use pget to transfer every single file
--loop          loop until no changes found
-i RX, --include RX include matching files
-x RX, --exclude RX exclude matching files
-I GP, --include-glob GP include matching files
-X GP, --exclude-glob GP exclude matching files
-v, --verbose[=level]    verbose operation
--log=FILE      write lftp commands being executed to FILE
--script=FILE        write lftp commands to FILE, but don't execute them
--just-print, --dry-run   same as --script=-
--use-cache          use cached directory listings
--Remove-source-files    remove files after transfer (use with caution)
-a             same as --allow-chown --allow-suid --no-umask

   When using -R, the first directory is local and  the  second   is remote.  If the
second directory is omitted, base name of   first directory is used.  If both directories
are  omitted,   current  local  and  remote directories are used.  If target   directory
ends with a slash  (except  root  directory)  then   base name of source directory is
appended.

   RX is an extended regular expression, just like in egrep(1).

   GP is a glob pattern, e.g. `*.zip'.

   Include and exclude options can be specified multiple times.   It  means  that  a file
or directory would be mirrored if it   matches an include and does not match to excludes
after  the   include,  or  does not match anything and the first check is   exclude.
Directories are matched with a slash appended.

   Note that symbolic links are not created when  uploading  to   remote  server, because
ftp protocol cannot do it. To upload   files the links refer to, use `mirror  -RL'
command  (treat   symbolic links as files).

   For  option  --newer-than  you  can either specify a file or   time specification like
that used  by  at(1)  command,  e.g.   `now-7days' or `week ago'. If you specify a file,
then modi-   fication time of that file will be used.

   Verbosity level can be selected using --verbose=level option   or by several -v options,
e.g. -vvv. Levels are:
0 - no output (default)
1 - print actions
2 - +print not deleted file names (when -e is not specified)
3 - +print directory names which are mirrored

   --only-newer    turns   off   file   size   comparison   and   uploads/downloads only
newer files even if size  is  differ-   ent.  By  default  older  files  are transferred
and replace   newer ones.

   You can mirror between  two  servers  if  you  specify  URLs   instead  of  directories.
FXP  is  used  automatically for   transfers between ftp servers, if possible.

   Some ftp servers hide dot-files by default (e.g. .htaccess),   and show them only when
LIST command is used with -a option.   In such case try to use `set ftp:list-options -a'.

This entry was posted on Thursday, May 1st, 2008 at 1:40 pm and is filed under Informative. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.