Anyone who has used the SFTP command line client to do bulk transfers has probably run into the problem that it will not copy or get files recursively. That means, it won’t spider into directories. It just gets files in the current (or specified) directory.
The normal solution is to use scp -r instead, since they both us the SSH protocol. However, I’ve run into a problem with Gippy hosting where they only allow SFTP and not scp. Of course, I could use a GUI client. But GFTP crashes when copying bulk files (common error in Ubuntu Edgy / Feisty / Gutsy), and kftpgrabber refuses to connect.
Enter lftp. It doesn’t actually recursively copy as much as mirror. But it mirrors the directories you tell it, which is effectively a recursive copy.
Usage:
lftp sftp://user@host
mirror remote-dir local-dir
A full usage list of the mirror options is:
mirror [OPTS] [source [target]]
Mirror specified source directory to local target directory. If target directory ends
with a slash, the source base name is appended to target directory name. Source and/or
target can be URLs pointing to directories.
-c, --continue continue a mirror job if possible
-e, --delete delete files not present at remote site
--delete-first delete old files before transferring new ones
--depth-first descend into subdirectories before transferring files
-s, --allow-suid set suid/sgid bits according to remote site
--allow-chown try to set owner and group on files
--ascii use ascii mode transfers (implies --ignore-size)
--ignore-time ignore time when deciding whether to download
--ignore-size ignore size when deciding whether to download
--only-missing download only missing files
--only-existing download only files already existing at target
-n, --only-newer download only newer files (-c won't work)
--no-empty-dirs don't create empty directories (implies --depth-first)
-r, --no-recursion don't go to subdirectories
--no-symlinks don't create symbolic links
-p, --no-perms don't set file permissions
--no-umask don't apply umask to file modes
-R, --reverse reverse mirror (put files)
-L, --dereference download symbolic links as files
-N, --newer-than=SPEC download only files newer than specified time
--on-change=CMD execute the command if anything has been changed
--older-than=SPEC download only files older than specified time
--size-range=RANGE download only files with size in specified range
-P, --parallel[=N] download N files in parallel
--use-pget[-n=N] use pget to transfer every single file
--loop loop until no changes found
-i RX, --include RX include matching files
-x RX, --exclude RX exclude matching files
-I GP, --include-glob GP include matching files
-X GP, --exclude-glob GP exclude matching files
-v, --verbose[=level] verbose operation
--log=FILE write lftp commands being executed to FILE
--script=FILE write lftp commands to FILE, but don't execute them
--just-print, --dry-run same as --script=-
--use-cache use cached directory listings
--Remove-source-files remove files after transfer (use with caution)
-a same as --allow-chown --allow-suid --no-umask
When using -R, the first directory is local and the second is remote. If the
second directory is omitted, base name of first directory is used. If both directories
are omitted, current local and remote directories are used. If target directory
ends with a slash (except root directory) then base name of source directory is
appended.
RX is an extended regular expression, just like in egrep(1).
GP is a glob pattern, e.g. `*.zip'.
Include and exclude options can be specified multiple times. It means that a file
or directory would be mirrored if it matches an include and does not match to excludes
after the include, or does not match anything and the first check is exclude.
Directories are matched with a slash appended.
Note that symbolic links are not created when uploading to remote server, because
ftp protocol cannot do it. To upload files the links refer to, use `mirror -RL'
command (treat symbolic links as files).
For option --newer-than you can either specify a file or time specification like
that used by at(1) command, e.g. `now-7days' or `week ago'. If you specify a file,
then modi- fication time of that file will be used.
Verbosity level can be selected using --verbose=level option or by several -v options,
e.g. -vvv. Levels are:
0 - no output (default)
1 - print actions
2 - +print not deleted file names (when -e is not specified)
3 - +print directory names which are mirrored
--only-newer turns off file size comparison and uploads/downloads only
newer files even if size is differ- ent. By default older files are transferred
and replace newer ones.
You can mirror between two servers if you specify URLs instead of directories.
FXP is used automatically for transfers between ftp servers, if possible.
Some ftp servers hide dot-files by default (e.g. .htaccess), and show them only when
LIST command is used with -a option. In such case try to use `set ftp:list-options -a'.
