Move gobs of small files quickly with ssh and "|" in one go

If you need to copy a lot of files to another computer in Linux, you can use scp with the "r" and "C" flags.

"r" makes it copy recursively and "C" makes it compress the files before sending them, which saves a bit of bandwidth. However the overhead of using scp for copying many small files is huge, at least an order of magnitude (10 times), maybe two, bigger on my machine transporting a couple of thousand of files. One solution is to use tar and first create a tarball, and then copy  that over.

However sometimes you may not have space on the source device to make a tarball. In that case you can tar it "on the fly" and pipe it to not scp, but ssh. It turns out that you can supply commands to ssh already on login:

tar zcf - SOURCEDIR | ssh user1@remotehost "cat > /DESTDIR/DESTFILE.tar.gz"

If you are not already authorized with keys on the remote host, it will prompt you with a password.

I pretty much stole the command above from:
Lâmôlabs » Pushing & Pulling Files Around Using tar, ssh, scp, & rsync

I just omitted the "v" flag, since even though you can still supply your password, the prompt gets buried in verbose output with the "v" flag.

And if you do not want a tar ball as the end result on the remote computer, you can unpack it on the fly:

localhost% tar zcf - SOURCEDIR | ssh user1@remotehost tar zxvf -

Again taken from the Lâmôlabs page, again omitting the "v" flag.