scp-turbo.sh (replay)
A new iteration on my old idea (half-broken) /r/bash/comments/1ajkmvh/scpturbo/ of a tar-based scp replacement primarily for making faster backups over ssh, especially when you have tens of thousands of small files which takes forever over scp/rsync. After hours of trial and error my most wanted key requirement was added: the --sudo flag which lets you tar huge folders (bigger than the amount of free space on your disk) directly to a stream that is saved on your local machine, compressed to a single file or uncompressed. The destination can be any combination of remote vs local, similar to rsync/scp.
Am I reinventing the wheel? I've searched for existing tools that could do this but didn't find any. Or maybe I didn't know what to search for?
The twist of this script: --sudo requires a passwordless sudo on the target box; I am not comfortable with having no root password on world visible boxes so added an alternative workaround when you DO have to interactively get sudo: --tmux-sudo; the idea is that you manually create a tmux session as your regular user on the remote, do sudo -i inside the session and then detach from it (ctrl+b > d); after this scp-turbo --tmux-sudo remote42:/etc backup-etc-remote42.gz should be able to pick the existing tmux session to run your backup as root with no password prompting (more detailed explanation in --help). Not sure how portable/reliable it is but got it working with a regular ubuntu remote. Is this too crazy? Or am I trying to solve a problem that does not exist?
latest final (?) version: https://gist.github.com/glowinthedark/e56fed33be889cea6ec2326dc33f535d
the key takeaway: all this can be done directly with tar+ssh, the script just wraps the invocation and builds the command line, if you run the script with --verbose --dry-run source target then the script will spit the actual raw tar+ssh command line that you can run directly, which reads pretty much like vogon poetry.