← Back to Toolbox
You can download
rclone from the official website. On macOS you can also install it with Homebrew:
brew install rclone.
Filtering mental model
The filtering model in
rclone was not immediately obvious to me, and misunderstanding can cause unwanted things to sync. Here’s how to think about it:
Each file and folder under the directory provided to
rclone as a CLI argument is run against each filter pattern in order, until a pattern is matched. At this point, the file is included in the list, or excluded from the list, depending on the
- sign in front of the pattern.
Once a pattern matches,
rclone makes the decision and doesn’t look at subsequent patterns. So patterns don’t combine: once a file is excluded by a pattern, it cannot be included by subsequent patterns.
Although they look similar, patterns for
rclone are not 100% compatible with
rysc filtering rules. Some gotchas:
- to match a file in the root folder, use
file.txt. The latter matches
file.txtin all parts of the file tree. For peace of mind, literally just start any pattern with
- As opposed to how globstar (
**) works in other places, in
folder/**/index.phppattern will not match
- folder contents needs to be specified as
folder/**, not simply
Recipes for syncing things
Sync a website to a remote host via (S)FTP
myremotehost FTP host is defined using the
rclone config command, which starts an interactive configuration process.
💡 Pro tip: Before performing a potentially destructive and/or irreversible operation, it is advisable to first test it with the
--dry-runoption, which prints out what would happen when the command is run without the flag.
To copy the content of the
my-stuff folder into the
their-stuff folder on the
myremotehost FTP host:
rclone copy my-stuff myremotehost:their-stuff --progress --ftp-concurrency 4 --checkers 2 --transfers 2 --exclude-from=".rclone-exclude"
In the command above, the
--exclude-from option prevents some files from being copied over. The files are defined as glob patterns, similar to
To copy a file regardless of timestamp, you can use the
rclone copy my-file.txt myremotehost:their-stuff --progress --ignore-times
FTP servers are a heterogenous bunch, and occasionally a server will advertise support for a feature, but then break when the client tries to use said feature. For these situations,
rclone has a bunch of
--ftp-disable-* flags you can use on your FTP connections.
You can get a sense of what is causing the transfer to fail or hang by using the
--dump headers --dump bodies pair of options, which instruct
rclone to print out the communication between the FTP client and server. For example, if the server hangs at the EPSV command, we can use the
--ftp-disable-epsv flag to disable the feature:
rclone copy my-stuff myremotehost:their-stuff --ftp-disable-epsv --progress --ftp-concurrency 4 --checkers 2 --transfers 2 --exclude-from=".rclone-exclude"
Do a dry-run to see what files are about to be synced
--dry-run flag produces output that is too verbose to scan. To extract the list of filenames, we can use
rclone --dry-run ... | rg 'NOTICE: ([^\s]+):' -or '$1' | sort