rclone recipes

Published Jan 2, 2023

Installation

You can download rclone from the official website. On macOS you can also install it with Homebrew: brew install rclone.

Filtering mental model

The filtering model in rclone was not immediately obvious to me, and misunderstanding can cause unwanted things to sync. Here’s how to think about it:

Each file and folder under the directory provided to rclone as a CLI argument is run against each filter pattern in order, until a pattern is matched. At this point, the file is included in the list, or excluded from the list, depending on the + / - sign in front of the pattern.

Once a pattern matches, rclone makes the decision and doesn’t look at subsequent patterns. So patterns don’t combine: once a file is excluded by a pattern, it cannot be included by subsequent patterns.

Although they look similar, patterns for rclone are not 100% compatible with .gitignore or rysc filtering rules. Some gotchas:

to match a file in the root folder, use /file.txt, not file.txt. The latter matches file.txt in all parts of the file tree. For peace of mind, literally just start any pattern with /.
As opposed to how globstar (**) works in other places, in rclone the folder/**/index.php pattern will not match folder/index.php, only folder/subfolder/index.php.
folder contents needs to be specified as folder/**, not simply folder or folder/.

Recipes for syncing things

Sync a website to a remote host via (S)FTP

The myremotehost FTP host is defined using the rclone config command, which starts an interactive configuration process.

💡 Pro tip: Before performing a potentially destructive and/or irreversible operation, it is advisable to first test it with the --dry-run option, which prints out what would happen when the command is run without the flag.

To copy the content of the my-stuff folder into the their-stuff folder on the myremotehost FTP host:

rclone copy my-stuff myremotehost:their-stuff --check-first --progress --ftp-concurrency 4 --checkers 2 --transfers 2 --exclude-from=".rclone-exclude"

In the command above, the --exclude-from option prevents some files from being copied over. The files are defined as glob patterns, similar to .gitignore.

The --check-first flag instructs rclone to make a list of all files that need to be transferred and perform all the transfers at once, rather than transfer files as it finds them. This shortens the interval in which the website may exist in an inconsistent state when, for example, you update a PHP framework or your CMS.

To copy a file regardless of timestamp, you can use the --ignore-times flag.

rclone copy my-file.txt myremotehost:their-stuff --progress --ignore-times

FTP servers are a heterogenous bunch, and occasionally a server will advertise support for a feature, but then break when the client tries to use said feature. For these situations, rclone has a bunch of --ftp-disable-* flags you can use on your FTP connections.

You can get a sense of what is causing the transfer to fail or hang by using the --dump headers --dump bodies pair of options, which instruct rclone to print out the communication between the FTP client and server. For example, if the server hangs at the EPSV command, we can use the --ftp-disable-epsv flag to disable the feature:

rclone copy my-stuff myremotehost:their-stuff --ftp-disable-epsv --check-first --progress --ftp-concurrency 4 --checkers 2 --transfers 2 --exclude-from=".rclone-exclude"

Do a dry-run to see what files are about to be synced

The --dry-run flag produces output that is too verbose to scan. To extract the list of filenames, we can use Ripgrep:

rclone --dry-run ... | rg 'NOTICE: ([^\s]+):' -or '$1' | sort

Other tips

rclone GUI

Starting with v1.74.0, rclone bundles a web-based graphical interface, accessible with rclone gui.