File sync with rclone and rsync.net
The rsync.net service isn't the cheapest way to store files, but it's arguably the most versatile. You get access to a big filesystem on their servers via SFTP. That means you get to access the files as files, and use any tool that can deal with with shipping files over SSH. Auth is also based on SSH keys which is easy to provision on new machines.
Also, rsync.net has been in business for decades.
Until now I've been using rsync.net for ad hoc backups with FileZilla and rsync scripts. I wanted to explore something closer to a live file-sync experience. And I need it to work across multiple computers.
Enter rclone's new (and still beta) bisync command.
Sharing the Documents Folder
Most of my life is programming, so a great deal is backed up and restorable from git servers. But the ~/Documents folders on each of my machines is a little out of control. And they each have different stuff that I don't want to lose track of.
The initial bisync from a new computer uses the --resync-mode newer flag, which will build (and sync) a superset of the files on the local computer and the remote (a folder on rsync.net in my case).
I also specify --dry-run the first time to see what will be created on each end of the sync.
rclone bisync \
~/Documents rsync.net:sync/Documents \
--create-empty-src-dirs \
--resync-mode newer \
--dry-run
If that all looks good, we remove the --dry-run flag.
rclone bisync \
~/Documents rsync.net:sync/Documents \
--create-empty-src-dirs \
--resync-mode newer
Note: the "rsync.net" referenced in this command is the name of my remote created with rsync config, not a domain name.
There are other variations of "resync", but none of them do deletes. They build a superset of all the file paths, and with --resync-mode newer the modtime is used to pick a "winner" in the case of duplicate file paths. This is good enough for me, but read the docs carefully.
After doing an initial resync on each machine's ~/Documents folder, subsequent commands are the same, but without the resync flag.
rclone bisync \
~/Documents rsync.net:sync/Documents \
--create-empty-src-dirs
Run that on each machine under your user, and even deletes and renames will propagate.
Failure Modes
This is a beta feature, so be warned. Furthermore, there are some failure modes to be aware of:
- Modifying a file during a sync is asking for trouble. The rclone devs have written code to try and detect this and abort to prevent data loss, but there is no guarantee your new changes will propagate.
- Syncing directories where a majority of files are changing on every sync is possible, but it doesn't seem like rclone is ready for that.
Ready for Cron?
I'm on the fence about writing a cron job for automatic syncing. I'm going to roll with triggering a manual sync until I get the hang of things. In particular, I want to manually explore the many flags rclone has to offer to handle failure modes, conflict resolution, retries, etc.
The good news is that rsync.net keeps 7 days of snapshots I can recover from to avoid total catastrophe.