Selfhosted

That part threw me off. Last time i used it, I did incremental backups of a 500 gig disk once a week or so, and it took 20 seconds max.

Compared to something multi threaded, yes. But there are obviously a number of bottlenecks that might diminish the gains of a multi threaded program.

Hardlinks need to be on the same filesystem, don't they? I don't see how that would work with a remote backup...?

I couldn't tell you if it's the right way but I used it on my Rpi4 to sync 4tb of stuff from my Plex drive to a backup and set a script up to have it check/mirror daily. Took a day and a half to copy and now it syncs in minutes tops when there's new data

yes, it's the right way to go.

rsync over ssh is the best, and works as long as rsync is installed on both systems.

It depends

rsync is fine, but to clarify a little further...

If you think you'll stop the transfer and want it to resume (and some data might have changed), then yep, rsync is best.

But, if you're just doing a 1-off bulk transfer in a single run, then you could use other tools like xcopy / scp or - if you've mounted the remote NAS at a local mount point - just plain old cp

The reason for that is that rsync has to work out what's at the other end for each file, so it's doing some back & forwards communications each time which as someone else pointed out can load the CPU and reduce throughput.

(From memory, I think Raspberry Pi don't handle large transfers over scp well... I seem to recall a buffer gets saturated and the throughput drops off after a minute or so)

Also, on a local network, there's probably no point in using encryption or compression options - esp. for photos / videos / music... you're just loading the CPU again to work out that it can't compress any further.

@calliope It’s also great for local or remote backups over ssh, smb, etc.

I feel this too. I have a couple of "spells" that work wonders in a literal small notebook with other one liners over the years. Its my spell book lol.

I’m not super familiar with Syncthing, but judging by the name I’d say Syncthing is not at all meant for backups.

Syncthing is technically to synchronize data across different devices in real time (which I do with my phone), but I also use it to transfer data weekly via wi-fi to my old 2013 laptop with a 500GB HDD and Linux Mint (I only boot it to transfer data, and even then I pause the transfers to this device when its done transferring stuff) so I can have larger data backups that wouldn't fit in my phone, since LocalSend is unreliable for large amounts of data while Synchting can resume the transfer if anything goes wrong. On top of that Syncthing also works in Windows and Android out of the box.

its for a different purpose. I wouldn't use syncthing the way I use rsync

yeah, more often than not I notice the bottleneck being the storage drive itself, not rsync.

comparing seafile to rsync reminds me the old "Space Pen" folk tale.

You get incremental backups (snapshots) by using

--link-dest=DIR         hardlink to files in DIR when unchanged

To use this you pass in the previous snapshot location as DIR and use a new destination directory for the current snapshot. This creates hard links in the new snapshot to the files which were unchanged from the previous snapshot, so only the new files are transferred, and there is no duplication of data on disk (for whole-file matches).

This does of course require that all of the snapshots exist in the same filesystem, since you cannot hard-link across filesystems.