WikiBox

USB 3.1 Gen2 is hard to beat. 10Gbps. But for HDDs the HDDs themselves will be the bottleneck. For sustained file transfers, perhaps no more than up to 3Gbps, even if installed in the same PC. Good SATA SSDs may manage closer to 6Gbps. NVMe SSDs are likely to saturate USB 3.1 Gen2. Then PCI-e directly between installed drives is the fastest. Normal cabled network is only 1Gbps.

File transfers to/from external multibay USB DAS can be done in parallel, involving multiple HDDs, and come close to saturating 10Gbps USB3.1 Gen2 speeds.

Then there is thunderbolt.

Start a rsync transfer when you go to bed. In the morning it will most likely be done.

I use snapraid as one of my backup methods. Mainly for long term mostly static archive backups. Things that no longer change, but is added to, and I still want to have accessible read-only. Not for daily backups or for frequently changing files or folders, nor for "permanent" off-line cold storage.

https://www.snapraid.it/

I use 8 storage drives and two snapraid parity drives.

Using snapraid I can then easily verify that all backed up files are 100% OK, exactly as they were when I had just backed them up.

Snapraid can detect and fix bitrot (has never happened so far), undelete accidentally deleted files or folders and even recreate up to two failed drives.

When I backup/archive files, I simply copy them to one of the storage drives and then ask snapraid to update the parity.

Done!

One way to solve your storage needs might be a three tier system. PC, DAS and cloud/external.

On the PC you have two SSDs, say 4-8TB. You use SSD1 (4TB?) as usual for the OS and for current projects. You use SSD2 (8TB?) for backups of project files and documents on SSD1. Perhaps automatically update an incremental backup or snapshot sync every boot.

In a multibay DAS, "DAS1", you install three big 20TB HDDs. "HDD1P1" , "HDD1S1" and "HDD1S2". P=Parity, S=Storage.

Using separate timestamped project folders, with project start/end dates, you first fill HDD1S1 with the oldest projects and then continue on HDD1S2 with the rest. You write-protect each closed project to avoid unnecessary mistakes, deletes or modifications.

Then, using Snapraid, you record parity data for HDD1S1 and HDD1S2 on HDD1P1. This means that if any of HDD1S1 or HDD1S2 fail, or files are corrupted or deleted by mistake, you can use Snapraid and HDD1P1 and the still working HDD to recreate the contents of the failed HDD or the corrupt/deleted files.

Every time you finish a project you archive it on the DAS, write protect the project folder, and update the stored parity on HDD1P1 using Snapraid.

When HDD1S2 starts to fill up, you add HDD1S3 to the Snapraid and simply carry on. Assuming a 5 bay DAS, when HDD1S4 has filled up you add DAS2 and add HDD2P1 and HDD2S1 and carry on.

When you have filled DAS1 it is possible that you choose some other methods or at least start using 40TB SSDs in DAS2. You may need it because you by then may be using high resolution 360 autonomous multicam AI augmented 3D photogrammetry or perhaps even videogrammetry.

In addition to using SSD2 for backups and Snapraid for parity, you need at least one more level. Perhaps sync current project folders on SSD1 with cloud storage as well as with SSD2. Also store achieved projects not only on the DAS, with Snapraid, but also on loose external cold storage drives.

You might also consider using more than one parity drive per DAS. Especially if you use a DAS with many drives. Only parity drive for three storage drives is commonly used. For more storage, or if you are paranoid, you might want more parity. HDD1P2. Or even two archive DAS.

Conveniently, Snapraid can create a read-only pool of all the drives in the DAS. Making it much easier to navigate, browse and search, than looking in separate drives.

I have very good experience with this 5 bay DAS: ICY BOX IB-3805-C31. I think it is also sold under the brand "Sabrent".

I use SSD1/SSD2 automatic backups/snapshots technique both on my PC and on my Laptop. I have two DAS, one for storage and one for archiving. Currently I use a mix of 12TB-18TB Exos and Ironwolf drives. Slowly replacing 12TB drives that are out of warranty and use them for extra cold archive. I use Snapraid with dual parity drives and also mergerfs. I also use an old remote NAS for some additional backup.

I use TMM. Tiny MediaManager.

It can be used for free with limited functionality. Or you can pay in order to get full functionality.

Very compressed low bitrate 4K may be much worse than less compressed high bitrate 1080p.

I prefer to collect REMUX copies of movies I really like.

Then you have to assume that the file already is at least a little corrupt. What you need to determine is if the level of corruption is so bad that it is a problem. If it crash an audioplayer or the file sounds bad. Ideally you have a backup copy that is better or you can download or create a new file that works OK. Possibly you can re-encode the file to fix problems. It will degrade audio quality if the encoding is lossy. But perhaps not enough to be noticeable.

Some audio file formats have embedded checksums. FLAC or WavPack. Perhaps more? You should be able to find utilites that can compare the embedded checksum with the current stored data.

In the future use a checksummed format or store separate checksums. Or zip the files. The zip format has embedded checksums. (Same with all(?) other compressed formats.)

It is important that you check to see if the file really is pristine after you take the first checksum. That you can read and use the file. Otherwise you may take a checksum of a file that is already corrupt. That is not very useful.

Most likely it is not actually spontaneous bitrot. It is much more likely that somebody made the file corrupt. Usually it is some form of user error.

It is a very good idea!

A friend. Make your friend hoard data. And share it with you.

Otherwise large HDDs, at least 18TB, are most likely still the cheapest per TB. If you mostly write once and read much, much more, then SMR is fine. As long as it isn't in a RAID array that may have to be rebuilt.

SSD is, I think, better in all ways, except price and capacity. So SSDs are very bad for bulk storage.

Multiple backup copies on different media, stored at different locations, is how you reduce the risks of data loss.

I use a 5 bay JBOD DAS ICY BOX IB-3805-C31. Not cheap but good. Used for shared media and backups.

Exos drives pooled using mergerfs. Backups on another DAS.

Perhaps thinks that RAID is a form of backup. It isn't...

As long as you don't know how fast your storage requirements will grow, you can't make a rational decision.

So do something temporary, until you do know.

Get one 16TB drive for your bulk storage. Use your other HDDs for backup. When the backup drives are full, get another 16TB HDD. Make a new decision later.