I'm planning on having a main storage system like 1U potentially, that will store the OS media, as well as flash storage cache drives for the mass media, and will control things such as network shares for other servers to connect to, netboot servers for all my other servers to boot from (so no storage mediums on my other servers) and so forth. The mass storage will be traditional hard drives on some form of JBODs. My current mission is to figure out what the base storage layout will be. All drives in the pool will be identical storage sizes, I am aiming for highest capacity, not as much redundancy (ex: 1 drive failure per pool, potentially 0, all critical data will be backed up, other data can be rebuilt from other sources) so I am trying to make decisions of RAID or equivalent (ex: RAIDZ) and filesystem (ex: btrfs, ext4, jfs, etc.) The array will be built on either 20tb or 22tb drives, potentially 40 or more. the setup needs to be capable of adding additional drives with minimal downtime. drive failure downtime is less of a concern. I haven't made any specific hardware decisions yet, based on not knowing what base storage control I will be using, because for example ZFS requires much more RAM overhead to perform properly vs. EXT4. I'm looking specifically for advice from people that have setup something this large, not just someone who is working on their 10 bay synology NAS or likewise. Your recommendations and reasonings behind your setup, and any shortcomings you have had with it would be greatly appreciated. (I wasn't sure if I should use the flair help or project, so I went with help)
I'd probably do 4 x 10 disk RAID Z2 vdevs. Probably just under 600TB useable with 20TB disks.
This all sounds like pipedream stuff though, to have that much storage requirements but not really have any idea of RAID, parity etc. and you're somehow also backing this up somewhere too.