Selfhosted

I'm far from an expert sorry, but my experience is so far so good (literally wizard configured in proxmox set and forget) even during a single disk lost. Performance for vm disks was great.

I can't see why regular file would be any different.

I have 3 disks, one on each host, with ceph handling 2 copies (tolerant to 1 disk loss) distributed across them. That's practically what I think you're after.

I'm not sure about seeing the file system while all the hosts are all offline, but if you've got any one system with a valid copy online you should be able to see. I do. But my emphasis is generally get the host back online.

I'm not 100% sure what you're trying to do but a mix of ceph as storage remote plus something like syncthing on a endpoint to send stuff to it might work? Syncthing might just work without ceph.

I also run zfs on an 8 disk nas that's my primary storage with shares for my docker to send stuff, and media server to get it off. That's just truenas scale. That way it handles data similarly. Zfs is also very good, but until scale came out, it wasn't really possible to have the "add a compute node to expand your storage pool" which is how I want my vm hosts. Zfs scale looks way harder than ceph.

Not sure if any of that is helpful for your case but I recommend trying something if you've got spare hardware, and see how it goes on dummy data, then blow it away try something else. See how it acts when you take a machine offline. When you know what you want, do a final blow away and implement it with the way you learned to do it best.