Data Hoarder

1 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 1 year ago

MODERATORS

communick@selfhosted.forum

optimized simultaneous copy to multiple local destinations (alien.top)

submitted 11 months ago by mervincm@alien.top to c/datahoarder@selfhosted.forum

4 comments fedilink hide all child comments

I want to create a copy of my NAS hosted media folder that is about 30TB. I have a bunch of 4-8TB local (most USB3, some SATA) disks, and I would like to copy these files to the various destinations maximizing space used and time required. Since I have a 10GBE network, I can read data far faster than I can write to any of the destinations, so multiple simultaneous file copies is required at the same time to maximize this activity. doing this manually is painful. trying to select the maximum number of files that can fit (but not go over) each destination is a pain. Any thoughts on a script or an app I can use to assist here is appreciated. I want to leave the files in their native format, so I am looking for a file copy, not block-based backup etc.

you are viewing a single comment's thread
view the rest of the comments

[–] AzureCerulean@alien.top 1 points 11 months ago (1 children)

trapexit/mergerfs: a featureful union filesystem

mergerfs is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to mhddfs, unionfs, and aufs.

FEATURES

Configurable behaviors / file placement
Ability to add or remove filesystems at will
Resistance to individual filesystem failure
Support for extended attributes (xattrs)
Support for file attributes (chattr)
Runtime configurable (via xattrs)
Works with heterogeneous filesystem types
Moving of file when filesystem runs out of space while writing
Ignore read-only filesystems when creating files
Turn read-only files into symlinks to underlying file
Hard link copy-on-write / CoW
Support for POSIX ACLs
Misc other things

HOW IT WORKS mergerfs logically merges multiple paths together. Think a union of sets. The file/s or directory/s acted on or presented through mergerfs are based on the policy chosen for that particular action.

https://github.com/trapexit/mergerfs

[Users like you provide all of the content and decide, through voting, what's good and what's junk.]

[–] mervincm@alien.top 1 points 11 months ago

This is an interesting option but is it suitable for a one time copy?