Data Hoarder

1 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 1 year ago

MODERATORS

communick@selfhosted.forum

Need a much more convenient way to manage multiple copies of backups (alien.top)

submitted 11 months ago by SuperElephantX@alien.top to c/datahoarder@selfhosted.forum

8 comments fedilink hide all child comments

I truly understood the 3-2-1 backup rule and I'm trying to stick with it. The thing bothers me is that, those drives are all cold storage drives and I only connect those when I have new stuff to backup.

Now I have different sets of data with different levels of backup safety, and it's really hard to manage. I use SyncBackPro to do direct sync of the same copy (In different drives) to keep the data in sync. Since Raid is not a backup, I'd not choose the Raid1 mirroring solution. My drives have historically reasons to be in different sizes... (sigh)

Now the cold storage disks are laying around waiting for me to manually sync data sets one by one, and I already have a hard time keeping track which sets of copy is already on which disk. (I can only connect 2 drives at the same time, maybe NAS could help?)

How can I easily manage the backup synchronization and if not, at least having a better way to mark down which set is already on drive A, B, C at the same time? I hate updating data and synced between drive A and B, then mark it on the folder to remind myself to sync to drive C when I connect drive C in the future? Also hate manual data tracking with excel...e.g

In drive A (folder names I'm using):
Data 001 - [A, B] -> The data set exist in drive A and drive B
Data 002 - [A, B, C]
Data 003 - [A, B, D]
Data 004 - [A, C, D]

In drive B
Data 001 - [A, B]
Data 005 - [B, C, D]

top 8 comments

sorted by: hot top controversial new old

[–] Malossi167@alien.top 1 points 11 months ago (1 children)

Automating and streaming backups is definitely the way to go. Otherwise, it is really likely your backups are outdated and/or incomplete.

Please add the capacity of your drives. Upgrading to fewer, high capacity drives can be well worth it.

[–] SuperElephantX@alien.top 1 points 11 months ago

As the cold storage drives only connect from time to time, wouldn’t it be challenging for any automations to work in this situation? I definitely want to automate it but it might require some ledgers logging or something else for pending sync tasks.. e.g when drive A and B simultaneously plugged in, sync these set of data.. etc

[–] hobbyhacker@alien.top 1 points 11 months ago

at this complexity you should use a proper backup software instead of manual syncing.

[–] TADataHoarder@alien.top 1 points 11 months ago (1 children)

Seems like a complete mess. Not sure what to tell you.
List the capacities for your drives and the total space used for each set of data.

How big is Data 001? Data 002? etc.
What is Drive A? What is Drive B? etc.

I can only connect 2 drives at the same time, maybe NAS could help?

Look for a USB DAS instead. Maybe a multi-bay docking station like a 5-bay that can let you plug everything in at once. Much simpler. A NAS would require reformatting your drives. Not bad for the future, just not ideal for right now IMO. Simplify how you access the drives you have in use before going to a NAS. They're much slower and usually limited by 1Gbps LAN.

[–] SuperElephantX@alien.top 1 points 11 months ago (1 children)

Thank you for your feedback! I've made the changes to the post as you suggested, and I also included an image of the Excel sheet I created. Take a look and let me know if there are any further improvements I could have made.

[–] TADataHoarder@alien.top 1 points 11 months ago (1 children)

It looks like all of your data fits on the 6TB and be backed up to your two 4TBs. Why aren't you doing that?
That would ensure an easy and simple way to maintain 3 copies.

[–] SuperElephantX@alien.top 1 points 11 months ago

The thing is that, I got way more data (estimating maybe 3TB more) of data to be 3-2-1 backed, thus 3x3=9TB total space. My setup here can barely handle 2 sets of my important data sadly.. I'm have to purchase some large disks in order to do that, thus the problem goes back to management (too many disks already)

[–] q1525882@alien.top 1 points 11 months ago

This reminded me how I had to create montly view spreadsheet, because without such view I was losing track of when and which backup runs. Macrium Reflect.