Linux

You're asking the right questions, and there have been some great answers on here already.

I work at the crossover between IT and digital preservation in a large GLAM institution, so I'd like to offer my perspective. Sorry of there are any peculiarities in my comment, English is my 2nd language.

First of all (and as you've correctly realizes), compression is an antipattern in DigiPres and adds risk that you should only accept of you know what you're doing. Some formats do offer integrity information (MKV/FFV1 for video comes to mind, or the BagIt archival information package structure), including formats that use lossless compression, and these should be preferred.

You might want to check this to find a suitable format here: https://en.wikipedia.org/wiki/List_of_archive_formats -> Containers and compression

Depending on your file formats, it might not even be beneficial to use a compressed container, e.g. if you're archiving photos/videos that already exist in compressed formats (JPEG/JFIF, h.264, ...).

You can make your data more resilient by choosing appropriate formats not only for the compressed container but also for the payload itself. Find significant properties of your data and pick formats accordingly, not the other way round. Convert before archival of necessary (the term is normalization).

You might also want to consider to reduce the risk of losing the entirety of your archive by compressing each file individually. Bit rot is a real threat, and you probably want to limit the impact of flipped bits. Error rates for spinning HDDs are well studied and understood, and even relatively small archives tend to be within the size range for bit flips. I can't seem to find the sources just now, but iirc, it was something like 1 Bit in 1.5TB for disks at write time.

Also, there's only so much you can do against bit rot on the format side, so consider using a filesystem that allows you to run regular scrubs and so actually run them; ZFS or Btrfs come to mind. If you use a more "traditional" filesystem like ext4, you could at least add checksum files for all of your archival data that you can then use as a baseline for more manual checks, but these won't help you repair damaged payload files. You can also create BagIt bags for your archive contents, because bags come with fixity mechanisms included. See RFC 8493 (https://datatracker.ietf.org/doc/html/rfc8493). There are even libraries and software that help you verify the integrity of bags, so that may be helpful.

The disk hardware itself is a risk as well; having your disk laying around for prolonged periods of time might have an adverse effect on bearings etc. You don't have to keep it running every day, but regular scrubs might help to detect early signs of hardware degradation. Enable SMART if possible. Don't save on disk quality. If at all possible, purchase two disks (different make & model) to store the information.

DigiPres is first and foremost a game of risk reduction and an organizational process, even of we tend to prioritize the technical aspects of it. Keep that in mind at all times

And finally, I want to leave you with some reading material on DigiPres and personal archiving on general.

• https://www.langzeitarchivierung.de/Webs/nestor/DE/Publikationen/publikationen_node.html (in German)
• https://meindigitalesarchiv.de/ (in German)
• https://digitalpreservation.gov/personalarchiving/ (by the Library of Congress, who are extremely competent in DigiPres)

I've probably forgotten a few things (it's late...), but if you have any further questions, feel free to ask.

EDIT: I answered to a similar thread a few months ago, see https://sh.itjust.works/comment/13922388