Selfhosted
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
view the rest of the comments
Buy used Samsung mSata or m.2 2230 drives on fleaBay. Stick with Samsung and other well known brands with decent spec sheets and warranties, that's the cheapest way to handle durable storage on a pi. USB enclosures are like $5-7 on AliExpress or fleaBay.
Buy MLC drives if you need higher endurance (check the model no and look up the datasheet.) TLC will usually be fine for a few years, MLC will last a bit longer. If you're killing drives faster than you expect buy larger (512 instead of 256GB),
blkdiscardthe entire device once it's installed and then only partition 60-80% of it. Never touch the rest of the freed storage and the drive controller should be able to use those blocks for wear levelling to reduce the NAND wear rate.Edit: One heads up, I usually buy used drives from eBay because their buyer protection is top tier, if there's anything wrong with the drive when it's delivered or when I test it it goes right back for a refund. This makes buying blind viable thanks to an easy return policy.
If you're sourcing used drives somewhere else insist on seeing SMART data before purchasing and don't buy heavily worn drives. Look at the drive model datasheet, find the warranted endurance of the drive (if it's a 512GB drive rated for 1 DWPD over 3y that means the rated endurance is ~ 0.5T * 365 * 3 or roughly ~550TB written over 3y. Pass on buying drives approaching their rated endurance, try to buy lightly used drives wherever possible and you shouldn't have problems with reliability.
whoa whoa, I would not recommend a cheap AliExpress USB enclosure at all. As others have already pointed out there's a whole ever-growing blacklist of partially incompatible enclosures that basically flake out whenever they feel like it. Worse yet, not every device is on the list so you frequently have to research and add devices yourself.
The last generic Inland m.2 enclosure I bought worked fine... for 1 hour. Then it disconnected and reconnected. I thought it was just random chance, until it happened again and again and again. Did the deep-dive research, found the chipset was partially incompatible and I had to return it.
DO NOT BUY CHEAP ENCLOSURES FOR EXTERNAL MEDIA ON RPI
My approach to this has always been to buy one enclosure and validate it, then go buy like 8 more after thorough testing. Obviously don't place an order for 10 units of an unknown tech item from AliExpress or you're looking at a bad time. Look for enclosures that use known good chipsets and there's not as much risk as you're expecting. I have something like 8 msata enclosures here that work flawlessly and another half dozen sata+nvme rtl9210b enclosures that also work well.
What about enclosures from Sabrent? That should be a reliable manufacturer, no?
Unfortunately I don't have any recommendations I can give you as each enclosure could use a different chipset. It seems that the brand does not have a good reputation for compatibility but that list is fairly old at this point. All I can say is if you find an enclosure you like, plug the model number into the raspberry pi forum and see if anybody had to add it to the quirks list.
I heard it's not so good practice to buy used drives so I will not do that for now but thanks for the second part I will consider that!
Since you mentioned MLC, maybe you have some suggestions for eg used server grade disks? Would the Rpi be able to run something like the Intel datacenter SSDs eg S3700? The power loss protection is really something I would like to have, especially in a homelab scenario. Or any other notable MLCs with larger capacities? I am having trouble finding a good list sorted by max potential storage.
Most enterprise drives are TLC these days, MLC just doesn't provide the storage density that enterprises require anymore. I only mentioned MLC because you'll occasionally find mSATA drives in the <=256GB range that use MLC. You have to check the datasheet for each model, look for endurance rated at 5DWPD or higher, those will typically be MLC or heavily over provisioned TLC. If you want enterprise drives with greater endurance than the usual 0.5 or 1 DWPD look for the over provisioned models with capacities like 400GB, 800GB, 1.6T or 3.2T. those are 512GB, 1TB, 2TB and 4TB raw capacity drives with a bunch of flash set aside for wear leveling purposes. You don't often see 300GB, 600GB, 1.2T or 2.4T drives anymore but those are often very high endurance (write intensive, 10 DWPD or so) models.
Check the datasheets for drives when you're shopping and you can get a pretty good idea of what their durability is like, I usually buy 1 DWPD drives for write occasional bulk storage and 3+ DWPD for anything with a serious write workload. You can also help the drive controller a bit by running blkdiscard against the entire device before partitioning, then only partition and use ~80% of available space. The drive controller will typically grab free unused blocks and use them for wear leveling but only if they've been marked free (TRIMmed) and never allocated after. If you can't find or can't afford high endurance drives you can usually buy a larger lower endurance drive and over provision it in this way to extend its lifespan.
(The last time MLC flash was really common was back in maybe 2014-2015, some of the older Samsung pro drives like the 850/860 pro were built using MLC. Those had legendary real world endurance, I think they'd get up to 10+PB written before actually failing. It's a shame they didn't have PLP because they would have made good budget array storage if they did.)
Legend, thank you!