Ask Lemmy
A Fediverse community for open-ended, thought provoking questions
Rules: (interactive)
1) Be nice and; have fun
Doxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them
2) All posts must end with a '?'
This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?
3) No spam
Please do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.
4) NSFW is okay, within reason
Just remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com.
NSFW comments should be restricted to posts tagged [NSFW].
5) This is not a support community.
It is not a place for 'how do I?', type questions.
If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.
6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !askusa@discuss.online
Reminder: The terms of service apply here too.
Partnered Communities:
Logo design credit goes to: tubbadu
view the rest of the comments
Oh I got this. Real good fuck up. So, I run a bunch of services for me and family and friends, like Nextcloud, Immich, Mastodon, etc. All of their data is stored on a big RAID array built with mdadm. I built it before I knew about dm-integrity, so I wanted to rebuild it with dm-integrity running on all the disks. So I took out one disk (it’s a RAID6, so it can stand losing a disk just fine), and added dm-integrity on top of it. When I tried to add it back to the array, mdadm was like, “that’s not a big enough device”. Ok, yeah, makes sense, dm-integrity takes up some space. So I took all the services offline and started a resize2fs to reduce from 22TB to 18TB. Well that was taking forever, so I canceled and ran e2fsck, no worries. I started a new resize2fs reducing from 22TB to 21.8TB, which should be good enough. On pass 4, it’s looking like it’s gonna take 8 DAYS. I can’t have my services offline for that long, so I cancel and run e2fsck. Thousands of errors. Oops.
At least it’s looking like all the errors are just in Mastodon’s cache files.
So now I’m moving all the files to an external drive in order to migrate the array properly. In its degraded state though, the array is painfully slow. At least my services are online.
So basically I risked my files (though I have a backup from about three days before all this), and gave myself an extra maybe ten days(?) of work.
Moral of the story, don’t ever use resize2fs unless you have time to wait.
I've done something like this before, but thankfully it only impacted services that I host for my own use, didn't affect any family and friends.
Btw, I've found the easiest (but not the cheapest) way to fix this is to simply buy bigger disks. Swap out each disk for a bigger one one-by-one, then resize the whole volume to fill the new disks. Resizing upwards is much faster than shrinking a volume.
I've never had a volume shrink operation work without errors, and yes it takes days if you have more than 4TB.