In Lithuania, healthcare e-services went down after the basement where the servers were kept got flooded in a rainstorm. They went down for a couple of weeks.
this post was submitted on 06 Oct 2025
330 points (99.1% liked)
Programmer Humor
26762 readers
2856 users here now
Welcome to Programmer Humor!
This is a place where you can post jokes, memes, humor, etc. related to programming!
For sharing awful code theres also Programming Horror.
Rules
- Keep content in english
- No advertisements
- Posts must be related to programming or programmer topics
founded 2 years ago
MODERATORS
dies after fall
Well, if that ain't a whitewashed headline.
So I have lived in South Korea for 6 years now. The fact that this fire has had such a major impact is quite typical of Korean bureaucracy and tech administration. Very few backups, infrastructure held together with scotch tape and bubblegum, overworked devs and maintainers. It's a bit sad, especially for a country that exports so many tech products.
Their explanation for having no backups was that 858 TB of data was "due to its large capacity". They stored eight years of data without backups. Even with systems where they had backups, it sounds like there's no redundancy – nobody can work because the single building where all the servers are located is currently out of order.
Sounds like the acute symptoms of chronic penny-pinching when it comes to IT infrastructure. I hope they take some good lessons from it at least. Just a shame that it's such a devastating way to learn.
Some moron deleted 75 TB of prod database the other day and sure that was catastrophic (for him, mostly) but it was backed up. We are a mid-size company, maybe a few hundred people across the country. I can't imagine the governement of freaking Korea, land of fiber years before everyone else, running so short on storage they can't do backups.
This shit is gonna go into It school books, like the OVH data center fire from 2018 (iirc)
"we can't ensure the data is safe because there's too much of it"
..sounds like an especially big reason to figure something out, huh? Not to mention, 858 TB isn't even that much for a whole ass government. For a consumer it might be 10$ per TB for a new drive, so it would be less when you're a government, which makes it just a bit under 10 000 USD for a full backup. That's it. Even if you budget in having to replace all drives once a year, 10 000 USD/yr is a bargain
New drives are more like 20€ per TB. Factor in redundancy with something like raid 5 and boom, off-site storage costs you a government toppling 20 000€ in hard drives.
I know this is programmer_humour, but can we express some condolence for the manager who ~~jumped~~fell from the building?
Vladimir Putin is to blame for every death in Russia. In South Korea, thoughts and prayers for unavoidable window sill slipperiness.
Ironically, the Govt. official is a resource they have enough backups for.
Can this now go to darkhumor@lemmy.world ?
Yeah, I agree I was chuckling till I saw the last article and audibly said oooooh. A real sobering moment.
Oh, that's more sad, I thought they had gotten the same treatment russians do.
This was propably some honor thing. Or avoidence of responsibilty. Or maybe taking one for the team as scape goat. I don't know. Nevertheless, it is sad.
Edit: If you or someone you know is feeling emotionally distressed or struggling with thoughts of suicide, you can find international contacts at https://befrienders.org/.
Oh, my... I had just skimmed so I had not noticed that until you pointed it out...
Bit of an overreaction. It may be a disaster recovery but its not that bad.
They apparently lost eight years of data, 858TB, with no backups at all because it's "too much". That's a disaster for sure, but I'm not so optimistic about there being a lot of recovery...
Literally who ever said "backups are overrated"?
I say that often. Especially after I forget to back something up and lose it.
Penny counters who don't like paying for storage
The people who laid off 85% of the IT dept.
My ex company had for more than 10 years keept all the data customers shared with us. Structured and standardized, should have been easy peasy.
Somehow they were "appending wrong" in some way and data was useless. In think they were trying to reduce the size by aggregating a bit, but they did in a way that rendered the data useless.
Of course the CEO wanted to train models with it anyway...
10 years and no one bothered to pull some information at random? I mean generally companies have a schedule of assessments to ensure records. Even if it's as simple as checksum.
The thing is they had data that expected to be slightly aggregated, do not a 1:1. The problem comes when you try to use the data for analysis and realize it didn't make any sense
I like train models
Everything is fine. Why would you need a backup?
Backups are always considered to be too expensive up until the point that not having backups becomes more expensive. This applies to redundancy of all kinds except the one that means firing employees for not setting up the other kinds.
Even at home. I do one usb backup and one internal backup of photos, home videos and documents. I would love to make backups of other stuff, but I can replace a lot of the other crap if need be, because hard drives kinda stalled in price drops.
The person in charge of allotting budget. "You want how many thousands for backup solutions? Here, take this flashdrive I picked up in the parking lot and use it for backups, that should be plenty enough. I mean, how many bytes can our data be? Two, three maybe?"
Apparently someone in the South Korean goverment xD
I doubt anyone said it verbatim, but it happens that they're deemed lower priority ad infinitum.
You met me at a very strange time in my life.
♫♫ Where Is My Mind? ♫♫ plays
In the background, explosions rip through buildings and they start to collapse.
If I had a nickel for every time someone didn't backup their datacenter, I'd have two nickels.
Which isn't a lot, but it's weird that it happened at least twice.
Last time we lost disks at work, there were full backups.
They were just in the same disks as the data. And because everything is abstracted two times into virtual disks on virtual machines, and containers and volumes, the people responsible for the backups didn't even know it.
But won't you like...check? That the backups are own their own drive? The whole 3-2-1 rule kinda make you want to check this, no?
Or was it like they knew where the drives of the backups were, but they didn't know those drives were being virtualized away and were in like production use?
I dunno what possibilities they actually had. But knowing the place, I can fully believe both that they weren't allowed to check and that they never bothered.
The most likely scenario in my head was that they sent a request to the provisioning team asking for the volume to be in a different disk, and that detail never made into the technician actually doing the work (that sits on the next chair, but the requests have to come from the system).
(And the long term backups were fine. We lost 3 days of data.)
If everything is serverless, hardware can't fail!
Maybe they stored backups on-site
You can't lose computerised services if you don't have computerised services. Checkmate SysAdmins.
I collect stories like this for when I need to make a case for purchasing new gear or services.