this post was submitted on 31 Jul 2024
429 points (98.4% liked)

News

23301 readers
4659 users here now

Welcome to the News community!

Rules:

1. Be civil


Attack the argument, not the person. No racism/sexism/bigotry. Good faith argumentation only. This includes accusing another user of being a bot or paid actor. Trolling is uncivil and is grounds for removal and/or a community ban. Do not respond to rule-breaking content; report it and move on.


2. All posts should contain a source (url) that is as reliable and unbiased as possible and must only contain one link.


Obvious right or left wing sources will be removed at the mods discretion. We have an actively updated blocklist, which you can see here: https://lemmy.world/post/2246130 if you feel like any website is missing, contact the mods. Supporting links can be added in comments or posted seperately but not to the post body.


3. No bots, spam or self-promotion.


Only approved bots, which follow the guidelines for bots set by the instance, are allowed.


4. Post titles should be the same as the article used as source.


Posts which titles don’t match the source won’t be removed, but the autoMod will notify you, and if your title misrepresents the original article, the post will be deleted. If the site changed their headline, the bot might still contact you, just ignore it, we won’t delete your post.


5. Only recent news is allowed.


Posts must be news from the most recent 30 days.


6. All posts must be news articles.


No opinion pieces, Listicles, editorials or celebrity gossip is allowed. All posts will be judged on a case-by-case basis.


7. No duplicate posts.


If a source you used was already posted by someone else, the autoMod will leave a message. Please remove your post if the autoMod is correct. If the post that matches your post is very old, we refer you to rule 5.


8. Misinformation is prohibited.


Misinformation / propaganda is strictly prohibited. Any comment or post containing or linking to misinformation will be removed. If you feel that your post has been removed in error, credible sources must be provided.


9. No link shorteners.


The auto mod will contact you if a link shortener is detected, please delete your post if they are right.


10. Don't copy entire article in your post body


For copyright reasons, you are not allowed to copy an entire article into your post body. This is an instance wide rule, that is strictly enforced in this community.

founded 1 year ago
MODERATORS
 
  • Delta Air Lines CEO Ed Bastian said the massive IT outage earlier this month that stranded thousands of customers will cost it $500 million.
  • The airline canceled more than 4,000 flights in the wake of the outage, which was caused by a botched CrowdStrike software update and took thousands of Microsoft systems around the world offline.
  • Bastian, speaking from Paris, told CNBC’s “Squawk Box” on Wednesday that the carrier would seek damages from the disruptions, adding, “We have no choice.”
you are viewing a single comment's thread
view the rest of the comments
[–] riskable@programming.dev 21 points 3 months ago (31 children)

Yeah... Maybe don't put all your IT eggs in one basket next time.

Delta is the one that chose to use Crowdstrike on so many critical systems therefore the fault still lies with Delta.

Every big company thinks that when they outsource a solution or buy software they're getting out of some responsibility. They're not. When that 3rd party causes a critical failure the proverbial finger still points at the company that chose to use the 3rd party.

The shareholders of Delta should hold this guy responsible for this failure. They shouldn't let him get away with blaming Crowdstrike.

[–] clstrfck@lemdro.id 18 points 3 months ago (30 children)

So you think Delta should’ve had a different antivirus/EDR running on every computer?

[–] riskable@programming.dev 1 points 3 months ago (2 children)

Adding another reply since I went on a bit of a rant in my other one... You're actually missing the point I was trying to make: No matter what solution you choose it's still your fault for choosing it. There are a zillion mitigations and "back up plans" that can be used when you feel like you have no choice but to use a dangerous 3rd party tool (e.g. one that installs kernel modules). Delta obviously didn't do any of that due diligence.

[–] catloaf@lemm.ee 3 points 3 months ago

Sounds like they executed their plans just fine.

And due diligence is "the investigation or exercise of care that a reasonable business or person is normally expected to take before entering into an agreement or contract with another party or an act with a certain standard of care". Having BC/DR plans isn't part of due diligence.

[–] ricecake@sh.itjust.works 3 points 3 months ago (1 children)

Kernel module is basically the only way to implement this type of security software. That's the only thing that has system wide access to realtime filesystem and network events.

Yes, they're ultimately liable to their customers because that's how liability works, but it's really hard to argue that they're at fault for picking a standard piece of software from a leading vendor that functions roughly the same as every piece of software in this space for every platform functions, which then bypassed all configurations they could make to control updates, grabbed a corrupted update and crashed the computer.
It's like saying it's the drivers fault the brakes on their Toyota failed and they crashed into someone. Yes, they crashed and so their insurance is going to have to cover it, but you don't get angry at the driver for purchasing a common car in good condition and having it break in a way they can't control.

What mitigations should they have had? All computer systems are mostly third party tools. Your OS is a third party tool. Your programming language is a third party tool. Webserver, database, loadbalancer, caching server: all third party tools. Hardware drivers? Usually third party, but USB has made a lot of things more generic.

If your package manager decides to ignore your configuration and update your kernel to something mangled and reboot, your computer is going to crash and it'll stay down until you can get in there to tell it to stop booting the mangled kernel.

[–] riskable@programming.dev 1 points 3 months ago (1 children)

It is absolutely not the only way to implement EDR. Linux has eBPF which is what Crowdstrike and other tools use on Linux instead of a kernel module. A kernel module is only necessary on Windows because Windows doesn't provide the necessary functionality.

Mitigating factors: Use (and take) regular snapshots and test them. My company had all our virtual desktops restored within half an hour on that day. If you don't think Windows Volume Shadow Copy is capable or actually useful for that in the real world then you're making my argument for me! LOL

Another option is to use systems (like Linux) that let you monitor these sorts of EDR things while remaining super locked down. You can run EDR tools on immutable Linux systems! You can't do that on Windows because (of backwards compatibility!) that OS can't run properly in an immutable share.

Windows was not made to be secure like that. It's security contexts are just hacks upon hacks. Far too many things need admin rights (or more privileges!) just to function on a basic level.

OSes like Linux were built to deal with these sorts of things. Linux, specifically, has gone though so many stages of evolution it makes Windows look like a dinosaur that barely survived the asteroid impact somehow.

[–] ricecake@sh.itjust.works 3 points 3 months ago

eBPF, the kernel level tool? Because you need to be in the kernel to have that level of access, which is what I was saying? The one with a bug that crowd strike hit that caused Linux servers to KP?
Yes, I said "kernel module" when I should have said "software executing in a kernel context". That's on me.

By the way, eBPF? Third party software by most metrics. Developed and maintained by Facebook, Cisco, Microsoft, Google and friends. Also available on windows, albeit not as deeply integrated due to the layers of cruft you mention.

I'm glad you were able to recover your VMs quickly. How quickly were you able to recover your non-virtualized devices, like laptops, desktops or that poor AD server that no one likes?
Airlines need more than just servers to operate. They also need laptops for various ground crew, terminals for the gate crew and ticketing agents, desktops for the people in offices outside the airport who manage "stuff" needed to keep an airline running.

You seem to be much more interested in talking about Linux being better than windows, which is a statement I agree with, but it's quite different from your original point that "Delta is at fault because they used third party tools".

My point was that it's unreasonable to say that Delta should have known better than to use a third party tool, while recommending Linux (not written by Delta), whose ecosystem is almost entirely composed of different third parties that you need to trust, either via system software (webserver), holding your critical data (database), kernel code (network card makers usually add support by making a kernel patch), or entire architectural subsystems (eBPF was written by a company that sells services that use it, and a good chunk of the security system was the NSA).

None of that bothers me. I just don't get how it doesn't bother you if you don't trust well regarded vendors in kernel space to have those same vendors making kernel patches.

load more comments (27 replies)
load more comments (27 replies)