This is a most excellent place for technology news and articles.
But if they do that then how will one guy monitor and manage 20 plants at the same time?
It's so fucking dumb. They are getting attacked, and they still need to be told to unplug? What the actual fuck?
yeah, but like...that won't happen to me.
security through obscurity is nothing but a fantasy.
I will take things that never should be connected to Internet for $200 Alex.
Remote monitoring? Sure. Remote managing? Fuck that. These facilities shouldn't be unstaffed for days at a time.
Oh for sure, air gapped sensors that relay information are going to be necessary. We should be building these systems with absolute security because they are critical to infrastructure.
Some people will say it's too hard to retrofit onto legacy equipment, but all it really means is adding a secondary system on top of the legacy system. Just a sensor array that has no interaction with existing systems.
Yup. Fancy architecture diagram:
One Way Radio
Transmitter
[Nuclear Reactor] ------------> [Internet connected receiver] -> tHe ClOUd
It's called paying people to watch the systems during off-hours. It's often way cheaper than setting up firewalls, setting up alerts, maintaining patches, paying a VPN company to manage connections, paying another company to handle authentication, and so on. Security constraints are getting so bad at my office I'm seriously thinking of how we can do more things with pen and paper.
Right. Why would you do this? Monitoring being available on the Intranet, of course. But controllers on the open internet?
Might as well really focus on ease of use and just make a live website with one big button that says "cut off all electricity and turn all the water into poison, lol"
And guarded by default passwords
And the user support AI the website automatically opens in in popup can just tell you if you tell it you're the King of Water and you forgot the account info.
It is also very easy to airgap monitoring.
A simple 300 bps connection over POTS to relay the information from the process controller is all you need. But even POTS is disappearing.
Exactly! Whenever I see one of these stories of a utility being hacked that's my first thought.
Not saying I ever accessed it, but there's a generator in Germany open to the web without security. It makes a few MW. My guess is hospital or any other critical infrastructure.
Went on the public tour of our local waste-treatment facility. Really fascinating stuff. Highly recommend it if you're even slightly curious.
There are definitely industrial-scale automation systems. Mostly PLCs, but also SCADA for monitoring. Lots of 20-30 year-old tech. Those networks have no reason to be connected to the internet.
But there were also office equipment and personal devices everywhere. All it takes is for someone wanting to monitor the state of a digester on their laptop from home.
You just crossed the beams.
Automation engineer here who does a lot of SCADA
There are soooo many good ways to securely monitor stuff from home. But especially in critical infrastructure it's just usually air gapped which is unbeatable in cyber security.
Tours of any local facility are super cool! I find it all so interesting, seeing how the things we use work!
I worked on new SCADA for a water company across all theirs sites. Previously they were a mismatch of different systems and all airgapped.
They standardised everything and put them all on VPN. Suddenly you could access the secure water treatment works with 24/7 security guards and the control room was a bunker, from the unmanned rural sewage works with a portacabin for the server, the key hidden under a brick, and wifi for the PLCs.
The sewage works had always operated like that because it was deemed low risk. No one considered the VPN changed the threat model.
The hacks target internet-facing programmable logic controllers (PLCs)
Why the fuck
is your PLC
facing the fucking internet!
Jesus Zombie Christ an a pogo stick. Has no one been paying attention for the last two decades? Seriously, we learned this sort of lesson in Two Thousand and fucking Three. Your critical assets do not get public IP addresses.
It usually happens in 2 steps.
First step is that everything gets connected to the LAN and you can only access the PLC network from within the building's network. Then some time later, management finds out that keeping someone on-call to go out costs a fortune, so they request that access be made so they can make the change from anywhere.
The IT team argues security, but no new hardware can be provisioned and a developing a new process is too hard. Then the magical phrase is uttered: "Just make it work". So IT punches a hole in the firewall, adds a NAT rule, and job done.
Except you skipped the part where IT is ridiculously underfunded and treated like crap ("We're a utility, not a tech company!")
With constant turnover, the person receiving the "Just make it work" call is about 6 layers removed from the person who originally set up the infrastructure, and probably completely unaware of all the previous security holes put in "because it's air-gapped, it doesn't matter."
Sadly yes, this is exactly what happens. And it ends up being IT holding the bag at the end, with the managers having long since cashed out their stock options and left.
"Well, we just bought this new monitoring platform from my cousin's company and they need access to it. No, their platform doesn't support any real secure protocols and won't route over a VPN, it needs to be on the same subnet as the devices. Ok, then just open all ports to the network so they can connect. No, they don't know what port or protocol, it also runs out of my cousin's house so it's on a dynamic IP so you can't create some type of access rule for it. Just put an any/any rule in and it's good. What do you mean we already have a monitoring platform? Well, this one is better!"
"No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. [...]"
They won't know that much. They don't understand that their job isn't running a water department, it's running the computers that maintain the water department and that includes security. For someone familiar with computer security it's easy to recognize negligence, but until the law sees it as criminal negligence nothing's going to change.
Small town municipal water departments aren't going to be able to afford the guy holding the shield against international state actors.
Literally this. It starts with “everything needs to be on an air gapped LAN so we can control it.” And then suddenly that air gapped LAN isn’t air gapped, because someone got tired of paying the call-out fee every time the on-call person had to drive to the site to push a few buttons on a keyboard. But figuring out a proper VPN (or at least a reverse proxy with access control) was too difficult or expensive, (management thinks the point is to save money on call out fees, not spend money on proper hardware) so they just poked a bunch of port-forwarded holes in the firewall and called it a day.
What’s extra ridiculous is VPNs are not that hard. I am fantastically awful with networking and even I can wrap my head around a WireGuard tunnel.
The issue is doing one at scale, with proper access controls. Also, let’s not skip past the whole “network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene, etc… All without any additional IT support or hardware, because management is doing this as a cost cutting measure.
I can pretty much guarantee that the systems that get hacked aren’t the ones that do everything right. They’re the ones who have a single shared VPN password, haven’t updated their firewall since it was installed 8 years ago, and they haven’t even changed the default passwords for their control systems. Because they got tired of answering phone calls about what the password was, so they just made every username and password something stupid like admin/admin.
“network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene,
This is all shit they should be doing now. They have to have a firewall to get to the net, so they should be maintaining and patching it. Managing users is standard AD shit, and password hygiene is t as easy, but setting reqs in AD for complexity goes a long way (as does not forcing passwords to rotate on an arbitrary basis).
And yes, I know it doesn't happen in a lot of places, which is why I said 'should'. Having been on the cleanup side of multiple security incidents, doing the basics is harder than you'd expect for a lot of places ....
Fine. New rule, anything that’s your fault and costs a company money you get to pay for out of your golden parachute, your pension, your savings, anything. You caused a problem to save money this quarter forsaking the next quarter when you aren’t there? Screw off.
Even when a company is doing things pretty well with a VPN, all it takes is one mistake and the cat is out of the bag. At one of my previous employers, we had a fairly good setup with VPNs using single sign on via our IdM provider. IdM enforced SSO and the VPN had a number of policies setup which kept good control over what users could login. And then someone had the brilliant idea to take an old VPN appliance, reset it to factory defaults and configure it just enough to work on our network and then hung it out on the internet for some sort of test (our configuration management left a lot to be desired). Of course, that led to it being left there for a couple months and an attacker finally found it. They also had a valid username/password combination which got them in the door.
Security is hard and often expensive. But still, compared to "we put out OT devices on the internet" a basic VPN is a huge step up.
I did some work for a company that owned a /16 public IP range and used it on their internal network. I had to put a safety rated control system on it that ran mining machines that weighed 60 tons and cost a couple of million bucks each.
They gave me a subnet range for the system that was clearly non-private and I was like, "ummmmm, is that on the Internet?"
Apparently it wasn't routed to the rest of the Internet but it always felt like it was one misconfigured router away from disaster.
not so funny when it isn't happening in the middle east but right at your doorstep
This has been talked about for years and about how exposed SCADA is. I just assumed that it was fixed/isolated years ago. The fix appears to have been to stop talking about it.
That's why NIS2 in Europe demands that OT is on a separate network from IT. My old workplace (well, the head of FO Technology and his gang anyway) was up in arms about it because it "was a waste of time" (and completely ruin the ideal architecture he had in his mind).
I am repeating what I said elsewhere on a different comm:
I call FULL bullshit on this being Iran.
I personally worked on the cybersecurity incident that happened to St Paul municipal and water systems last year as the Incident Response Technical lead.
The threat actors were not Iranian then. They were Russian.
I personally rebuilt most of their networks, VMware hosts, storage, and servers. The majority of the VMware environments did not have config backups and were running 5 and 6 branches of VMware, only two had 7+.
Their network was insecure. They had IPSEC VPNs between the various location, but there was no further security in that respect. Their firewalls were not being kept up. AAAAND... I warned them that we thought they were still compromised.
In fact, the forensics team called them out as being clean on a group call one week into the incident, but while we were on the call with leadership, they got RE-infected.
I had my team compile a massive set of recommendations for them and I wrote the report. Water Systems specifically. They did not listen to us, apparently. The forensics team also did this.
They had FBI, national guard (fucking incompetent), and law enforcement (useless) in all meetings.
If they got into Water, they’ll get back into city and municipal again. I guarantee it.
This is not Iran. It's the fucking incompetence of city government and Russian threat actors.
Minor nit: Russia and Iran are already working together on shaheds and targeting
No surprise at all if they also coordinate cyber
Whatever. I worked the fucking case. The infiltrators were Russian. After that, it's a political or financial negotiation.
why is any critical control equipment internet accessible.
I understand some machines might need internet to send reports and make document filings and shit. But it should not be the critical control equipment that makes the system function. And there should be a massive physical gap between the internet machines and the critical machines.
Jesus christ, even by the abysmally low bar I have for humanity, it still finds ways to dig itself a path under it.
The entire tech sector screamed about the vulnerability of IoT devices, plenty of which are now participants in zombie botnets.
Does anyone remember in 2015 when YESCO billboards by the thousands were hacked to show Goatse instead of an AT&T ad? Pepperidge Farm remembers. IoTs are still vulnerable, including big municipal ones and, yes, Flock Safety cameras.
And CISA ~~was severely broken~~ had its staff severely reduced by DOGE.
In unrelated news the CDC, also a victim of DOGE, is entirely unprepared for the next epidemic.
Why in the everloving fuck would scada ever be connected directly to even the regular lan, let alone the Internet!?!
Laziness. Poor training. Stupidity.
Why wouldn't you want a key utility controlled from around the global?
According to our peerless leader, I blame it on Minnesota because they are grossly incompetent
But then he tells California (and Canada) we are to blame for wildfires for our failure to rake our forests.
Yup, Sleepy-Don really do have the whole Iran situation under control.