So as we all know on the news, the cybersecurity firm Crowdstrike Y2K’d it’s own end customers with a shoddy non-tested update.
But how does this happen? Aren’t there programming teams and check their code or pass it to a quality assurance staff to see if it bricked their own machines?
8.5 Million machines too, does that effect home users too or is it only for windows machines that have this endpoint agent installed?
Lastly, why would large firms and government institutions such as railway networks and hospitals put all their eggs in one basket? Surely chucking everything into “The Cloud (Literally just another man’s tinbox)” would be disastrous?
TLDR - Confused how this titanic tits up could happen and that 8.5 Million windows machines (POS, Desktops and servers) just packed up.
Lastly, why would large firms and government institutions such as railway networks and hospitals put all their eggs in one basket? Surely chucking everything into “The Cloud (Literally just another man’s tinbox)” would be disastrous?
Because they are best in class. No one else does EDR like Crowdstrike does. Can you imagine the IT support headaches if you had 200,000 PCs and servers some running one EDR and others running a different one. The amount of edge cases you would come across is ridiculous.
It would make data correlation a nightmare if an actual security incident occured.
Best in class should be questionable now when they’re the sole direct cause of a worse global outage than any group of determined black hats ever has managed. If they’re considered best in class, the whole class needs to be held back to repeat the grade.
8.5 Million machines too, does that effect home users too or is it only for windows machines that have this endpoint agent installed?
This software mandated by cyber insurance companies to ‘keep your business secure’ aka, your staff broke policy so we don’t have to pay this out claim.
No home user should ever run something like this at all. This is entirely a corporate thing.
But how does this happen?
It’s destined to happen, according to Normal Accident Theory.
Aren’t there programming teams and check their code or pass it to a quality assurance staff to see if it bricked their own machines?
Yes, there are probably a gigantic number of tests, reviews, validation processes, checkpoints, sign-offs, approvals, and release processes. The dizzying number of technical components and byzantine web of organizational processes was probably a major factor in how this came to pass.
Their solution will surely be to add more stage-gates, roles, teams, and processes.
As Tim Harford puts it at the end of this episode about “normal accidents”… “I’m not sure Galileo would agree.”
That is the risks of DevOps continuous improvement/continuous development (ci/cd) . why break things one at a time, when you can break them in the millions at once.
I fully expect to see CS increase their QA for the next year or 2 then slowly dwindle it back to pre fuckup levels when their share price recovers.
Doesn’t CI stand for Continuous Integration?
