On 19th July 2024, a single content update from CrowdStrike put around 8.5 million Windows machines into a boot loop, grounding flights, freezing hospital systems and taking down payment terminals worldwide. How did one update cause such widespread disruption?
Well, contrary to some of the initial reaction, the chaos wasn’t down to a cyberattack or a sophisticated failure. Instead, CrowdStike had shipped a Falcon channel file with a parameter mismatch that its sensor’s interpreter couldn’t parse, and the resulting out-of-bounds read crashed the kernel driver every time the host tried to boot.
It was a standard change pushed out to production on a Friday. Yet the full consequences of the update weren’t visible until the file had already ended up everywhere. Two years on, the lesson most organisations have taken from the incident is about vendor testing and staged rollouts. This, of course, matters – but it misses the larger point.
CrowdStrike was visible because it broke everything at once. Yet the changes that should worry IT teams more are the ones that don’t, the small configuration shifts that accumulate inside an estate until one of them lines up with the others and something fails in a way nobody can trace back. This could be a service that won’t restart after a reboot weeks after the change that actually broke it, or an integration that works for everyone bar one group of users because a permission changed somewhere upstream and nobody logged it.
So, why do such changes go undetected?
Updates beyond a team’s control
Since I’ve worked in IT, ‘read-only Friday’ has been a running joke. And there’s a reason it’s now a frequent topic of discussion on sysadmin forums. The discipline of not shipping anything you’d then have to keep an eye on over the weekend exists because experienced teams learned – repeatedly – that if you make a change when half the office has gone home, it will be the one that ruins your Saturday. It’s a sensible human guardrail, but it’s designed for a system where a person decides when each change goes out.
In a SaaS estate, however, change is constant, and most of it isn’t yours to schedule – vendors push updates at their cadence. As such, many organisations have auto-update enabled on the working assumption that staying up-to-date is of course safer than falling behind. This strategy holds, right up until the vendor ships a breaking change.
In April 2026, Microsoft’s Patch Tuesday update put Windows Server domain controllers into a reboot loop by crashing LSASS on startup, taking authentication and directory services down with them and, in some environments, the whole domain. Microsoft confirmed it and issued an out-of-band fix, as it has had to do often enough that out-of-band updates have stopped being the exception. This is precisely why experienced admins hold back from the latest release and run n-1 or n-2, letting someone else’s estate be the test environment.
However, this approach only governs the update classes it can see. CrowdStrike’s channel files were configuration content rather than a sensor version, so they bypassed the ring-deployment and version-pinning most teams apply to the sensor binary and instead went to the entire fleet at once. Running an older sensor version didn’t help either, as the change arrived through a channel that sat outside the version control.
So, while the n-2 instinct is sound, it does nothing about the SaaS integration that can update itself overnight, or the AI copilot acting on an engineer’s behalf, or the automated remediation tool correcting one setting but moving another in the process.
This is the part that the CrowdStrike post-mortems mostly skipped.
Change that goes unnoticed
Drift is a measurable quantity, the delta between a system’s running state and its last known good baseline across registry values, service states, driver versions and group policy settings. Most organisations only ever sample that delta at a point in time – through a periodic audit or a manually maintained CMDB – rather than watching it continuously. But even when an organisation runs rigorous change control on everything it deploys deliberately, it can still be exposed, as the drift that causes the outage usually isn’t a deployment anyone approved.
The traditional answer to this change risk was the Change Advisory Board (CAB), a meeting where proposed changes were reviewed and signed off before they went near production. CABs still exist, but again, they work on the basis that a human submits each change in advance, and so they became insufficient when most of the change in an estate stopped passing through human hands.
You obviously can’t convene a board to approve a configuration shift if nobody knew it was going to happen in the first place. And having more meetings won’t close this gap either, as this isn’t a process problem. The problem is trying to govern change that isn’t visible.
The summer risks
Summer makes this worse in a specific, practical way. Throughout July and August, IT teams run on reduced cover and the people who hold the undocumented knowledge about a given system are often the ones taking annual leave.
While the rest of the team is aware of who is out of office when, the patching schedule and the AI agent acting on their behalf aren’t, and they will keep making changes whether or not anyone is positioned to catch a problem. A drift that would be spotted and reversed in an afternoon in March, for instance, can sit unnoticed for a fortnight in August, and an incident that surfaces on a more stretched team can take longer to diagnose because the person who understands the system can be unreachable.
None of this is an argument for freezing change over the summer, though slowing transformation work when cover is thin is a reasonable call. The argument is that, as outlined above, the human guardrails of read-only Friday, the CAB, the n-2 discipline are all designed around the fact that a person was in the loop deciding when change happened, and that structure no longer accounts for how a modern estate actually moves.
Continuous visibility is the key lesson
If an estate is being modified continuously by vendors and automated actors, the only thing that scales is continuous visibility into what is actually changing, measured against a known-good baseline, rather than a quarterly audit that’s out of date the week after it takes place. Change will happen whether it’s approved or not. The only question that matters is whether an IT team finds out when it does.
That is the lesson CrowdStrike should have left behind. Updates carry risk, and they always will. What turned this one into a global outage was that the organisations impacted had no way to see how that single change would interact with everything else running in their estate, the other software, drivers and dependencies it would meet on each machine.
Two years later, with far more change happening automatically and far less of it visible, that exposure is wider. The question for any IT team this summer is a simple one: if something drifts in August, would you know before it broke?
Kashif Nazir
Kashif Nazir is Senior Technical Architect at Cloudhouse, working at the point where cloud strategy, AI, and enterprise modernisation meet. His work spans architecture direction across multiple product lines, the company’s cross-team R&D function, and an enterprise AI programme that has moved from evaluation into production automation across the business.
