Jeff Watkins, Chief Product and Technology Officer at CreateFuture, said:
What’s happened:
“Today, the skies seem to be empty, and that’s because many airports across the world are on a break, along with government departments, hospitals and more. There seem to be two sources for this, a Microsoft Azure outage, and an issue with a popular piece of cybersecurity software by Crowdstrike, which is reportedly taking down Microsoft Windows based systems. It’s not yet 100% clear if they’re related, but it does seem likely that a problem with Crowdstrike’s Falcon Sensor software could be affecting Microsoft’s own estate.
“Thankfully, remedial steps have been published to recover machines and a fix made available for download, but putting millions of Windows machines into recovery mode takes time. This could well be one of, if not the, biggest IT outages of all time, and it should make us all pay attention.
What can we learn:
“This situation brings up a wider discussion, one we should lean into after this has all cleared up, as if there’s one thing that is hitting organisations across the globe really hard right now, it’s software supply chain issues. These could be attacks, such as the near miss we had with the xz-utils, or the current Crowdstrike Falcon Sensor problem. The outcomes of a failure in a supply chain can be catastrophic, with loss of service across the globe, financial impacts and even loss of life.
“This all stems from our need to update our software frequently in order to remain secure in a time when cyber attacks are on the increase. Herein lies the rub, because as more software moves towards automatic updates, another avenue of failure and/or attack was opened up, that of a rapidly updating supply chain, and our ability to control that flow safely.
“Endpoint management and protection software, such as Crowdstrike, Kandji or Jamf can help an organisation manage this updating, balancing the currency of updates with safety, allowing testing or gradual rollout. But when this critical piece of the puzzle misfires, it seems there’s no fallback, as this software itself automatically updates. Although the failure seems to be in the threat detection parts of Crowdstrike rather than patch management, the point is still the same, who watches the watchmen?
“Resorting to having separate supply chains purely for disaster recovery purposes is probably a bridge too far for even large organisations. If Microsoft are struggling with this, I don’t think there’s much of a chance for the rest of us. What is clear is that we need a conversation about how our endpoint protection and management software is maintained and updated, maybe demanding control over when to apply updates. Aside from that, it’s a timely reminder that we should all take an interest in update management at all stages in the supply chain, as this time it was a bug, and not an attack. If this was a successful attack scenario, things would be much, much worse. It’s only a matter of time, unless we get serious about this.”
