Understanding the Crowdstrike crisis 

Richard Ford at Integrity360 explains how Crowdstrike became a single point of failure

 

The outage of worldwide systems caused by an update by Crowdstrike to its security software on 19 July has been momentous and the cost of disruption will no doubt run into the billions. Crowdstrike were very bullish in their mission statement: "We Stop Breaches". Unfortunately, this time, they’ve created the outage. But what exactly went wrong and why?

 

Crowdstrike’s catastrophic error took a large percentage of global IT systems offline. On the one hand it’s shown how large Crowdstrike’s market share is, but it’s also shown how fragile the interconnected world we live in can be. It grounded airlines, halted broadcasters and took channels offline, and, at the most critical end, severely impacted emergency services. 

 

The Crowdstrike ecosystem revolves around a single agent deployment to deliver their portfolio of security solutions, which operates permanently online, connected to their SaaS-based management platform.

 

In a world where threats are constantly evolving, and we need to move quickly and often to counter them, it’s an effective approach that has become the industry norm. Updates are delivered directly to the endpoint agents as they become available ensuring systems have the real-time protection they need. The downside, and what has happened with Crowdstrike today, is that a bad update can have a wide-ranging impact. 

 

How the update works

On this occasion, as it does on a regular basis, Crowdstrike pushed what it refers to as a Channel File, which would likely include updates to their threat detection definitions to all Crowdstrike agents.

 

This file, when processed by the Crowdstrike agent running on a Windows device, caused the agent to dramatically crash, creating a Blue Screen of Death (BSOD) and a restart. Unfortunately, the file is run by the agent during system boot, crashing the system and repeating the process, thereby creating a restart loop. 

 

The fix is relatively trivial. Once the agent is online it will just download the fixed Channel File. The challenge is getting it online, and this is no small feat. As the system crashes and reboots before getting online, it will require manual intervention to fix in many cases as the user will need to end a special administrative mode before the system boots and use the command line to search for and delete the file.

 

As many users aren’t IT experts or won’t be old enough to remember the days of MS-DOS, this will be entirely new to them, and a nightmare for IT teams to orchestrate. 

 

Why recovery could take some time

But unfortunately, it gets worse. Best security practice is to have your data stored on the system encrypted at the hard drive level. This prevents data from being directly extracted from the drive, should it be stolen and importantly also protects the boot process.

 

The knock-on effect is that to access the administrative mode for systems with drive encryption (provided as part of Microsoft Windows), systems will need to be put in recovery mode and may require a recovery key, unique to that system, in order to implement the fix. Consequently, we can expect the recovery process to be long and really test IT teams and resilience of organisations.

 

Questions need to be asked about how this happened. Is this the product of agile, CI/CD (Continuous Integration/Continuous Delivery) software development? If you’re introducing an update, even to an external file, has this not been thoroughly tested through a quality assurance (QA) process? The widespread impact calls this into question.

 

Or, if it has gone through the testing & QA process, has the file been subverted further along in the process by a threat actor? There’s absolutely no evidence currently that this is the case, but we only need to look at the SolarWinds breach to see evidence of this happening in the past. These are all questions Crowdstrike will need to answer over the coming days and weeks. 

 

Lastly, we should also think about the current approach to security, and the unification of technologies and vendors. The move to XDR, for example, is putting a lot of eggs in a single basket.

 

Will the impact of this incident cause the more risk adverse organisations to distribute their security controls and risk across more vendors and segment areas of the business? Potentially, but all will be acutely aware of the trust we put into vendors and our recovery plans. CrowdStrike haven’t been alone in this, with Microsoft only this week confirming a major outage for Microsoft 365 caused by a configuration change.

 

This crisis is a not so gentle reminder that Availability is equally as important as Confidentiality and Integrity in the CIA security triad.

 

---

 

CrowdStrike Remediation Guidance

Prerequisites: If you are using Bitlocker across your devices, you will need the BitLocker Recovery Key to decrypt the encrypted drive prior to following the steps below.

 

Boot into Safe Mode:

1.     On the Recovery screen, select “See advanced repair options.”

2.     Navigate to: Troubleshoot > Advanced options > Startup Settings > Restart.

3.     Start your PC in Safe Mode.

 

Open Command Prompt (Admin):

1.     Type cd C:\Windows\System32\drivers\CrowdStrike and press Enter.

2.     Identify the file by typing dir C-00000291*.sys and press Enter.

3.     Delete the file by typing del C-00000291*.sys (replace C-00000291*.sys with the actual file name found).

 

---

 

Richard Ford is CTO at Integrity360

 

Main image courtesy of iStockPhoto.com and DKosig