A major disruption to Windows PCs in the U.S., U.K., Australia, South Africa and other countries was caused by an error in a CrowdStrike Falcon Sensor update, the cloud security company announced on Friday. Emergency services, airports and law enforcement reported downtime, which is ongoing. About 8.5 million Windows devices were affected.
“This is not a security incident or cyberattack,” CrowdStrike said in a statement Friday morning.
CrowdStrike expanded on that statement by Friday afternoon, adding “We understand the gravity of the situation and are deeply sorry for the inconvenience and disruption” and assuring customers that the CrowdStrike Falcon platform itself is “operating normally.”
Blue Screen of Death widespread due to CrowdStrike outage
Affected organizations saw the infamous Blue Screen of Death, the Windows system crash alert. According to The Verge, the problem originated with an update to a kernel level driver used to connect CrowdStrike to Windows PCs and servers.
American Airlines, United and Delta flights were delayed on Friday morning due to the issue impacting the airlines’ IT systems. U.K. media outlet Sky News reported on its own television outage early Friday morning. The New Hampshire emergency services department reported it is back online after disruption to 911 services early Friday.
“The issue has been identified, isolated and a fix has been deployed,” CrowdStrike said on Friday. However, outages on some machines that were initially affected are still being reported.
Microsoft 365 reported a service degradation warning on Friday morning, but this appears to be a separate incident.
CrowdStrike made 14.74% of the total software revenue for security software segments and regions in 2023, according to data Gartner sent to TechRepublic by email. Microsoft made 40.16%.
SEE: Downtime costs the world’s largest companies $400 billion a year, according to Splunk.
What steps can businesses take if they are affected by the CrowdStrike outage?
The first step is to identify which hosts are impacted. From there, follow CloudStrike’s instructions for repairing or recovering Windows.
On Saturday, Microsoft released a Recovery Tool using a USB or Preboot Execution Environment.
On Friday, Microsoft recommended restarting Azure Virtual Machines running the CrowdStrike Falcon agent. This may require a lot of reboots, with some users reporting success after as many as 15. Other options are to restore from a backup earlier than July 18 at 04:09 UTC, or to try to repair the OS disk by using a repair VM.
“Because of the way in which the update has been deployed, recovery options for affected machines are manual and thus limited,” said Forrester VP and Principal Analyst Andras Cser in a prepared statement emailed to TechRepublic. “Administrators must attach a physical keyboard to each affected system, boot into Safe Mode, remove the compromised CrowdStrike update, and then reboot. Some administrators have also stated they have been unable to gain access to BitLocker hard drive encryption keys to perform remediation steps.”
CrowdStrike recommends that its customers keep in touch with CrowdStrike representatives. Organizations, even those not directly affected, should check in with their SaaS partners to see whether they might be experiencing issues.
Beware of misinformation
Because this incident affects such a wide range of major organizations, the possibility for misinformation is high.
“There will be a lot of misinformation about how to reconfigure your computers or which critical system files to delete,” said former NSA cybersecurity expert Evan Dornbush in an email to TechRepublic. “Don’t fall victim to downloading phony solutions.”
On Saturday, CrowdStrike highlighted a malware campaign targeting Spanish-speaking CrowdStrike customers which disguised itself as a fix for the outage. The malware is a ZIP file attached to a bogus “utility for automating recovery,” according to CrowdStrike’s blog post.
“This is a great time to reflect on password management, since the fix may eventually require administrative access to systems that have not rebooted in quite some time,” Dornbush said.
Assess your recovery plan and support your team
Assess your organization’s reliance on one provider or service, and be sure your organization has a strong recovery process in place.
It’s also a good time for IT team leaders to make sure their personnel have the support they need.
“This disruption hit on Friday evening in some geographies, right as people were headed home for their weekend,” noted Forrester Principal Analyst Allie Mellen in a prepared statement emailed to TechRepublic. “Tech incidents like this require an all-hands-on-deck approach, and your teams will be working 24/7 over the weekend to recover. Support your teams by ensuring they have adequate support and rest breaks to avoid burnout and mistakes. Clearly communicate roles, responsibilities, and expectations.”
When reached for comment, CrowdStrike directed TechRepublic to the official statement.
This article will be updated as more information becomes available. TechRepublic has reached out to Microsoft for comment.