CrowdStrike solution caused Blue Screen of Death in MS Windows

On July 19, 2024 at 06:09 CEST, CrowdStrike deployed a sensor configuration update to Windows systems as part of routine operations. This update resulted in a logic error causing system crashes to blue screen (BSOD) on affected stations. Customers using the Falcon sensor for Windows version 7.11 and above who were online between 06:09 CEST and 07:27 CEST on 19 July 2024 may have experienced system failures on stations.

Stations that downloaded a configuration update during this period were prone to crashing. Specific configuration files, known as "Channel Files" and integral to the Falcon sensor protection mechanisms, were the trigger for the incident. These files are regularly updated to accommodate new tactics and procedures identified by CrowdStrike. The affected Channel File in this incident, identified as 291, controls the evaluation of named pipes execution on Windows systems.

 

Confirmed Facts:

The issue only affected stations running the Windows operating system with the Crowdstrike Falcon version 7.11 agent installed.
Stations that were not online on 19 July between 06:09 and 07:27 CEST were not affected
Mac or Linux stations were not affected
Channel file "C-00000291*.sys" with a timestamp of 0527 or later is now fine
Channel file "C-00000291*.sys" with timestamp 0409 was causing the outage
 

Technical details:

On Windows systems, the "channel files" are located in the following directory:

C:\Windows\System32\drivers\CrowdStrike\

and have a file name that starts with "c-". Each channel file is identified by a unique number. The affected file in this event is 291 and will have a filename that starts with "C-00000291-" and ends with ".sys". Although these files end with the SYS suffix, they are not kernel drivers.
The "channel file" 291 controls how Falcon evaluates the execution of "named pipes1" on Windows systems. "Named pipes" are used for normal inter-process or inter-system communication in Windows. The update, which took place at 06:09 CEST, was designed to target newly observed, malicious "named pipes" that are used by common C2 channels in cyberattacks. The configuration update triggered a logic error that led to a crash of the operating system. CrowdStrike has already fixed the logical error by updating the contents in "channel file" 291. The Falcon agent continues to evaluate and protect against "named pipes" exploits.
 

Identification and fix procedure:

Step 1: Identify stations on the Falcon platform
Using an advanced event lookup query.
The queries used by the dashboards are listed at the bottom of the relevant KB dashboard articles.

Through the toolbar
There is an updated granular dashboard that shows Windows hosts affected by the content update glitch described in this technical alert.  See "Granular status dashboards to identify Windows hosts impacted by content issue (v8.6)".  All queries used by control panels are listed at the bottom of the respective KB articles on control panels.

Step 2: Remedy
If stations are still crashing and are unable to stay online to receive updates from the "Channel File", the steps below can be used.

Correcting individual stations:
Reboot the stations to allow them to download the corrected channel file.  We strongly recommend connecting to a wired network (not WiFi) before restarting the station, as the station will get an internet connection via Ethernet much faster.
If the host crashes again on reboot:

Option 1 - manual procedure
Please refer to this Microsoft article for a detailed procedure.
Note: Hosts with Bitlocker encryption may require a recovery key.

Option 2 - Automated via USB boot key
Follow the instructions in this KB article.
Note: Hosts with Bitlocker encryption may require a recovery key.

For more details on remedies, please visit Crowdstrike's support portal.

Crowdstrike's description of the incident can be found on their blog.