CrowdStrike update triggers Windows blue screen crisis, Microsoft reveals the root cause

July 29 news, over the past ten days, CrowdStrike and Microsoft have been working hard to assist users affected by the massive Windows blue screen of death problem. The problem was caused by a faulty update of CrowdStrike. In addition to providing a solution, CrowdStrike has released a preliminary post-incident review report on the outage. According to the report, the blue screen of death was caused by a memory safety issue, and an out-of-bounds read access violation occurred in CrowdStrike's CSagent driver.

Microsoft yesterday published a detailed technical analysis of the outage caused by the CrowdStrike driver. Microsoft's analysis confirmed CrowdStrike's findings that the crash was caused by an out-of-bounds memory safety error in CrowdStrike's CSagent.sys driver. The csagent.sys module registers as a file system filter driver on Windows computers to receive notifications about file operations, including the creation or modification of files, which allows security products, including CrowdStrike, to scan any new files saved to disk.

At the time of the incident, Microsoft was under heavy criticism for allowing third-party software developers to have kernel-level access. In a blog post, Microsoft explained why it provided kernel-level access to security products:

The kernel driver allows system-wide visibility and the ability to load early in the boot process to detect threats such as bootkits and rootkits, which can load before user-mode applications.

Microsoft provides functions such as system event callback and file filter driver.

The kernel driver provides better performance for situations such as high-throughput network activity.

Security solutions want to ensure that their software cannot be disabled by malware, targeted attacks, or malicious insiders, even if those attackers have administrator privileges. To this end, Windows provides Early Launch Antimalware (ELAM) early in the boot process.

However, kernel drivers also come with tradeoffs because they run at the most trusted level of Windows, increasing risk. Microsoft is also working on migrating complex Windows core services from kernel mode to user mode, such as font file parsing. Microsoft recommends that security solution providers balance the need for visibility and tamper resistance with the risks of kernel mode operations. For example, they can use minimal sensors that run in kernel mode for data collection and execution, thereby limiting exposure to usability issues. The remaining functions, such as managing updates, parsing content, and other operations, can be performed in isolation in user mode.

In the blog post, Microsoft also explained the built-in security features of the Windows operating system. These security features provide multiple layers of protection against malware and attack attempts. Microsoft will work with the anti-malware ecosystem through the Microsoft Virus Initiative (MVI) to further improve security and reliability using Windows built-in security features.

Microsoft currently plans to:

Provides secure deployment guidance, best practices, and technologies to make security product updates more secure.

Reduces the need for kernel drivers to access important security data.

Provides enhanced isolation and tamper resistance through technologies such as the recently announced VBS Islands.

Enable zero-trust approaches, such as high-integrity authentication, which determines the security posture of a machine based on the health of Windows native security features.

As of July 25, more than 97% of Windows computers affected by this issue are back online, and Microsoft is now looking to prevent such issues in the future. John Cable, Microsoft's vice president of Windows program management, recently published a blog post about the CrowdStrike issue, in which he mentioned that Windows must prioritize change and innovation for end-to-end resiliency, which is what customers expect from Microsoft.