Incident Commenced: 19JULY2024
Product Family Affected: All Products
Abrigo Reference#: IM-125
Incident Summary: Beginning at 1:30 am ET on July 19, Abrigo-hosted software applications became unavailable due to a global outage caused by a faulty update from our service provider, CrowdStrike. This update led to issues with the Falcon sensor software, affecting our systems. Once remediation steps were provided by CrowdStrike, Abrigo promptly tested and implemented the fix, restoring functionality to our applications.
Resolution Summary: CrowdStrike provided resolution steps, which included either rebooting the affected machines up to 15 times or booting into safe mode to delete any .sys file beginning with c-00000291. Microsoft suggested an alternative resolution of restoring systems to backups from prior days. Abrigo implemented all these solutions in order of complexity. If repeated reboots did not resolve the issue, we either restored the system or deleted the identified .sys files. We prioritized production environments and subsequently addressed UAT environments over the weekend.
Root Cause Statement: On July 19 at 04:09 UTC, CrowdStrike released an update to the Falcon sensor software on Windows PCs and servers that contained a faulty configuration. The update included a change to the configuration file responsible for monitoring named pipes, specifically Channel File 291. This modification led to an out-of-bounds memory read in the Windows sensor client, triggering an invalid page fault. Consequently, affected machines either experienced continuous reboot cycles or entered recovery mode.
Mitigation Summary: To mitigate the issue going forward, Abrigo will update our break glass process to ensure rapid and efficient responses. We will audit our backup settings in AWS to enhance reliability and ensure quick recovery. Additionally, we will review and refine our recovery precedence procedures to handle multiple production failures more effectively. These measures will strengthen our resilience and maintain our critical operations.