I’m putting together a bunch disparate resources that might help organizations or individuals tackle their #crashstrike fire and start to consider post-fix ramifications. However, the priority before putting this together was to obviously write a limerick:
When CrowdStrike updated their driversThere were very few global survivorsTo the endpoint we goWith fixes in towWe’re now the world’s trusted advisors
BSOD Fixes
22/07/24 If you need bulk rotate your Bitlocker keys see the Github script below from Ugur Koc. Good suggestion from Merill Fernando, in accordance with best practice, is not to use access tokens but switch to an interactive session…..
https://lnkd.in/gahWeVFY
21/07/24 This fix looks incredibly useful and kudos to the author. This is the first unattended option I’ve come across that works with exported Bitlocker keys and a USB-FIX dirve that can be shipped to different sites /users. Note: desperate times call for desperate measures and you will definitely want to rotate your encryption keys after this but there is a way to bulk automate this as well. Bootable USB to Fix Crowdstrike Issue (Fully unattended with Bitlocker Support) : r/msp (reddit.com) 2. There are lots of suggestions that approx 15 restarts can fix the issue Microsoft Recommends Rebooting Your Computer 15 Times as Blue Screen of Death Strikes Worldwide (futurism.com). You can imagine the memes circulating about this particular fix. Helpdesk “have you tried turning it off and on again” User: “Of course I have” Helpdesk “16 times?”
For computers that are in a blue screen state it’s seemingly unavoidable that manual steps will be required and CrowdStrike is still the best source for identifying those steps:
support article / https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/ ) (both are being updated in real-time with additional fixes & solutions)
Here is another summary of the fixes across different systems from Morten Knudsen.
Here is a fix developed by Adelaide University (my home city) to update the driver when you do not have the Bitlocker keys available (i.e major emergency situation)
Intune
If you are using Intune and wish to tackle machines that haven’t rebooted or blue screened (and worried the CS update might not apply correctly) you can look at this detection and remediation script by Mobile Jon. We haven’t test this yet but it looks like it would do the trick. He also discusses the following topics:
Using Intune Remediations to Fix the CrowdStrike Driver Bug (mobile-jon.com)
- What is a Driver Channel File?
- Intune Detection and Remediation Scripts for the CrowdStrike Outage
- Deploying Detection and Remediation Scripts in Intune
- Letting Users Self-Service Their BitLocker Keys in the Company Portal Website
Bitlocker
For devices that are bluescreening with Bitlocker drive encryption enabled, your life suddenly got a lot more complicated (and even more so if they weren’t backing up to Entra). There will be a need to retrieve those keys before getting your devices into safe mode so it’s worth exploring bulk export methods. Check out Nathan McNulty’s trick here:
With all this decryption activity and key retrieval underway I’d highly recommend keeping an eye on your audit logs and then rotating the keys once this saga is over. If you are using Microsoft Sentinel you should also put a detection rule in place so you can get an alert when these keys are accessed in your environment. Thanks to Jess Dodson for this:
If you are unable to retrieve your Bitlocker keys during the CrowdStrike BSOD then see this fix developed by Adelaide Uni.
KQL
Our team also wrote a simple query that will check the environment and determine where the corrupt Crowdstrike driver is location and give you some additional insight.
union DeviceFileEvents,DeviceProcessEvents,DeviceNetworkEvents
| where FolderPath == “C:\\Windows\\System32\\drivers\\CrowdStrike\\C-00000291-00000000-00000030.sys”
search “C-00000291*.sys”
| distinct DeviceName, ActionType,FolderPath
Scams and Threat Intelligence
ACSC have reported a number of malicious websites and unofficial code being released claiming to help entities recover from the widespread outages caused by the CrowdStrike technical incident. If you have the ability to upload any URL indicators into Threat Intel library I’d recommend doing that as soon as possible or at least making people aware to use the official channels for getting information and reviewing fixes.
Widespread outages relating to CrowdStrike software update | Cyber.gov.au
Here are a list of some of those URL’s but more will obviously appear and best to remain cautious and make people aware of these scams.
Suspicious Domains Emerged After Faulty CrowdStrike Update – Daily Dark Web
Finally
Now is definitely not the time to make any knee jerk reactions like disabling your AV or migrating to another vendor, despite the catastrophic impact this is having. At CSP, we only deliver Microsoft Defender for Endpoint and fortunately for us do not use CrowdStrike for any of our customers. However, I’ve helped some organizations over the last 24 hours that do still have the product and been impressed with CrowdStrike’s level of response and support. That said, once the dust has settled I would definitely be asking your IT team or CS directly if they support gradual roll-out (ring deployment) of updates (as Microsoft and others do). If they do not, then it might be time to reconsider your options……
Hopefully the wrong lessons are not learnt from this, such as a sudden surge in update avoidance. The benefits of keeping our systems updated still far out weigh the costs and we should have a strategic ring-based update schedule in place wherever possible. For example, we can, and should, set rings across all of the following:
-Workstation OS updates (Intune / Autopatch)
– Microsoft 365 Apps (i.e Word /Excel etc)
-Third party applications (with tools like PatchmyPC or Scappman)
-Azure Update Manager (for on-premise and cloud based servers)
-Anti-virus /EDR solutions (see links for Defender below)
https://cloudbrothers.info/en/gradual-rollout-process-microsoft-defender/
https://learn.microsoft.com/en-us/defender-endpoint/microsoft-defender-antivirus-ring-deployment