Enterprise Remediation Part 1: Five Tips for Preparing and Planning

Timing is everything when it comes to responding and recovering from a widespread, destructive attack. As threat actors operate undetected across a victim network and get deeper into the attack lifecycle, it becomes increasingly more challenging to recover and avoid the business disruption that comes from a compromised environment.

In many cases, when discovered in earlier phases of the attack lifecycle, it is possible to recover compromised endpoints in real time — meaning we can effectively undo the malicious actions that were executed by the threat actor with minimal to no disruption to either users or the business. But, in more severe cases in which the threat actor has moved beyond the initial attack stages and exfiltrated data or disrupted systems, we must turn to an enterprise-wide remediation approach because systems have been rendered unrecoverable using real-time response techniques. For example, if an organization lacked an appropriately deployed endpoint defense tooling, a ransomware operator could quickly move from an initial entry point to encrypting thousands of systems and bringing business to a halt.

CrowdStrike and MOXFIVE work together to help get victim organizations back to business and minimize disruption. Using CrowdStrike’s intelligence-led rapid recovery approach, we’re able to gain immediate threat visibility and gather the forensic data needed to understand which systems have been compromised and to what degree, which is essential to effectively restore operations. As we move into the recovery phase, our goal is to recover as many systems as we can using the Falcon Real Time Response capabilities that are built into the CrowdStrike Falcon^® platform. This minimizes the impact to the business by reducing the number of systems requiring more extensive efforts to remediate. MOXFIVE’s remediation services, including engineers that provide a variety of hands-on-keyboard expertise, drive the enterprise remediation effort in close collaboration with the CrowdStrike team.

There are times when an incident has progressed beyond the realm of straightforward, rapid response. The victim organization typically arrives at that point through a series of events, that when unraveled through an investigation and viewed retrospectively, looks like the following:

• A threat actor, whether targeted or opportunistic, leveraged stolen credentials or identified a weakness in a perimeter system to exploit.
• That attacker (or other threat actor that has purchased access to the victim network) gained initial access to one system.
• Undetected, the attacker escalated their privileges, evaded defenses, obtained privileged credentials and moved laterally to many systems in the environment.
• The attacker stole information and, in many cases, destroyed backups before deploying ransomware widely across the environment.

With a focus on incident management and business resilience, CrowdStrike and MOXFIVE have helped organizations recover from attacks that have progressed to this stage.

Welcome to Enterprise Remediation. This two-part blog series by CrowdStrike and MOXFIVE will guide you through key considerations as you:

• Prepare and plan for a ransomware attack
• Contain the threat during an attack
• Recover the environment with full enterprise remediation

Prepare and Plan

Like a patient who’s suffered potentially life-threatening trauma being wheeled into the emergency room, when enterprise remediation becomes necessary, the organization should recognize that it stands at a crossroads. This point in the attack lifecycle often comes with severe business disruptions, such as systems rendered unusable, customers unable to transact, and partners and other stakeholders asking tough questions. The response process has shifted from something that can be handled within IT to a significant company-wide endeavor. The following five tips have proven to reduce stress and increase response effectiveness time after time.

1. Establish clear ownership and define an incident management team

Establishing clear ownership and defining an incident management team will drive efficiency. First, designate an incident manager who will be charged with organizing the response team, managing internal communications and managing major tasks.

The most effective team structures establish an overarching response framework and then delegate ownership of operational decisions to the workstream management level. When designing the structure, consider how to enable the engineering teams and other “doers” to remain focused on executing, rather than getting bogged down on frequently reporting status or reprioritizing their efforts to chase down the latest request for information from above. Create pathways for information to flow upward from the front line and for thoughtfully prioritized tasks to flow down to the front.

To enable senior executives to effectively execute their day jobs in addition to their crisis management roles, designate a proxy for the executive team who will be empowered to remove obstacles. Serving as an ultimate escalation point for the workstream managers, this person would ideally have a long tenure and broad relationships across the organization within and outside of IT.

2. Assign skilled incident managers

Skilled incident managers can be indispensable in managing the response process — an often-overlooked ingredient to success. The incident manager should have project managers (PMs) aligned to their team to assist with building the plan, updating status for leadership and other stakeholders, and pushing down modifications to the plan. In coordination with workstream project managers, the PM-to-PM connection provides a low-friction conduit into and out of workstreams, saving precious time.  

3. Define measurements

Define measurements to lower the blood pressure of stakeholders. In the hectic world of incident response, position your incident manager to be a voice of reassurance by adopting a consistent presentation style centered around minimizing business impact. Capture meaningful metrics and weave them into a narrative that explains today’s status in the context of the short-term and longer-term milestones to restore the business. The following measurements help to communicate status in an impactful way:

• Business process status, ranked by criticality, with estimated dates of partial and full recovery;
• Supporting IT infrastructure components necessary for the entire environment to function (for example, Active Directory and Microsoft 365 email) and their statuses;
• Detailed status for each of the infrastructure components showing:
• What business processes it supports — for example, which servers are necessary to run payroll.
• Their statuses (inoperable: backup available, inoperable: no viable backups, restored from backup, recovered and validated by business).

4. Leverage collaboration technology for real-time communications

Leverage collaboration technology for real-time communications within and between workstream teams, including outside vendors and consultants. In a fast-paced crisis involving many teams, smooth communication between operational teams can significantly impact performance.

5. Establish a single source of truth for recovery activities

It is crucial that all engineers working on recovery activities use a single source of truth to track all systems’ status, which business process that system supports, the system’s business and technical owners, and planned actions. This mechanism provides myriad benefits including:

• Reduced frequency of errors. For example, without a single source of truth for system status, two well-intentioned engineers may both assign the same network address to two different servers, causing insidious impacts that take time away from the recovery effort to troubleshoot and resolve.
• Increased efficiency, reduced business impact. With a clear, complete and prioritized server list, resources can be directed to recovering the systems that will most rapidly reduce the most significant business disruptions. Additionally, such a tracking document enables third-party resources to be optimally leveraged. Absent this prioritized list, precious time will be misallocated toward lower-impact items (from a business perspective), increasing downtime.
• Better reporting. Without a foundational tracking mechanism that ties current system status to business applications and processes, there is no way to answer senior leadership’s key question: When will we be recovered? With that detail available, the incident manager can brief leadership in their language.  

We’ll cover two additional dimensions of this topic, containment and recovery, in Part 2 of this blog series.

Additional Resources

• Learn more about how CrowdStrike Breach Services can help you respond to an attack with speed and recover from an incident with surgical precision.
• Explore the speed and efficiency of MOXFIVE’s Platform for incident management and business resilience.
• Download the complete CrowdStrike Incident Response eBook to learn more about CrowdStrike’s modern approach to rapid response and recovery from today’s widespread security incidents.
• Get on-demand access to CrowdStrike incident responders, forensic investigators, threat hunters and endpoint recovery specialists with a CrowdStrike Services Retainer.

‍