Dispatch from the Front Line

In July 2019 we published our inaugural blog post titled “The Next Phase in Cyber Insurance.” At that time, the lost laptops and basic malware incidents of earlier years had morphed into complex situations involving destruction and business interruption. That article shined a spotlight on an increasingly complex landscape that the existing approaches were ill equipped to handle. We introduced the MOXFIVE Technical Advisor concept: a role that is positioned to deliver better outcomes to clients and carriers alike.  

Fast forward two years; MOXFIVE is deeply engaged in the proverbial trench. Our Technical Advisors have provided a steady voice of experience. As we expanded into incident coordination and business recovery, we and our partners’ engineering teams have helped hundreds of clients get back on their feet quickly.  

Reflecting on the road behind us, we humbly offer the following observations and recommendations.

First Things First

The following technical capabilities are critically important to prevention, detection, and failing that, to enact a swift response.  If you only take away one thing from this article – implement these capabilities starting today. These represent the minimum pillars necessary for an effective security capability; learn from our clients’ pain.  

• Multifactor Authentication: Secure all Internet-facing entry points to your network with multifactor authentication (MFA). Without it, your internal assets are a guessed password, or a re-used password stolen in another attack, away from being compromised.
• System Management: Understand what systems you have. Have a means of managing the configuration of every system you own, whether a server in a physical data center, a virtual server in a cloud environment, an on-premises workstation, or a remote laptop off-network. This means the ability to install software, manage the configuration, and execute administrative actions as needed. Patch things!
• Endpoint Detection & Response :Install endpoint detection and response tooling that provides both blocking (known-bad and likely-bad executables) and remote investigative capabilities. Block what you can and have a capability already in place for when prevention fails.
• Robust Backups: Follow the 3-2-1 rule: have at least three copies of your data (one primary, two backup copies), use two different types of storage media, and ensure one backup is off-site. In the era of ransomware, that off-site backup needs to be configured such that it cannot be easily compromised if an attacker gains privileged internal access. (If you are not sure if this applies to you, run a tabletop exercise and/or a Purple Team exercise to simulate an attacker targeting backups – this happens every day.)
• Privileged Access Management: Protect administrative credentials and authenticators (e.g., hashes). These, along with administrators’ systems, should be considered crown jewels in your security planning. Do not use the same local administrative password on multiple systems! Make them unique, or better yet, disable them in favor of multifactor-authenticated, managed privileged access.

Slow Down to Go Fast

“I need as many people as I can get engaged right now!” exclaimed the senior executive on the first day of the incident response effort. While it is an understandable reaction to a crisis, more people involved early on can be counterproductive and delay recovery. Rather than focus on more, it is vitally important to accomplish the following at the outset of the response effort – before scaling up the number of doers executing the work.

• Identify specific business and operational goals down to a prioritized list of assets to be restored.
• Based on the near-term goals, define tasks and associated work instructions to minimize confusion (for example, document and share the step-by-step process to be used to restore a server from backup, install tooling, investigate, validate functionality, and return to production).
• Begin structured communications, including a means of real-time communication within/between teams, and regular status updates that deliver predictable, quality information to decision makers.
• Organize workstreams, with designated leaders and project managers, to determine how to best tackle goals – make payroll by Monday morning, get email running.

Once lanes are defined, additional resources can be added to turbocharge some workstreams. But not all processes benefit from more people. For example, if you are only able to restore one virtual system image from backup per hour, adding 10 more people to that workstream won’t make it go faster.  

Even still, sometimes one may be tempted to add additional people, regardless of a real need, to improve the optics of the situation. While it may feel good in the short term, I have seen this approach degrade teams by creating more (unnecessary) meetings and increased coordination challenges. This increases risk when multiple, well-intentioned people are trying to do their part but lack clearly defined lanes and coordination. For example, if two engineers independently start restoring servers from backups and causing IP address collisions, this will cause more trouble as other efforts will need to stop to troubleshoot unusual errors that stem from this avoidable mistake.

There is also a cost component to consider. Insurance carriers are increasingly scrutinizing recovery costs, so if you plan on filing a claim, keep in mind that the reasonableness of costs-versus-impact will be reviewed as part of their processing the claim.  

Consult the Right Experts

While a neurologist may be an extremely capable physician in their field, when you have a heart condition, you need the expertise and experience of a cardiologist.  In the field of incident response, when you need to make decisions about how to proceed with investigation and recovery, you need a Technical Advisor.  

Back in 2019, we said that a Technical Advisor is “best suited to identify possible investigative rabbit holes, inefficient business recovery activities, or potential technical items / tasks that may not be aligned with the policy coverage.” Looking back two years later, we’ve helped many clients avoid these sand traps.  

For example, several clients initially planned to reimage large numbers of systems affected by ransomware. After assessing those situations in collaboration with the investigative teams, we successfully executed containment plans that avoided the large time and financial costs of massive reimaging projects. A combination of EDR tooling to block known and unknown malware and some (automated, scripted) elbow grease to decrypt and clean up systems en masse often does the trick.

Many voices are heard around the table during an incident response situation. Find advisors that have extensive experience in incident response within their domains of expertise: legal, investigative, recovery, communications. It’s best to establish these relationships before you need them. You’ll sleep easier knowing that you have someone to call.  

If you have questions or need help with a current incident, we're here to help. Contact a MOXFIVE Technical Advisor at ask@moxfive.com or use our Contact form.

‍