Ransomware Recovery Tales: A Tale of Two Clients

During the early hours and days of responding to a ransomware event, making the right decisions are crucial. While everyone is working towards the common goal of restoring business operations, tensions are high and rising stress levels are pervasive. At these times there are two paths a client can traverse: the easy path and the hard path. During this blog I’ll explore two use cases — one of each — and identify how to avoid the hard path.

A few months ago, I began two ransomware recovery projects with two similarly sized client organizations in rapid succession. Both clients had similar situations, fully offline due to ransomware but with viable backups to recover the infrastructure and business data. However, it was on day one that the differences between the two organizations’ decisions began to create a strong juxtaposition in how proper decision making can greatly influence the time and effort involved to get business operations restored.

The Race to Get Started

In the early hours of a ransomware recovery engagement, what MOXFIVE refers to as “The First 48”, getting organized and providing access to third party service providers is the main goal. The first client, who we’ll refer to as Acme Corporation, understood the limitations of their Incident Response expertise and available personnel. They made it their main priority to get MOXFIVE involved at the decision-making level with a comprehensive view of all workstreams and issues. This included the prioritization of providing external access to engineers, supplying proper permissions for accounts, and allocating time to give an overview of the environment and current status. While working with Acme Corp leadership we quickly delineated workstreams that needed to be handled internally by the client due to level-of-risk and/or environmental knowledge versus what MOXFIVE engineers could take ownership of, enabling us to quickly scale the recovery activities. The main takeaway is Acme Corp leaned on the MOXFIVE playbook to identify priorities, risks, and repeatable tasks that could leverage engineers in a surge capacity.

The second client, who we’ll refer to as Blues Corporation, quickly engaged MOXFIVE and stated they had identified workstreams for us to assist with. When trying to get an overview of the current state of the environment and overall tasks being worked on across all teams and third parties, Blues Corp quickly returned the focus to the narrow workstream they had identified. While this was less than ideal because we knew there were other areas that needed assistance, we pushed on hoping that a quick win and proof of our organizational and technical skills would build trust, allowing us to advise on the project at large. Very quickly another hurdle was introduced, receiving proper access to perform the work requested. An expected timeframe for access being provided came and went, with the stated reasoning being the internal system admins were too busy evaluating and restoring services to provide access. While attending executive meetings, we learned our estimated access timeline wasn’t the only promised timetable being missed. Internal IT was so heads down focused on singular tasks they couldn’t appreciate how the slight pause to introduce our playbook and engineering team would open them up to perform higher priority recovery work, focusing on the environmentally specific work that could not be handed off to a third party. Unfortunately, this is not an uncommon misstep we’ve seen clients make - often due to the initial stress of responding to such an event or fear of relinquishing control. In this case, the MOXFIVE team could only advise on containment/recovery strategies and assist the internal engineering team via screenshares for the first four (4) days.

Hitting Our Stride

Once the recovery engagement was in full swing, Acme Corp continued to leverage MOXFIVE’s experience. Introduction of a new workstream or a large hurdle to an existing one would trigger a planning meeting with Acme Corp IT leadership to ensure the decision was reviewed, discussed, and agreed upon before being implemented by either team. On more than one occasion, MOXFIVE Technical Advisors were able to identify more expedient or efficient methods for overcoming a hurdle or to streamline a task, allowing it to be done in parallel by our engineering partners. Thanks to our ability to quickly identify solutions and attain the green light from the Acme Corp leadership, the majority of the IT server environment was restored by the end of the first week.  

When access to the Blues Corp IT environment was finally obtained, MOXFIVE looked to identify areas where the containment and recovery tasks would benefit from engineering surge support. While we were successful in identifying repeatable processes that could be solely owned by our engineering partner, this was hampered by inconsistent decisions by Blues Corp IT leaders. After a strategy was discussed and agreed upon by all parties, our engineering partners would escalate that the internal IT engineers had been relayed a differing plan. The deviations from an initially agreed upon strategy caused the whole workstream to come to a stop for reconciliation. For example, the initial plan to recover core business application systems was to restore from viable backups and ensure containment by implementing an Endpoint Detection & Response technology being actively monitored by the forensic provider. However, once the internal and partner engineering teams began this process, Blues Corp leadership stated each system needed to be hardened using CIS Benchmarks as a standard, this change was only brought to MOXFIVE’s attention at the next scheduled technical status call, mainly due to an unexpectedly low number of systems successfully restored. I quickly looped in Blues Corp leadership and explained that not only would this lengthen the business interruption but would also receive unwanted scrutiny by the insurance carrier as it is often seen as an improvement to the environment. The decision was ultimately made to defer hardening until after the recovery had been completed but there had already been a delay in the restoration process. Due to these and other miscalculations, Blues Corp did not have the majority of their IT server environment recovered until into the fourth week.

Lessons Learned

One afternoon my department head asked how my current projects were progressing, it wasn’t until this moment that I realized the contrast between these two projects, or more accurately, these two clients’ decisions. Acme Corp had fully accepted our assistance seeing MOXFIVE as a partner with a common goal, recovering the business from a detrimental event. Acme Corp had the final say on all decisions but increasingly sought our opinion, as well as the forensic provider engaged, on how to contain and recover the environment in a safe and expedited manner. At first, Blues Corp perceived our services to solely be “smart hands”, simply additional resources to task out what they saw as the current priority. This misperception resulted in delays to MOXFIVE being engaged at full capacity which was ultimately to their disadvantage. Eventually as our advisement and assistance was found to be increasingly beneficial, the Blues Corp leadership turned to us as a valuable resource.  

During security incidents of any nature, leveraging the expertise of experienced practitioners is essential, but this only works if the assistance is accepted. Allowing our team to integrate with all respective parties and leverage our experience will lessen business interruption losses, unneeded efforts, and burnout of internal IT professionals.  

For more information or if you need help with a current incident, contact MOXFIVE at ask@moxfive.com or use our Contact form.