Ransomware Recovery Tales: Prepare for Battle

No one could have predicted the events of 2020 and not surprisingly the number of ransomware compromises. Adversaries have once again evolved and it is extremely important to pause, step back, and see if your organization has everything it needs to be prepared for a ransomware incident. This is only perpetuated with the increase of Ransomware as a Service (“RaaS”) bringing more threat actors into the ransomware space. This leads to more incidents, longer downtimes, and higher business interruption costs.

At MOXFIVE, we have seen many organizations invest in various security technologies and solutions that help monitor and protect their environment to varying degrees. However, when all their servers were completely down because of a ransomware incident, MOXFIVE has only seen a few of organizations that were able to recover quickly and effectively. For the organizations that did recover quickly, we found that the key to success wasn’t flashy — spoiler alert, it was ample preparation. Let’s break this down.

Prepare when you can, not when you are forced to

Abraham Lincoln said “Give me six hours to chop down a tree and I will spend the first four sharpening the axe.” The core of that quote is about spending the time to prepare so you can quickly achieve the task you are attempting to complete. If we apply that to a Disaster Recovery (“DR”) plan today, it parallels to spending the time to plan for quick recovery due to an unforeseen event. Unfortunately, traditional DR plans emphasize recovering from natural disasters. Given the rise in ransomware attacks, it is important to revise your DR plan to include cyber-attacks and the impact they have on traditional DR plans.

Defined recovery process

With an increasing number of different technologies and solutions that organizations use, it is important to understand what it takes to restore or rebuild them in the event of an incident. Work with your application teams and determine what are the requirements for each application, and what is the appropriate workflow. While each application will have specific requirements, a recovery process can be defined. At MOXFIVE, we have summarized what is critical to include when defining the recovery process:

• Roles and responsibilities: Define the different team members involved in the recovery process and their responsibilities. For example, backups team will be responsible in identifying and extracting backups, applications team will be responsible in validating and verifying the functionality of the applications, etc.
• Process to identify impacted systems: How do you identify systems on your network that have been impacted? Identify multiple mechanisms or processes that can help your organization determine which systems have been impacted. For example, you can use the priority list of systems and review each system or application from top to bottom to determine the level of impact, or if the majority of your systems are virtual, one could check each physical host and review the status of each virtual system.
• Process to determine the state of backups: As you are planning how to verify whether backups are viable or not, make sure to understand what types of backups you have in place (i.e. full system backup vs. data backup, cloud vs. on-premise), the backup restore process, and how long it takes to recover a system using these backups. Knowing this ahead of time will allow your organization to get a better estimate on how long the recovery efforts might take.
• Establish a prioritized list of systems: Identify all systems and determine their dependencies. From there, prioritize them based on the criticality of the system, application, and function. Bring in all key stakeholders across the business and technology teams to ensure this is as comprehensive as possible. This single step will instantly shave weeks of time from any recovery effort.
• Rebuild process for applications: Worst-case scenario, you might need to rebuild a specific application from scratch. If so, it is important to understand what the rebuild process for specific applications entail (e.g. system dependencies, vendor assistance, etc.), so that the rebuild process can start much more quickly. Separate guides and instructions should be created for specific applications.
• Layout communication channels and processes: Here you want to define the communication flow and how responsibilities are going to be passed on from one team to another. For example, once the backup team has confirmed that a server has been restored from backups, they should communicate to the applications team that the server is ready for validation. Being able to communicate during the recovery efforts is critical as it will prevent potential issues from happening, such as: process bottlenecks, systems that get forgotten, etc.

The overall goal for defining a recovery process is to set a structure and a workflow that an organization can use when faced with a ransomware incident. Because of the nature and urgency of ransomware attacks, having a recovery plan in place will help set the stage and the foundation for the recovery efforts.

Preparing Your Way To Success

One of MOXFIVE’s most successful tales involves a ransomware threat actor who managed to encrypt 300 servers and over 1,000 workstations in an environment. The organization under attack had many critical applications and dependencies across multiple systems. However, as soon as we were engaged to assist with the recovery efforts, we were provided a detailed list of all servers and their relationships which were grouped and prioritized based on criticality. Not only did the relationships between servers help us understand their core functions, it allowed us to plug their list into our MOXFIVE Recovery Tracker and collaboratively work together. A process was then formulated between MOXFIVE and the organization, which was based on their defined recovery process (i.e. how they needed to get their systems up and running based on their sophisticated environment).

MOXFIVE typically sees the identification and assessment phase take around three to seven days for an organization of similar size (i.e. 300 servers and 1,000 workstations) that is not prepared. In some cases, we have experienced organizations take longer than seven days to fully develop a list of priority systems. However, for this client, because they were well prepared, it took them one day to provide a full list of prioritized systems and a general recovery process. While recovery efforts take time, depending on whether they restore from backups or decrypt systems, getting everything set up in preparation of execution is crucial. Being prepared ensures that the recovery efforts will run smoothly. In this situation, we were able to decrypt 300 servers (with 20% of them being decrypted within 12 hours of obtaining a decryption utility) and restore core infrastructure within three days with a total recovery time of less than 12 days.

It is important to focus on the time it took to get things set up, rather than the time it took to recover the environment. For this client, we were decrypting systems with a reliable decryption utility provided by the threat actors. However, before that utility was provided, the client had everything prepared allowing MOXFIVE to quickly surge a team to execute on the defined process based on the criticality of the systems.

Conclusion

The secret to success is not always flashy. Sometimes it is just a bit of elbow grease and muscle to get things lined up the way they need to be. Much like with incident response, preparing ahead of time makes all the difference. Aligning internal technical and business teams is a critical component to this to ensure all key stakeholders are accounted for prior to being forced into a live fire exercise of recovery. While we all hope that no organization falls victim to a ransomware attack, following these basic steps can help ensure organizations are prepared for battle so they can find a quick and easy victory.