How to Write and Execute Great Incident Response Playbooks

Introduction

Security incidents of any magnitude are bound to happen within any organization, and they should be thoroughly investigated to prevent and protect critical data, resources and services. While it is hard to fully automate the investigation process, we can always introduce scripted plays for common occurrences we might come across – that is where the playbooks come in. Incident response teams rely on multiple game plans and guidelines to successfully prosecute security incidents. Playbooks are very important to establish in an organization since they will contain the necessary instructions to  handle a potential threat or suspicious event.

What is an IR Playbook?

While the incident response plan takes care of the general overview, playbooks will be tailored towards different, individual scenarios. Playbooks act like helpful manuals we can consult while facing a specific situation – playbooks should be concentrated around the type of incident that is likely to occur and should provide detailed, relevant instructions to encompass all possible cases for the scenario. These playbooks will be applicable for many iterations, thus it is important for them to be reusable. Although they would need to go through constant review with the latest findings and updates, these playbooks demonstrate support for litigation and historical experience which improve the IR process holistically.

It is important to establish the distinction between various incident response terminology before we dive deeper – an incident response plan is a guide that includes who and what should be involved in the response and it is crafted in accordance with company policy and legal & regulatory requirements. Standard operating procedures (SOPs) are about how to do something specific – running a set of terminal commands, processing and filtering log files. Tied to both of these concepts, incident response playbooks take into account how to determine the course of action in a potential incident – what questions to ask, SOPs to follow, what procedure to implement in the face of an event. We will be focusing specifically on playbooks and how to organize them for the purposes of this article.

IR Playbook Types

It is a valuable idea to consider structuring playbooks around specific events and analysis actions – such as entity playbooks, investigation playbooks, and containment playbooks.

One of the most common ones are investigation playbooks. They provide a series of questions we can pursue in specific scenarios, as well as additional routes to get to the root of the security event. Containment playbooks are often used in ransomware incidents and they inquire the extent of contamination within the system. As another example, remediation playbooks ask questions to

Writing Playbooks

While writing a playbook, an initial risk and security assessment should yield the type of security incidents we might expect to see with the particular systems and configurations in hand. Once we have determined potential cases, we need to think about how to outline the specific questions that may arise from the incident, then try to pinpoint the right places to search for answers.

In addition to the general overview, here are a couple of starting points to consider when drafting incident response playbooks:

• Consider the first time reader – the engineer who was just hired or the new member on the project team. The plan should be easily understandable and implementable, with careful instructions that anyone could follow.
• Remember the human is always at the center of the investigation – a real person will discover the incident, another will execute the remediation plan, another will answer questions. Putting the human at the center of the questioning and executing process will remind us of our purpose.
• Employees and affiliates within the company should be made aware of the incident response plan and given training on how to properly execute it. Suspicious activity could be noticed by anyone in the company at any given time, so this becomes especially important to reduce response time.
• Re-consider document access protocols when it comes to sharing playbooks – the best place to store them is the company Wiki for accessibility.
• Incidents are high stress situations and responders are likely not operating at the peak cognitive ability during an incident. Playbooks should ensure that nothing is forgotten in the fray of battle.

Playbook Organization

Praetorian recommends playbooks be organized around common “alerts” and “entities” that may indicate suspicious activity upon further investigation. These are generally good starting points to keep track of how an attack started and progressed through the system, in order to figure out attack paths. Some of the common entities that could give us useful information about the system are IP addresses, file hashes, domain names and usernames. Related to these credentials, we can look for interesting events and common alerts, across the resources and platforms used, that might indicate a potential compromise – such as “user was added to the privilege group” or “unmanaged cloud resource instance discovered”. These events could mark the potential roadmap of an attacker in the system and will help us identify exactly what type of security event we are dealing with.

After taking a look at these alerts and initial points of investigation, you should start thinking about the major components of a playbook – questions asked and actions taken.

1. Questions are asked to pursue evidence and build a timeline. Building a timeline of events from logs while collecting evidence is very helpful when it comes to tracing down the attacker’s path and understanding which parts of the platform are impacted.
• Ex: was there a request to a specific endpoint in the past day?
2. Based on the answers, investigative actions are taken to seek answers and find new questions. It is likely that one log and piece of evidence will lead to another, it is important to leverage action as appropriate and making queries to the entities that seem suspicious.
• Ex: in a phishing incident, we need to gather the number of targeted accounts by checking how many users received the email. An investigative action for this will be to query email logs, based on subject or sender’s address.

Upon receiving an alert, it is important to remember that the first questions we ask will most likely be predictable – the answers to these will help shape further questions, which may not always be predictable. As we dive deeper, the third layer of questions will not be predictable and may require more work to find answers.

Example of an IR Playbook

Let us walk through an example playbook which will hopefully better illustrate the concepts above.

Consider a phishing incident playbook. As first steps, we think about what questions we can ask to reach some evidence and draw some more connecting lines to further inquiries. Eventually what a playbook amounts up to is a series of questions that lead us to the correct path of collecting data, and this is what an investigative playbook for phishing events will demonstrate.

Case: employees come across potential phishing emails sent to company addresses.

An example line of questioning could be:

1. How many users received the email?
   • Gather the number of targeted accounts. Check email headers to get an understanding of the source.
2. How many users opened the email?
   • Out of all who has opened, did anyone come across suspicious content?
   • Did they interact with the email in any way, i.e. download attachments or click any links?
3. What kind of content does the email have? 
   • Make sure to cover all that is included in the email.
   • Examine the body to check for any links, attached addresses and files, images.
   • If the email does not contain any links or attachments, refer back to the message body. Potential language analysis, continue search for message ID and headers.
4. If the email included any attachments, were they malicious?
   • Who created the attachment file? Check out the file history and/or look for author clues within the metadata.
   • Was it an executable? Check current processes within the attacked system.
   • Gather the file hash, check OSI for any related information.
   • Check the file extension. If it is an HTML-based page, try opening it in the notepad to see if there is a referrer address. If it is a PDF file, examine file in a separate VM for any potential threats.
5. If the email included any links, what are their origins?
   • Check out the domain names and addresses to see if they were flagged as malicious.
   • Monitor web traffic to the destination domain from the time of the email delivery.
6. Did anyone respond to the email?

Executing Playbooks

Up to this point, we have a solid understanding for what goes into crafting the IR playbook – yet all of the instructions and potential scenarios describes will be useless if we do not know how to properly execute the necessary measures.

First steps for executing playbooks properly are awareness and proper communication. Anyone within the company should be made aware of the incident response procedures and plans that are in place, apart from the IR team themselves.  All users need to know proper escalation paths, which should be made available in a company-wide resource, such as a Wiki page, and updated regularly as the need for different policies may arise. Only investigators (IR team) should have access to IR playbooks to avoid investigative data breaches.

It is important to note that while IR playbooks will address a wide spectrum of cases they should also adapt to unforeseen issues and situations that may arise. Platforms used across the company may change and update, the IT structure may be altered, additional security measures could be implemented… all of these factors will most likely require a change in the existing playbooks depending on the situation. It is possible that most of the cases that we plan playbooks around are purely based on assumption and hypothesis, which will be updated over time.

Conclusion

Incident response plans and accompanying playbooks are vital to an organization as potential threats arise.  They should be highly readable and easily understandable as well as detailed, specifically tailored towards the nature of the situation – technical or not. As a rule of thumb, it is a good idea to keep them short, sweet but concise. Having organized and communicated the plan augmented by the playbooks will mend the impact of the potential incident ahead of time.

Resources

Incident Response: Writing playbooks
NIST Incident Response
Crowdstrike: Incident Response
Incident Response Plan for Startups
Creating Incident Response Plans
Investigative Playbooks – Symantec