Incident Response Best Practices: Building an Evidence Wiki

What is an evidence wiki?

As Blue Teams work to secure systems, it becomes especially important to keep track of interesting and helpful information gathered through the investigation process. During the investigation of a security incident, one of the very first things teams do is to create a timeline of events via checking various resources. Depending on the comprehensiveness of the investigation, this process could take a long time especially if it is not clear where and what to look for.

Alongside correctly evaluating the situation and knowing where to begin the search, the successfulness of an incident response (IR) depends heavily on collecting the right information and evidence in an orderly manner. This makes it necessary to create a resource where evidence sources are conveniently documented.

An evidence wiki is a collection of the resources that an organization has access to that can be used to create a timeline – it is essentially a compilation of metadata about various logs sources and informative files that allows investigators to quickly look up artifacts and understand their efficacy to the current investigation. In addition, it should also include the steps to actually collect and extract information. It can also be very helpful to get any new hires, engineers and consultants to the company up to speed since it provides a resource that demonstrates data established collection requirements. Having a standard of what logs and resources are available and how to utilize them helps in identifying pivots during the research process – by providing alternatives and other tools to gather information, it can keep investigators from focusing too much on a a single source.

Praetorian recommends the mantra of “following the evidence” during engagements – the logs, records and various artifacts with timestamps will help construct the timeline of the incident and potentially lead to more questions as well as points of interest.

Why have an evidence wiki?

Creating an evidence wiki for incident response teams is a company-dependent solution, based on respective needs and factors such as size and assets. Some argue that a wiki is not needed since some small IR teams only contain a handful of people, others rightfully claim that it is a lot of maintenance work to update and maintain the resource. While these may be relevant concerns, having an evidence wiki could save time when new consultants and engineers need to be onboarded for tasks such as compliance assessments or incident responses and threat hunts, as they will have an established resource to refer to. Plus, the evidence wiki could also help with other engagements like tabletop exercises while looking for specific logs.

Writing an evidence wiki

The purpose of seeking evidence throughout an IR is to prove or disprove the hypothesis – we start the investigation process with a set of potential causes and risky endpoints in mind, theorizing on where exactly the incident may have originated from. Collecting logs, access and authentication information and records enables us to construct a roadmap and draw further conclusions.

First, we should identify the resources that the company is actively using and compile them – the following questions will help gather a better understanding of the system.

• What cloud provider does the company use, Azure,AWS,GCP?
• What are the multi-factor authentication and SSO platforms currently in place – Okta, Duo?
• What are some email platforms that are currently in use – Outlook, GSuite/Gmail?
• What operating system is used – Windows, MacOS, Linux?
• Any other relevant software/tools – O365?

After obtaining the list of relevant platforms, the evidence wiki will be constructed around it.

Here are important elements to consider while listing logs:

• Access: look for ways to access the logs for the platforms – is there a requesting access process?
• Location: how to access this log – is it local, stored within a SIEM, web-based? (local, SIEM, URL, index, etc.)
• Storage: note down how and where the logs are created as well as stored.
• Retention: include where the logs are retained and by who – it could be through internal IT infrastructure, software such as CrowdStrike etc.
• Legal: specify any legal retention requirements and standards for the logs, such as PCI and SOC2.
• Coverage: what systems/environments contain this log, and which ones are known not to have it?
• Usage: what sort of information will this log provide? which log should you go to if you need an IP address, domain, username, hash etc.?

Example Evidence Wiki

After identifying the key systems and tools, we can start building the evidence wiki based on the specific resources.

In an example scenario, say we have identified that our Endpoint Detection and Response (EDR) tool logs are retained through Crowdstrike. Here are the key elements that might be included in the evidence wiki:

EDR Logs:

• Access: request a ticket from the IT Helpdesk, assign the ticket to an employee
• Location: CrowdStrike logs can be found at at falcon.crowdstrike.com. Alerts are forwarded into the SIEM (siem.example.com).
• Storage: CrowdStrike logs are created on endpoints with the Falcon agent. They are also stored in the cloud which is outside of ACME’s control.
• Retention: CrowdStrike is configured to store 60 days of logs in the Falcon console. Additional logs can be stored in AWS S3 buckets for up to 1 year. Alerts forwarded into the SIEM are stored indefinitely.
• Legal: Currently, there are no legal requirements to maintain EDR logs
• Coverage: All workstations should have the CrowdStrike agent installed. Servers do not have CrowdStrike, but “flex” licenses can be purchased and used, if needed.
• Usage: CrowdStrike collects logon events, network events, registry events, services events, malware detection events as well as process execution.

Potential Evidence Data Sources

Evidence sources will differ based on what sort of information we are looking for. Some of the most searched data types that often give useful information are IP addresses, domain names, usernames and file names and file hashes. While collecting evidence, these data types will be our starting point, which we can obtain from a variety of logs.

As an example, information about the network could be obtained from VPN access logs and packet capture software such as Wireshark logs, in addition to examining the network flow throughout systems and applications.

Another example set of information sources for any of the above data types within the disk logs are Windows Security Logs and Authentication logs, which contain login and logout activity based on the system’s security policy.

Using Evidence Sources

Once the resources are compiled, the question becomes – what is this evidence useful for? What can we do with the information that the logs give?

There are various types of evidence to look out for, such as authentication logs, application access records, cloud resource access logs.

Amongst the many tools that an evidence wiki can contain, there are varying levels of efficiency while using each of them. Certain log types may be very detailed and harder to parse, and thus might be better for more general queries such as detection and alerting – PCAP network traffic logs is an example. Easier to parse DNS logs might be a better choice for investigation (alongside NIDS, HIDS, VPN logs), however, since they are so common, they would not be the best choice for alert checks. As we choose what evidence source to use, it is important to remember the tradeoffs between using one over the other – we should consider the environment they are stored in, how easy the log is to parse, how common it is.

A good rule of thumb to follow is that pre-parsed logs that pull out important information quickly will be efficient, compared to the longer, more complicated logs.

Conclusion

Evidence wikis are essential resources to conduct a successful, organized and quick to act incident response. Obtaining and preserving critical evidence helps us better understand the incident and its’ effects, while laying the groundwork for further investigation and potential threat eradication.

Resources

NISTIR 7622
Digital Evidence Collection
UN – Handling of Digital Evidence
Crowdstrike – Red Team vs. Blue Team