Introduction
In this blog post, Praetorian reflects on customer challenges, successes, and lessons learned from our response to the Log4j industry-wide response.
Background
On the Friday evening of December 10th, Praetorian research and development teams sprang into action, confirming vulnerable systems or exposed vulnerable endpoints for a large number of organizations. It is our belief that the correct approach to mitigating a zero-day vulnerability such as Log4j requires the conservative mindset that the attacker only has to succeed once. However, defenders have unique advantages over an attacker. We found that organizations who were able to quickly assess and provide an inventory of possible vulnerable systems – and were prepared to pivot into patching, detection, and response – fared far better than those that could not.
Organizations that were successful in mitigating Log4j have adopted the following Strategies:
- You can only protect what you can see.
- Proper security posture should be multiple layers of security controls. (Defense in Depth)
- Capitalize on your knowledge of the business and organizational context to inform your security teams on where to focus detection,reaction, and mitigation playbooks.
You can only protect what you can see.
Sun Tzu said a lot in The Art of War but one allegory we should consider is that knowing ourselves gives at least a 50-50 shot at success and knowing ourselves and our attackers gives us favorable odds; not knowing ourselves or our adversaries sets our organization up for failure.
Knowing ourselves:
Knowing our organization’s inventory, what we own, what has a support contract, what vendors are in the mix, and who is responsible from an operational and security perspective.
Knowing the enemy:
Knowing what our adversaries are doing in our environment through visibility and understanding what information and context is needed.
When Praetorian responded to Log4j, the first thing we asked for clients was for all DNS, hostname, and IP seed information for services they wanted us to evaluate. We chose this approach because many customers would only be able to give us seed domains and perhaps some limited IP ranges. We used discovery – just as an attacker would – and found things they didn’t know about.
Using our attack surface management platform’s feature of discovery, we enumerated their footprint and looked for active and responding hosts. For the responding hosts, the platform then searched for hosts that responded on commonly exposed web services. The platform then crawled all the exposed pages and tested the JNDI injection points. Through the platform, we monitored for callbacks and alerted customers to vulnerable systems.
We found that some organizations were unable to provide us with asset inventories, and opted to either provide a list of known hosts that they had in their CMDB or provided their entire external network ranges. In some cases, the client sent an engineer to manually search for all of their assets and collect this information. From all of these scenarios it was clear that providing this information was a struggle for organizations that had immature asset inventory capabilities.
Praetorian recommends leveraging an attack surface management tool to understand what assets are internet-facing, which of these assets are vulnerable, and as an organization establishing the following:
- An inventory of all systems (regardless of who is responsible for maintaining them).
- Contact lists of who to call if something is wrong and who is responsible for acting when something goes wrong. (Escalation procedures, RACI charts)
- Working partnerships and alignment between the security team and all business owners.
- Assigning risk as an organization to assets to determine response and priority.
- Collecting logs from systems related to the assets and having your security team create business centric SIEM use cases and playbooks to detect and respond to an emerging vulnerability.
Some examples of use cases for Log4j could include:
- Outbound LDAP connections.
- Outbound LDAP connections on non-standard ports.
- Anomalous internal enumeration of systems.
- Post exploitation command & control activity.
Defense in Depth
Defense in Depth is commonly referred to as the “castle approach,” because it mirrors the layered defenses of a medieval castle. There are multiple walls, gates, and drawbridges that have to be surpassed to enter the secured portion of the castle.
A common theme across Log4j as it unfurled was a false sense of security related to commercial tool vendors repeatedly stating that the vulnerability had been resolved, or a fool proof detection method had been found. In practice, there were several discovered bypasses for detection and mitigation technologies. Adequate defense of your organization requires the provision of multiple approaches and tools to secure your castle.
Praetorian recommends the following for a Log4j Defense in Depth strategy:
- Enabling commercial WAF provider Log4j rules as they are released.
- Patching systems to the latest version of Log4j and continue to monitor for newer more secure patches.
- Consider taking unpatchable systems off the public internet and requiring a corporate VPN or internal network routes to interact with such systems.
- Developing SIEM use cases as new exploitation techniques are discovered and make sure to conduct historical log searches to see if these have happened in the past and alert if they continue.
- Setup egress filtering rules for non business case applicable out-going system traffic such as LDAP.
- Ensure systems use internal DNS servers and leverage DNS filtering technologies such as InfoBlox.
It’s not Game Over
If your organization was affected by the Log4j vulnerability and an attacker did exploit a system in your environment, now is not the time to be discouraged. You need to focus on understanding how the attacker entered the network and what level of access they have achieved and routing the attacker. This requires following your organization’s security playbooks and focusing on what you can do in this situation versus what you can’t do.
Don’t worry about what you can’t do, worry about what you can do; and make do.
- What personnel, resources, tools and approaches do you have at your disposal.
- What is the quickest and most effective way to resolve the current situation even if it breaks with industry practice or convention.
- What are different facets of the problem that can be tackled at the same time? What facets can be worked in parallel?
- Assign teams to each problem with a central dedicated communication and update channel.
- Work to fix the problem at hand with what you have in the quickest way possible.
Lesson Learned
The most important lesson we learned from Log4j is that focusing on the foundations of your security program will place you in a less vulnerable position when these zero-days are discovered. We found that organizations who were more mature in their security foundations bounced back faster and faced fewer financial implications. By focusing on maintaining asset inventories, security processes/procedures, security-business priority alignment, and maintaining a strong and consistent security posture your organization will be less likely to see major effects of the next Log4j.
