We had a busy weekend here at Praetorian. Following the initial disclosure of the Log4j (Log4Shell)* vulnerability, we’ve added a capability to identify the issue to our attack surface enumeration tool. As we scanned production environments for the vulnerability over the weekend we’ve learned a lot more about how the issue can actually look in the wild. We’ve been continuously improving our scanner based on these lessons learned, and we wanted to share our observations with the wider community.
First and foremost, we know this issue will be prone to false negatives; that is, externally there is no way to cover all the possible paths that exploitation can take. Even when external scanning tools get more sophisticated in how they identify the issue, we strongly advocate not relying on scan results as strong indicator of your risk. We base this assessment on a few observations:
- The issue can take an unusually long time between a malicious request and the attacker payload executing since exploitation doesn’t occur until the malicious string reaches the vulnerable logging function. In our research, we’ve seen the time from request to exploit take more than twelve hours. We can conceive of some architectures where that time may stretch even longer. Most existing testing methodologies, both automated and manual, do not listen this long for a payload callback and instead close listeners after a few seconds or minutes. To have a reasonable level of confidence in negative results for this issue, it will be necessary but not sufficient to leave listeners on for dramatically longer than is typical.
- WAF providers have rapidly added filters for typical malicious strings that probe for the issue. It’s great to see their responsiveness to this issue; however, it means that straightforward POC probes can no longer be relied on. WAFs may provide a false sense of security when the underlying app remains vulnerable.
- Outbound filtering in many environments may prevent the non-harmful payloads that most scanners and researchers use to test for the issue. For example, we’ve explored using a DNS lookup payload to test if our input is executed by a backend system. In working with clients over the weekend we identified a case of a confirmed vulnerable system from which we couldn’t receive a response to our probes. In working with the client, we identified that outbound filtering was blocking the response and causing a false negative determination. In these circumstances, a true attacker who was willing to trial more aggressive payloads would likely be able to exploit the issue.
- Exploiting Log4j requires an attacker to get very specific text to be logged. That means that an external scan needs to understand every possible route that an attacker can travel to get tainted text to hit a vulnerable logging component. While Praetorian has written specific rules to exploit common services, it is not feasible to cover all the possible paths from the network to logging services; because of this, the only way to be completely safe is to patch to a known good version of Log4j.
Based on our experience in identifying and exploiting the issue thus far, we do have a few more recommendations to add to those we provided in our first blog post
- Monitor vendor notifications – Many vendors over the weekend have confirmed the presence of the vulnerability in their products and begun issuing mitigations. In the immediate term, these notifications may be the highest confidence way for you to identify vulnerable assets in your environment.
- Implement Defense-In-Depth Measures – An effective WAF and/or outbound filtering can make it much harder for an attacker to identify a vulnerable system and increase the likelihood that they will move on before they successfully identify and exploit the issue. If not implemented, we would suggest both of these tools as ways to reduce your risk. For outbound filtering specifically, we suggest blocking DNS and TCP traffic. If you are able to block outbound connections for those two protocols entirely from affected systems, you can effectively mitigate the issue.
*The Log4j flaw (also now known as “Log4Shell”) is a zero-day vulnerability (CVE-2021-44228) that first came to light on December 9, 2021, with warnings that it can allow unauthenticated remote code execution and access to servers. Praetorian was among the first to demonstrate a successful exploitation in the wild, and the potential impact of the vulnerability.
