Log4j: It’s worse than you think

On December 9th, 2021, a new 0-day vulnerability in the popular Java logging package log4j v2.x was announced. The vulnerability is particularly unpleasant as exploitation frequently requires only the ability to cause the system to log an attacker controlled string to a vulnerable logging instance. Thus, the routes of exploitation are broad and often require no level of authenticated access. The vulnerability is of critical importance, and needs to be addressed immediately by defenders.

There have been some solid technical writeups already by security researchers, and so we will not replicate that information here. Instead, our goal with this post is to help customers understand the seriousness of this vulnerability, how widespread it is likely to be, and what defensive actions they can take to at least reduce exposure while a permanent fix can be deployed.

There are three primary factors that make the log4j vulnerability (known more formally as CVE-2021-44228) particularly dangerous, leading to a CVSS score of 9.8 – making this vulnerability almost as bad as you can get.

1. The vulnerability is relatively easy to exploit – that is, an attacker just needs the ability to control strings that will get logged via log4j.
2. Due to the wide variety of different routes for attack (lots of different operations get logged and could be attacker controlled) the only complete way to be rid of the vulnerability is to upgrade to a patched version.
3. An extraordinarily large number of components make use of log4j – as such, the blast radius from this vulnerability is immense.

Praetorian researchers have already fully weaponized the vulnerability in order to better understand its implications, and we have used this tooling in the wild on customer systems to demonstrate the issue. We intend to fully disclose the technical approaches we took, but are planning on postponing any technical release that could further detract from customer security. However, once exploits become more widespread, we intend to share a full analysis plus detailed detection information.

We have also created a set of vulnerability scanner rules that allow us to detect many – but not all – instances of this vulnerability in the wild. This tool works by partially triggering the vulnerability, and as such, we feel that releasing details at this point would be detrimental to defenders. Instead, we are offering to scan a domain remotely if requested using our Attack Surface Management pipeline. We believe this approach maximimizes benefits while minimizing additional harm. One concerning issue we have discovered while testing this tool with customers is that we have inadvertently discovered this vulnerability in adjacent systems that were not the direct target of our scans – worse yet, these adjacencies were not obvious even to well informed defenders.

Technical Details

As noted above, there are already several good technical explanations of the issue, so this section is designed to briefly orient the reader to the “what” and “why” of CVE-2021-44228.

In short, the vulnerability stems from the fact that Log4j evaluates log statements for certain “lookups” before writing to the log file. The most dangerous of these performs a JNDI lookup on a user-controlled value. JNDI is a well-researched attack surface in Java (source: marshalsec). These Log4j lookups are evaluated even when user input is passed to a log statement like the following:

		import org.apache.logging.log4j.Logger;import org.apache.logging.log4j.LogManager;Logger logger = LogManager.getLogger();logger.error("Hello, {}!", userInput);	

A simple demonstration of Log4j lookups is to set the userInput value to `${env:PWD}`. If the Log4j version is vulnerable (i.e. it allows lookups), the `$PWD` environment variable will be written to the log. We’ll reserve discussion of the various JNDI attack vectors for a future post.

Advice to Customers – Mitigation

As part of our research, we have developed a log4j vulnerability scanner into our new Attack Surface Management module in Chariot. Based on the nature of the vulnerability, we cannot prove a site is invulnerable to this attack “from the outside in” (that is, by only scanning externally) but we can identify several cases when a site is vulnerable. We have discussed at length sharing the expoitation techniques with others, but believe that to release this tooling would lead to widespread abuse, as our tool could be trivially used to identify vulnerable sites and then compromise them – clearly not a good thing. As such, if you would like us to scan your domains for this vulnerability, you can reach out to Richard Ford (richard.ford@praetorian.com) and we’ll do what we can to assist.  If you are an existing customer, please reach directly back to your Praetorian points of contact.

If you are aware of a vulnerable system, there are a few different steps you can take to contain the issue:

1. Update Log4j. The best defense against this vulnerability is to patch all 2.x Log4j versions to 2.16.0.
2. Disable Log4j Lookups. At its core, this vulnerability requires Log4j lookups to be enabled. Per this [Apache Jira ticket](https://issues.apache.org/jira/browse/LOG4J2-2109), after Log4j version 2.10, passing `‐Dlog4j2.formatMsgNoLookups=true` to the JVM will disable Log4j lookups and protect the system from exploitation.
3. Remove dangerous .class files. Some systems may depend on Log4j lookups for logging, but may not need the JNDI lookup feature. It is possible to modify the log4j JAR file and remove these dangerous class files. This can be accomplished by extracting the JAR and deleting the `org/apache/logging/log4j/core/lookup/JndiLookup.class` class file.
4. Block dangerous requests at the WAF. The JNDI lookups required for exploitation have a required prefix for Log4j to perform the lookup. Any request with with a string like `${jndi:xyz}` will trigger a JNDI lookup. The prefix `${jndi:` could be added to a blocklist in a web application firewall to block exploit attempts. Note that this approach is by its nature incomplete – we are aware of methods to bypass this, but this simple approach will at least make scanning by some lower-level attackers less effective.

In addition to these approaches, if you are making use of dependency analysis and container scanning in your CI/CD pipeline, you should be able to quickly identify vulnerable versions of software. While patching can take time (test coverage is seldom as broad as it should be), ultimately, the only way to truly remove this vulnerability from your environment is by patching or otherwise replacing effected components.

Don’t Panic – but Do Act

Things are pretty messy right now, and the situation will continue to evolve overnight and through the weekend. There is most definitely cause for alarm, but as touched upon above, there are active steps you can take to reduce your risk – even if elimination can take significantly longer due to the many tendrils log4j has. Repeating the obvious steps an attacker will trace either manually or using our Attack Surface Management tooling (see above) can buy time by uncovering trivially vulnerable systems but our suspicion is that this vulnerability will follow the pattern of other hard-to-elimate weaknesses that we still see being targeted today.

Unfortunately, due to the widespread use of 3rd party libraries and components, the disclosure of 0-days is not likely to stop any time soon. In this case, our assessment is that things could have been worse, as closing the floodgates is relatively quick – but tracking down every last instance of vulnerable log4j instances is going to take time. Customers are strongly advised to take the steps outlined above as soon as possible – this isn’t something that can wait until Monday.

We do expect our understanding of the situation to evolve quite quickly – as our mantra is to first do no harm, we have witheld important information regarding log4j that we feel would only make the situation worse. As our understanding of risks becomes clearer, we will keep this blog post updated, calling out new information as it becomes valuable to do so. We’ll also update on Twitter (@praetorianlabs) when we have more to share.

Update: Dec 11 8:58am CT — There have been a few changes in public reporting, but it still appears log4j 1.x is not vulnerable – is correct. We have reviewed the code for Log4j 1.2.17 and 1.x does not contain any of the same lookup / template evaluation code, and the only class related to JNDI (JMSAppender) does not appear to connect to user-controlled remote systems based on log events. On the other hand, 1.x is EOL and any latent vulnerabilities are unlikely to receive patches. For this reason it is still worth prioritizing updates in the medium term given the likelihood of increased eyes on the 1.x codebase.
Update: Dec 13 1:40pm CT — Read additional updates here.