Overview
Praetorian recently performed an assessment of a platform responsible for downloading and building untrusted, user-supplied code. The client was concerned about the possibility of attackers leveraging this process to compromise the client’s AWS environment or gain access to sensitive data belonging to other users. Their solution to sandboxing untrusted code builds was to perform them on a serverless AWS Lambda Function with minimal access to the rest of the AWS environment. Indeed, Praetorian engineers found a way to execute arbitrary code using this process and set out to explore the extent to which they could exploit this foothold in the client AWS environment.
Lambda functions are, in many ways, a wise solution to this problem. They are provisioned with an IAM execution role that restricts the operations within the cloud environment the Lambda function can perform. In this case, the execution role was correctly provisioned with the minimal privileges needed to perform its intended behavior. Lambda functions also execute as a low-privileged user within the Linux environment they run on, and there are few other tools available to the Lambda runtime that allow adversaries to “live off the land“. However, after reading research from Yuval Avharami and Nick Frichette, Praetorian engineers leveraged the Lambda runtime API to find that the client’s method for isolation exposes critically sensitive data.
What is the Lambda runtime API?
The research by Yuval Avharami linked above does an excellent job of breaking down what the Lambda runtime API is and how it manages incoming Lambda function events. The key takeaway is that it is a listening service on port 9001 that is responsible for returning incoming event data and terminating events with either successful responses or errors. From a security perspective, this presents the possibility for code running on the Lambda to terminate the event that triggered it and then to listen for future incoming events. For Lambda functions that are provided sensitive information (such as tokens or PII) via the event payload, this API represents a way for attackers with code execution to expose these secrets.
Why was the Lambda isolation implementation vulnerable?
For this engagement, we tested a product that integrates with the user’s GitHub account to scan their repositories. The application may also resolve dependencies and build the project during this process. Since some project manifest files can have scripts that get executed during the build process, the developers of this application decided to use AWS Lambda instances due to security concerns. Their thought was that, even if an attacker scans a repository containing a malicious manifest file, they would only achieve code execution on an ephemeral Lambda instance. The only valuable secret provided to the Lambda is a GitHub token belonging to the user who initiated the scan. Hence, the attacker would only have access to a GitHub token that belongs to their own GitHub user.
However, after adding a GitHub repository to the application, it also attaches a GitHub workflow to the repository. We discovered that pull requests from any GitHub user account could trigger this workflow, leading to the Lambda being invoked with the code provided in the pull request. The untrusted code from the PR was then checked out and built on the Lambda instance. An attacker could then access the victim’s GitHub token by querying the Lambda API endpoint and viewing the event JSON. In our case, the GitHub tokens were used in the event payload and were configured to have the “repo” scope. Anyone with this token has permission to read and write all of the user’s public and private repositories. The attacker can also capture other users’ GitHub tokens by querying the Lambda API to terminate the current innovation and then wait for the Lambda to be triggered by another user.
Attack Chain
Praetorian created a script to perform automated retrieval of the event JSON containing sensitive data to show the client how an attacker might maintain persistent access to incoming sensitive information. Once this script is run on a compromised Lambda function, it will “call back“ to an attacker-controlled server with the contents of any future events. In this case, those events will contain captured Github tokens from other users. Because the execution of Lambda functions is capped at 15 minutes, it is necessary to continue to regularly invoke the Lambda as long as the collection script runs to prevent the Lambda from going “cold“ and starting anew with a fresh environment.
Praetorian engineers also discovered a command injection vulnerability that could be triggered by changing the name of the GitHub repository’s “default” branch. This represented another entry point into the target Lambda function and allowed engineers access to previously uncompromised build environments.
Command Injection via GitHub branch name
After gaining a foothold on a Lambda instance supporting one language, the attacker can leverage the stolen GitHub token to modify the GitHub branch of another repository to get code execution on a Lambda instance for another language (depending on how the repositories are configured).
Remediation recommendation
Due to the use case of the Lambda functions in use, it is not possible to entirely prevent attackers from running code on these Lambda functions. The types of projects that can be scanned by this service inherently include code that is run when the project is built. Thus, Praetorian’s recommendations focus on reducing the security impact of running arbitrary code.
Firstly, Praetorian recommends that the client no longer perform builds of code included in pull requests. Pull requests can originate from unauthenticated actors (with respect to the target platform), and executing this type of untrusted code allows adversaries to view the victim users’ Github secrets and subsequent updates to the project.
To address security risks stemming from executing authenticated users’ code on the Lambda functions, Praetorian recommends that the client consider installing a sandbox environment on the Lambda function that restricts access to the runtime API and any parts of accessible memory that contain the event payload. Praetorian also recommends AWS NACLs that prohibit all non-essential outbound traffic. The client could implement a whitelist of acceptable addresses to access and prevent traffic to all other addresses.
Conclusion
Though Lambda functions are ephemeral and run in containerized environments, they do not provide complete isolation from a security perspective. It is possible for attackers with the ability to execute code remotely to read sensitive event data and persist on the Lambda and intercept incoming events, even if the Lambda executes with proper IAM privileges and contains no other secrets. A complex use case like this one requires complex security controls, and luckily the client had Praetorian to identify this vulnerability and suggest a remediation strategy.
