How automated policy enforcement using OPA can help secure Red Team Architecture
Introduction
The typical Red Team engagement utilizes several different tools that are used to accomplish different goals throughout different stages in the engagement’s lifecycle. Engineers may use several different frameworks to gather and organize information about a target, a handful of domains and servers for phishing attacks and command-and-control, and an assortment of various post-exploitation tools and tricks to achieve an objective. All of these items and their associated dependencies results in a complex infrastructure that can take days to configure correctly. Additionally, starting with a clean slate for every engagement is a core principle of our Red Team operations as it provides a known state without residual artifacts from previous engagements and ensures proper segmentation of any potential client data. For these reasons, infrastructure provisioning automation is a logical step forward in improving our Red Team capabilities.
Terraform is a popular framework that enables engineers to define an infrastructure as code which leads to automated, configurable, and repeatable processes to handle the complicated nature of a Red Team Infrastructure. Instead of spending valuable time on each time-boxed Red Team engagement setting up the infrastructure and ensuring that servers, networks, and firewalls are all configured correctly, the infrastructure can be defined once and automatically deployed using Terraform. The more time that engineers get to spend working towards achieving the goals of the engagement has a direct correlation to the Red Team’s success rate.
While automating infrastructure provisioning can ensure the consistency of an environment, that does not necessarily mean that it complies with the policies that govern how the infrastructure should be configured. In this blog post, we will discuss how DevSecOps practices can be leveraged to preserve the agility and convenience of infrastructure automation while also increasing the confidence that engineers have in the integrity and confidentiality of the operating environment.
Policies and OPA
Taking time to define and document policies for configuration, management, and operations is imperative to creating structured processes and ensuring compliance requirements are met. Even with a single source of truth for policies, lackluster governance over their enforcement severely reduces their value. In the case of Red Team infrastructure, policies can define what network traffic is allowed in and out of the environment, what software and which version can and should be used, and how data should be handled. When defining the Red Team infrastructure in Terraform these policies should be kept in front of mind, but that is not always the case and policy violations can slip through the cracks and make their way into deployments.
Our Terraform code is stored in GitLab which provides version control and enforces peer reviews of changes made to the infrastructure definitions. However, engineers review changes – and engineers are human and prone to missing policy violations. This is especially true when you consider that reviews can take a non-trivial amount of time and aren’t exactly the most exciting part of a security engineer’s day. The obvious solution to this problem is to automate as much of the review process as possible, and that is where conftest comes to the rescue.
Before we jump into what conftest is and how it helps solve our problem of policy compliance, we need to talk about Open Policy Agent. OPA is a policy enforcement framework that allows you to decouple your policies from your software. When a principal in an environment wants to see if they are allowed to do something, they can query OPA and OPA will make a decision by evaluating the data it has been given against a set of policies that are defined within it. For example, say you have a policy that states that privileged docker containers are not allowed to run on production servers. You can define this policy in OPA’s Rego language and whenever a Docker container attempts to run, it will query OPA and OPA will evaluate the Docker container’s configuration to ensure that the Privileged parameter is set to false. The snippet below shows how this policy may look in Rego.
This is great, and OPA is able to scan Terraform plans for any policy violations. So why would we want to use conftest? OPA is designed to be set up somewhere in an environment where it can be queried by various different components for information about the decision that OPA has made in regards to their request. This design decision is great for enforcing runtime policies and for centralizing decision making across entire environments, but conftest provides features that play nicely when implemented into a CI/CD pipeline.
Conftest
Conftest is an extension of OPA and there are several similarities between the two. What conftest provides that makes it a better choice in our use case is that you don’t have to create explicit queries to conftest to get the results of the evaluation. Conftest looks for deny rules that you have defined, evaluates the input against it, and provides the output for any policy violations right back to you. Let’s use one of our use cases as an example to see how this looks in action.
A critical attribute of our Red Team infrastructure is that strict ingress and egress controls are maintained throughout the environment. We want to avoid situations where ingress traffic is allowed from the internet to the teamserver’s management port or any SOCKS listeners that it may have opened. To avoid this, and as a best practice, we want to deny all ingress traffic by default and explicitly define what traffic is allowed. The traffic that we do want to allow is ICMP and HTTP traffic from CIDR blocks owned by Fastly to our teamserver, and ingress SMB traffic from the internet to Responder. Now that our policy is formalized, we need to determine how we want to get our data from our Terraform code into conftest and what that data will look like.
As mentioned earlier, our Terraform modules are managed in GitLab, and to avoid accidentally deploying a non-compliant infrastructure, we want to catch any violations before they make it into the master branch. This is pretty straightforward, all we have to do is have conftest evaluate the changes that have been made in each PR and ensure that the new infrastructure doesn’t violate our policy. To accomplish this we can leverage Gitlab CI/CD and implement conftest into our pipeline to run each time a PR is made to the master branch. Additionally, since we just want to check the changes to the environment, we can generate a Terraform plan and use that as the input to conftest.
Writing our Policies
Now that we know what the data that we are going to be working with looks like, we can define our rules. First, we will define some helper functions to parse the data and get us the information that we will need to check if a firewall violation exists in the planned state of the infrastructure.
Here we have defined four rules: ingress_to_port_from_fastly, ingress_over_protocol_from_fastly, defined_ports, and defined_protocols. Defining these rules in their own module will allow us to write our deny statements in a way that is easy to read and also enables us to easily add and modify any of our policies if need be. Next we will define our statements which actually make the decision to either accept or reject the PR.
Here we have five deny statements. The first three ensure that any ingress ICMP traffic or any traffic to 80 and 443 is only allowed from Fastly owned CIDRs. The last two ensure that no port or protocol outside of our allowed lists are defined in any of the firewall rules. Now to validate that we have defined our policies correctly, we can run conftest against some compliant and violating Terraform plans. The proper way to do this is to write formal tests, but for the sake of the example we will use real Terraform modules for testing. First let’s make a compliant Terraform plan.
Now lets generate a Terraform plan and convert it to JSON then run conftest against and see if all of our checks pass.
Great, now let’s add some violations and try again.
Here we can see that conftest caught all of the violations that we added. The one test that passed is because we have not defined any firewall rules that allow ingress traffic to port 80; therefore, there are no rules that violate that policy. These were very rudimentary tests, and in practice testing is, and should be, much more comprehensive.
Integrating Into the Pipeline
The next, and final, step is to implement conftest into our pipeline. Lucky for us, there is a GitLab CI/CD implementation example in the conftest repository, so we can just piggy back off that and throw it into our .gitlab-ci.yml and tell it to only run on the master branch. There is obviously more configuration tweaking to be done here, but GitLab CI/CD is not the focus of this post, so we won’t dive into that.
Use Policy Enforcement for Your Use Case
At Praetorian, we needed a solution to ensure that our documented policies surrounding our Red Team infrastructure were being properly enforced during the development of the provisioning code. However, the basic concepts of policy enforcement can be applied to essentially any use case an organization may have. Using OPA, you can enforce policies to ensure that proper authorization controls are in place, ensure proper labeling of resources is applied across the entire tech stack, and enforce uniform design principles. And these are just a very small subset of the capabilities that OPA provides. A few things to consider when planning how you can use OPA in your development lifecycle are listed below.
Policies
What policies do you want to enforce? Documenting the policies that you want to enforce in your environment is crucial to the successful adoption of policy enforcement. One of the main goals of policy enforcement is to ensure uniformity across your environment, but that can’t be achieved if you don’t have uniformity across your policies; therefore, the policies that you plan on enforcing should be well documented before you begin implementing policy enforcement.
Data and Context
When creating your policies, you need to know what kind of data you are going to be processing. Do you want to ensure that a developer doesn’t go and delete a large portion of your environment? Then you are going to be working with infrastructure resources such as S3 buckets and compute instances, and you are going to be working with integers to count how many resources they are trying to delete, if any. Additionally, you need to decide how this data gets to OPA, and how OPA is returning the results of the validation checks. In our case, we are validating Terraform modules in our pull request checks so OPA gets the data from the PR, performs validation checks, then posts the results as a comment on the PR to facilitate reviews and auditing.
Policy Management
What access controls are going to be required regarding the modification and addition of policy definitions? What happens when a developer sees that OPA denied their PR and instead of fixing their code they go and change the policy to allow their changes to be merged? Strong authorization controls around CRUD operations for OPA policies need to be defined and implemented to protect against this type of behavior.
Conclusion
Policy enforcement can be a difficult problem to solve, and tools like OPA and conftest can be leveraged to automate some of the process while also speeding up the development process and reducing some of the churn that comes with reviewing pull requests. The example we showed only covered policy enforcement for part of the infrastructure, and we plan on extending our policy enforcement capabilities even further with tools like InSpec. Policy-as-code can be extremely powerful and can span all phases of the CI/CD pipeline from IDE integrated policy coverage in the Code phase, to runtime authorization checks in the Operate phase.
