The author presented this paper and corresponding tool at Black Hat: Arsenal 2023 on August 10, 2023. For a more general overview of Konstellation and its capabilities vis a vis Kubernetes RBAC, please see our earlier companion post.
Kubernetes Role-Based Access Control (RBAC) is a mechanism for controlling access to resources in a Kubernetes cluster. It allows administrators to specify who has access to what resources, and at what level of access. RBAC uses roles, which are sets of permissions, to determine what a service account, user, or group is allowed to do. Permissions in RBAC are additive, meaning that if a user has multiple roles, they will have the union of the permissions of all their roles.
Given the lack of “deny” statements in RBAC, if a user does not have a specific permission, they will not be able to perform the corresponding action. This means careful management of users’ roles and permissions is essential for ensuring that they only have access to the resources they need. One RBAC use-case is to enforce fine-grained access control on a Kubernetes cluster, making it an essential tool for securing and managing access in large, complex environments.
Enter: Konstellation
Praetorian’s internal research has focused on using graph theory to find security weaknesses in existing Kubernetes RBAC deployments, and ultimately led us to develop Konstellation. The approach consisted of explicitly mapping all RBAC permissions to the corresponding resources and analyzing the resulting graph for paths that could lead to elevations of privilege or compromises of sensitive resources. This is challenging to do manually at scale, so we developed Konstellation to facilitate this research by facilitating the enumeration of Kubernetes and ingestion of the resulting data into a Neo4j database. We are releasing the tool in conjunction with the underlying research and all cypher query examples the Konstellation schema played a role in developing.
This blog post will cover several of the most common privilege escalation paths including service account token abuse, RBAC policy manipulation, risky privileges, and over privileged roles.
A Note on Data Gathering and Ingestion
The prerequisites for the analysis herein are data gathering and insertion into a Neo4j database. Konstellation automates both of these tasks, which the Konstellation repository documents. The remainder of this blog post assumes the Konstellation user has completed both tasks successfully.
Privileged Roles
The most privileged role in Kubernetes is cluster-admin, which has full control over the cluster. Role bindings to this role should be minimal and tightly controlled.
The following cypher query can find subjects bound to the cluster-admin role:
MATCH (subject)-[r:ROLE_BINDING {kind: "ClusterRoleBinding"}]->(cr:ClusterRole {name: "cluster-admin"}) RETURN subject.name
Konstellation operators can use following cypher query to identify additional powerful roles, such as admin and edit:
MATCH (subject)-[r:ROLE_BINDING]->(role:ClusterRole) WHERE role.name IN ["cluster-admin", "admin", "edit"] RETURN subject.name, role.name
The following command can then identify these subjects with privileged bindings:
./konstellation.py k8s query -n "Subject bound to privileged role"
Privilege Escalation Paths
Service Account Tokens
Kubernetes versions prior to v.1.22 automatically create Service Account Tokens, which are long-term credentials. While no longer the default mechanism due to more recent versions’ use of the TokenRequest subresource, many still exist in Kubernetes clusters. Operators with new versions also can still create Service Account Tokens using the kubernetes.io/service-account-token Secret type.
Therefore, two paths exist to compromising service accounts directly by either stealing the long-lived service account token, or creating a token through the serviceaccounts/token subresource.
Stealing the Service Account Token
Service Account Tokens are simply namespaced Secrets. Therefore, the security of the service account depends on what resources can read the service account token secret. A common misconfiguration is to allow the GET or LIST verbs on secrets, thereby permitting the grantee to obtain the secret and, in the case of a service account token, compromise the corresponding service account.
Figure 1 summarizes the graph data structure for representing the relationship between the service account, service account token, and role.
Figure 1: The relationship between service account, service account token, and role.
The following cypher query is an example for finding principals that can obtain the service account token of another account, excluding principals in the kube-system namespace:
MATCH (sa:ServiceAccount)-[r1:SERVICE_ACCOUNT_TOKEN]->(s:Secret)<-[r2:FULL_CONTROL|GET|LIST]-(role)<-[:ROLE_BINDING]-(x) WHERE x <> sa and x.namespace <> 'kube-system' RETURN sa.name as vulnSA, type(r1) as type, s.name as vulnSaSecret, type(r2) as verb, x.name as subject, x.kind as subjectKind, role.name as subjectRole
Konstellation operators can find resources that can read secrets for service accounts using the following command:
./konstellation.py k8s query -n "Resource can read secret for Service Account"
Abusing the TokenRequest API
The implications of abusing the TokenRequest API are the same as stealing the service account token–service account compromise. However, the method and RBAC permissions for this are different as the TokenRequest API is a subresource of the ServiceAccount resource. Moving to the TokenRequest API is likely to reduce the opportunity to perform privilege escalation through service account token theft as roles must have explicit serviceaccounts/token subresource permission.
Konstellation implements subresources differently, graphing them as relationships between the parent resource and the role with the subresource privilege. The relationship name format is <verb>_<subresource>; therefore a role that can create a service account token for a given service account will have the CREATE_TOKEN relationship to the service account (see figure 2).
Figure 2: Konstellation conceives of subresource privilege as a relationship between the parent resource and the role with subresource privilege.
The Konstellation query to identify roles with the verb on the token subresource for a ServiceAccount is as follows:
MATCH (x)-[:ROLE_BINDING]->(role)-[r:CREATE_TOKEN]->(s:ServiceAccount) WHERE not role.name IN ["system:kube-controller-manager", "system:node"] RETURN x.name, x.kind, role.name, collect(s.name)
Konstellation operations can run this query using the following command:
./konstellation.py k8s query -n "Role can create service account tokens"
Risky Privileges
Impersonate
The IMPERSONATE verb allows a subject to act as another user or service account. If the impersonated identity has greater privileges than the subject, this will elevate the privileges of the subject. While granting this privilege may be intentional, it may precipitate unintended privilege escalation scenarios if the impersonated identity has wide-ranging permissions or administrators grant the IMPERSONATE privilege too broadly.
The following cypher query will identify subjects not bound to the privileged admin, edit, and cluster-admin roles with the IMPERSONATE privilege:
MATCH (subject)-[r1:ROLE_BINDING]->(role)-[r2:`IMPERSONATE`]->(y) WHERE NOT role.name IN ["admin", "edit", "cluster-admin"] RETURN subject.name, role.name, y.name
Konstellation users can run this query using the following command:
./konstellation.py k8s query -n "Role has ESCALATE privilege"
Escalate
Kubernetes’ normal operating behavior is to prevent the creation or modification of Roles or ClusterRoles unless the calling user already has permission they are attempting to set. The ESCALATE verb bypasses this restriction. Therefore, administrators should tightly control and monitor roles containing this privilege as they may offer a direct path to privilege escalation.
The following cypher query will identify principals bound to a role with the Escalate privilege:
MATCH (principal)-[r1:ROLE_BINDING]->(role)-[r2:ESCALATE]->(y) WHERE NOT role.name IN ["admin", "edit", "cluster-admin"] RETURN principal.name, role.name, y.name
Konstellation operators can run this query using the following command:
./konstellation.py k8s query -n "Role has ESCALATE privilege"
node/proxy and pod/proxy
The node/proxy and pod/proxy privileges can allow for privilege escalation as the subject of these privileges could run commands on nodes or pods scoped in the role.
The proxy resource is a subresource permission of the parent resources, and Konstellation uses the <VERB>_<SUBRESOURCE> relationship convention to represent this privilege. The following cypher query will identify these privileges for non-administrative roles:
MATCH (principal)-[:ROLE_BINDING]->(role)-[r:CREATE_PROXY|GET_PROXY]->(y) WHERE y.kind IN ["pod", "node"] AND NOT role.name IN ["admin", "edit", "cluster-admin", "system:aggregate-to-edit"] RETURN principal.name, principal.kind, role.name, type(r) as verb, collect(y.name)
Konstellation can run this query using the following command:
./konstellation.py k8s query -n "non-privileged role proxy pod or node"
Permissions granted to system:authenticated
The system:authenticated special group in Kubernetes represents any authenticated principal. Assigning privileges to this group grants those privileges to all authenticated users.
The following cypher query will identify role bindings for the system:authenticated group:
MATCH (g:Group {name: 'system:authenticated'})-[r:ROLE_BINDING]->(role) RETURN g.name, type(r), role.name
Konstellation operators can run the role bindings for the system:authenticated group using the following command:
./konstellation.py k8s query -n "Role bindings for the system:authenticated group"
Permissions granted to system:unauthenticated
system:unauthenticated is a special group that identifies unauthenticated requests to the Kubernetes API. Granting privileges to this group beyond the default system:public-info-viewer role could lead to information disclosure or privilege escalation for unauthenticated attackers.
The following cypher query can query the role bindings for the system:unauthenticated group.
MATCH (g:Group {name: 'system:unauthenticated'})-[r:ROLE_BINDING]->(role) RETURN g.name, type(r), role.name
Konstellation users can run the role bindings for the system:unauthenticated group using the following command.
./konstellation.py k8s query -n "Role bindings for the system:unauthenticated group"
Workload Attack Paths
Most of these document and privilege escalation scenarios focus on gaining complete administrative control over the cluster. However, administrative control is not the only potential goal during an attack. Identifying access paths to sensitive resources also can be the focus of an attack. Many organizations process sensitive data such as PII, PHI, and PCI in the applications deployed on their Kubernetes clusters. Targeting these deployments, pods, configmaps, secrets, etc., can yield an organization’s crown jewels without requiring privileged access.
Intra-Namespace Access
Konstellation captures the namespace context for all resources and sets the namespace property to that value. Konstellation operators can use this property to filter attack paths to target specific workloads.
Cross-Namespace Access
Namespaces in Kubernetes enforce resource isolation, environment segmentation, and access control. For various reasons, an organization may need to provide access between namespaces. The security implications of granting cross-namespace access should be evaluated, though. The following cypher query identifies subjects bound to roles in another namespace to identify cross-namespace access:
MATCH (x)-[r {kind: "RoleBinding"}]->(y) WHERE x.namespace <> y.namespace RETURN x.name, x.namespace, y.name, y.namespace
Konstellation operators can use the following Konstellation query to identify cross-namespace access:
./konstellation.py k8s query -n "Role has cross-namespace access"
Chaining Paths
Many of the queries we have highlighted in this post are starting points for deeper investigation. Konstellation users can chain each potential privilege escalation path with one or more others to achieve greater access and fulfill attacker goals.
From “Reader” to Admin
A common misconfiguration Praetorian identifies during engagements is overly broad GET and LIST verbs on Secrets resources, which can lead to service account token theft. In the example below, we use this misconfiguration as the starting point and combine it with the node/proxy risky permission to search for full privilege escalation paths.
We first use the “Resource can read secret for Service Account” Konstellation query, then use the vulnerable service accounts as the starting point for the node/proxy query. The final query below combines the two attack paths to identify a direct privilege escalation path from our “security-reader” to control our nodes.
MATCH (y)<-[r5:CREATE_PROXY|GET_PROXY]-(role2)<-[r4:ROLE_BINDING]-(sa:ServiceAccount)-[r1:SERVICE_ACCOUNT_TOKEN]->(s:Secret)<-[r2:FULL_CONTROL|GET|LIST]-(role)<-[r3:ROLE_BINDING]-(x) WHERE x <> sa and x.namespace <> 'kube-system' AND y.kind in ["pod", "node"] and not role.name in ["admin", "edit", "cluster-admin", "system:aggregate-to-edit"] RETURN *
Figure 3 illustrates how we can visualize this attack path in the Neo4j browser, seeing that our “security-reader” can gain control over the nodes by stealing the service account token for the node-proxy service account bound to the node-proxy role with wildcard node/proxy privileges.
Figure 3: Neo4j visualization of the “Reader to admin” attack path.
Generalizing the Paths
The previous attack path combined one RBAC weakness with a specific privilege to identify the privilege escalation path. While we could develop individual queries for each permutation of secret theft + privilege, that would be overwhelming to create and maintain. We can instead create a more generic query that will yield all privileges gained through the secret theft.
We can first match principals that can read the service account token for a different service account, as in the code below. Those service accounts have privileges over other nodes.
MATCH p=(target)<-[r5]-(role2)<-[r4:ROLE_BINDING]-(sa:ServiceAccount)-[r1:SERVICE_ACCOUNT_TOKEN]->(s:Secret)<-[r2:FULL_CONTROL|GET|LIST]-(stealerRole)<-[r3:ROLE_BINDING]-(stealer) WHERE stealer <> sa and stealer.namespace <> 'kube-system'
Next, we turn the vulnerable service account’s relationship types (the RBAC verbs) over the target node into a list using the type and collect functions, as follows:
WITH *, collect(DISTINCT type(r5)) as stolen
We can gather existing relationships between our stealer node and the target using an OPTIONAL MATCH…
OPTIONAL MATCH (stealer)-[:ROLE_BINDING]->(role)-[r6]->(target)
…and use type and collect to create a list of privileges between our stealer and target nodes.
WITH *, collect(DISTINCT type(r6)) as orig
Finally, using list comprehension, we can create a list of “gained” privileges by comparing the “stolen” relationship types (RBAC verbs) to the original relationship types.
UNWIND [priv in stolen WHERE not priv in orig] as gained
Following is the entire query for this analysis:
MATCH p=(target)<-[r5]-(role2)<-[r4:ROLE_BINDING]-(sa:ServiceAccount)-[r1:SERVICE_ACCOUNT_TOKEN]->(s:Secret)<-[r2:FULL_CONTROL|GET|LIST]-(stealerRole)<-[r3:ROLE_BINDING]-(stealer) WHERE stealer <> sa and stealer.namespace <> 'kube-system' WITH *, collect(DISTINCT type(r5)) as stolen OPTIONAL MATCH (stealer)-[:ROLE_BINDING]->(role)-[r6]->(target) WITH *, collect(DISTINCT type(r6)) as orig UNWIND [priv in stolen WHERE not priv in orig] as gained RETURN stealer.name, stealerRole.name as stealerRole, sa.name as vulnSA, s.name as vulnSAT, COLLECT(DISTINCT gained) as gainedPrivs
Konstellation users can accomplish this analysis using the following command:
./konstellation.py k8s query -n "Service account token privilege escalation"
Recommendations
Role-based access controls (RBAC) can provide solid resource-level workload protection if administrators configure them correctly. Praetorian recommends following the Principle of Least Privilege when implementing RBAC in Kubernetes. Adhering to the principle minimizes the risk of unauthorized access to sensitive resources and reduces the likelihood of privilege escalation scenarios. The following guidance mitigates many issues Praetorian has identified within client environments:
- Limit the number of subjects with administrative access
- Block secrets GET * with pod security admission
- Avoid the use of “risky” privileges where possible and tightly scope those privileges when they are necessary
- Audit all deployed RBAC for privilege escalation opportunities
- Limit RBAC manipulation privileges to administrative roles
Conclusion
While simple on its surface, Kubernetes RBAC can be difficult to implement as least privileged in large-scale environments. Minor issues in RBAC configuration can significantly affect the security and integrity of the cluster and workloads. Graph theory provides a robust set of tools to map and understand the complex relationships involved with Kubernetes RBAC. Praetorian’s Konstellation tool can simplify the collection and analysis of Kubernetes RBAC for potential security issues.
A Word of Thanks from the Author: I’d like to thank my fellow Praetorians for their support and feedback during the development and research of Konstellation; especially Joseph Cooper, Emmaline Eble, Tim Gonda, Connor Holm, and Tanishq Rupaal.
