Privilege Escalation from Node/Proxy Rights in Kubernetes RBAC

One of the side effects of Kubernetes’ rich API and extensive functionality is that sometimes there are security implications to granting users permissions. Security architects should be aware of these side effects when designing platforms that use Kubernetes. In recent research with Iain Smart of NCC Group, we looked at how granting rights to node/proxy resources in Kubernetes could allow for audit logs and other security controls to be bypassed.

What’s the Kubernetes API server proxy?

The first questions to consider are, “What is the node proxy, and how does it work?” A less well-known feature of the Kubernetes API is that it can allow users to proxy requests through the API server to workloads running in the cluster. This is useful for troubleshooting and monitoring because it doesn’t require users to have direct network connectivity to the workload. Instead, they can access the workload via the API server.

In general, this access is restricted to pod and service IP addresses, and it shouldn’t be possible to use the API server as a general proxy. However, as Kinvolk noted in 2019, it’s possible to trick the API server into allowing access to some arbitrary external IP addresses.

In addition, Kubernetes has a feature that allows users to access services running on the cluster nodes via the API server. In general, these requests are made without authentication. However, one thing we noticed when testing was that access to the Kubelet API was made with credentials. This means that users with node/proxy rights can essentially do anything the Kubelet API allows for, including executing commands in every pod on the node.

Talking directly to the Kubelet API

After doing some more reading in the documentation and asking members of the ever-helpful Kubernetes community, we realized there was another way this could be (mis)used. It turns out that the node/proxy role-based access control (RBAC) right is also used by the Kubelet API directly, so users with those rights can contact the Kubelet directly and bypass the API server.

From a security architecture perspective, this could have some serious consequences:

• The Kubelet API doesn’t support Kubernetes auditing, so it wouldn’t be possible to properly audit actions taken using this API.
• Access to the Kubelet API bypasses Kubernetes admission control, so a user with those rights might be able to take actions that would otherwise be prevented.

Here’s a real-world demonstration to show how this works. First, we’ll create a cluster role that only has node/proxy rights:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2022-01-07T11:48:17Z"
  name: nodeproxy
  uid: 01b56e4a-6109-43d9-92e8-14013c0054ed
rules:
  - apiGroups:
  - ""
  resources:
  - nodes/proxy
  verbs:
  - get
  - create

Then we’ll create a service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-01-07T11:51:27Z"
  name: nodeproxy
  namespace: default
  uid: a67ae864-a3da-402a-bdff-c181c86dd9b1

Next, we’ll bind the cluster role to that account:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2022-01-07T11:50:58Z"
  name: nodeproxybinding
  uid: e368c3ac-945f-41d2-a6b1-7caa2eaf8038
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nodeproxy
subjects:
  - kind: ServiceAccount
  name: nodeproxy
  namespace: default

With those pieces in place, we can get the service account token and use that to talk directly to the Kubelet API. In the video below, we first retrieve the secret and then access the Kubelet API/pods endpoint to show that we can see all the pod data — using the excellent jless to make it a bit more readable. Then, by choosing one of the pods on the node, we can execute a whoami command showing we can execute commands as root.

Mitigations and conclusions

Knowing that this kind of escalation is possible, what are some of the security good practices that you can use to reduce the risks?

• Be very careful about who is given node/proxy rights, either explicitly or as part of a wildcard RBAC grant.
• If this is a risk for your cluster, consider firewalling access to the Kubelet service. While users could still go via the node/proxy, that access would show up in Kubernetes audit logs.
• While exact Kubelet API access isn’t visible in audit logs, there will be subjectaccessrequest entries where the Kubelet confirms the users’ permissions. Learn how to detect direct access to the Kubelet.

This post highlights some of the complexity associated with Kubernetes RBAC and demonstrates why it’s important to take a least-privilege approach to granting users and services rights to use it. While Kubernetes is a very flexible and rich system, those attributes do have consequences from a security standpoint.