Kubernetes Secrets: How to Create, Use, and Secure Them

What are Kubernetes secrets?

In Kubernetes, a secret is any type of sensitive data – such as login credentials, tokens, and certificates – that can be used for authentication purposes.

For example, if you want to configure a container in Kubernetes to download an image from a private, password-protected container registry, you could configure a secret that contains the access information the container will need to connect to the register. Likewise, you could use secrets in Kubernetes to configure SSH keys to connect to remote servers.

The concept of secrets is not unique to Kubernetes; secrets can exist in any software system or platform that stores sensitive information. But Kubernetes manages secrets in its own way, as we’ll learn below.

Types of Kubernetes secrets

Kubernetes supports a variety of different types of secrets, such as:

• Opaque A generic type of secret that can store any kind of user-defined data.
• SSH-auth Credentials for SSH authentication.
• TLS Secrets for securing a TLS client or server.
• Service account Tokens for managing authentication for service accounts (which are non-human accounts that can automate processes in Kubernetes).

Opaque is the most commonly used type of secret since it can support virtually any type of data. However, if you’re using secrets for a specific supported purpose (like SSH authentication), you’ll want to choose the secret type that Kubernetes defines for that purpose. This will reduce the amount of configuration effort necessary to use the secret.

How to use Kubernetes secrets

The main functionality that Kubernetes offers for secrets management is the ability to define secrets using kubectl (the primary CLI tool that admins use for working with Kubernetes). Below, we’ll walk through some of the basic processes of working with secrets in Kubernetes.

Create a secret

For example, the following command generates a secret that stores a username and password combination:

kubectl create secret generic reg-pass \
--from-literal=username=admin \
--from-literal=password='some-password'

Here, we’re generating the secret by feeding a plain-text username and a password to kubectl. 

The output should look similar to the following:

Once the secret is created, Kubernetes stores it by default in Etcd, a key-value store that houses various types of configuration data for Kubernetes.

Look up a secret

If you want to validate that the secret has been created, you can use a command like:

kubectl get secret reg-pass

This tells Kubernetes to look up the secret that we defined in the previous command.

Expect output like the following:

Decode a secret

When you look up a secret and present it in JSON format, Kubernetes will display the secrets as an encoded value by default for the secret, rather than showing the secret data itself. 

If you want to view your secret data in decoded form, first display the encoded secret values using a command like:

kubectl get secret reg-pass -o json

The output will be similar to:

You can view the secret keys (meaning the password and username) in clear text by using kubectl again and piping the results into a tool like base64 to decode it. For example, to decode the password, run:

kubectl get secret reg-pass -o jsonpath='{.data.password }' | base64 --decode && echo

The output will look like the following:

There are other ways to work with secrets in Kubernetes, such as creating them based on data stored in a file instead of entering credentials in plain text on the CLI. For more details, check out the Kubernetes secrets documentation.

Are Kubernetes secrets really secure?

The built-in features that Kubernetes offers to create and work with secrets are secure in the sense that they provide an alternative to storing secrets permanently in plain text. But on their own, they are far from ideal.

Without Kubernetes’s built-in security features, developers might be tempted to do things like define container secrets directly in the .yaml manifest that they use to configure an application deployment, or store SSH keys in plain text files directly on Kubernetes nodes. Those practices are insecure because anyone who can view the text files would have access to the secrets. By storing secrets in Etcd instead, Kubernetes’s secrets features help reduce the risk of unauthorized access to secrets.

That said, relying on Kubernetes alone to protect your secrets is not ideal, for multiple reasons:

• Kubernetes doesn’t automatically look for or alert you about insecure secrets that may be lurking inside configuration files or manifests. It expects you to manage that risk on your own.
• By default, anyone with admin-level access to your Kubernetes cluster can decode secrets.
• In some cases, anyone who is able to log in as root on Kubernetes control-plane nodes could potentially access secrets by viewing Etcd through the node (as opposed to through Kubernetes itself).
• Secrets data may also potentially leak through shell histories, which are typically recorded in files like the .bash_history file that most Linux distributions generate automatically for each user account.

In short, while managing secrets using kubectl is better than storing them in plain text, it still leaves the door open to potential attacks.

Kubernetes secrets best practices

To mitigate the risk of insecure secrets management in Kubernetes, admins should adopt strategies for keeping Kubernetes secrets safe from unauthorized access. Best practices include the following.

1 – Scan for Kubernetes secrets

Since Kubernetes does not automatically check for insecure secrets, admins should implement a comprehensive secret scanning solution for Kubernetes. This includes scanning all configuration files, container images, application source code, and any other relevant resources where unencrypted, insecure secrets might exist.

2 – Enable Etcd encryption

By default, Kubernetes stores secrets in Etcd in unencrypted form. To change that, you must explicitly enable Etcd encryption for data at rest. This ensures that someone who manages to get access to Etcd (such as by logging in as root on the node that hosts Etcd) won’t be able to view the secrets unless they also have a decryption key.  

3 – Use RBAC to protect secrets

Using the Role-Based Access Control (RBAC) framework that is built into Kubernetes, admins can restrict which users are able to view and modify secrets by granting only certain users the ability to perform tasks like get or list secrets. (Get and list are specific types of actions that can be enabled or disabled using Kubernetes RBAC.)

Doing so helps prevent anyone with access to kubectl on a cluster from being able to access secrets at will.

4 – Consider an external secrets manager

Instead of relying on Etcd as your default secrets management solution, you can optionally configure an external secrets manager. A secrets manager is a tool whose sole purpose is to store secrets in a secure way.

Using an external secrets manager is valuable for two main reasons. First, it may provide more granular access control configurations and storage protections than you’d get from Etcd. Second, you can store secrets for other types of systems, not just Kubernetes, in a secrets manager, allowing you to centralize your approach to managing secrets across all of your systems.

Secrets management with Aqua

No matter which secrets you need to manage or where they exist, Aqua helps keep them safe. By comprehensively scanning all of the resources that exist in both development and production environments, Aqua alerts teams to secrets that are not properly secure – such as access credentials that are hard-coded into configuration files or unencrypted secrets within databases.