How to encrypt etcd and use secrets in OpenShift

A secret in OpenShift is any critical resource or object type that provides a mechanism to hold sensitive information, such as a password, sensitive configuration file, Transport Layer Security (TLS) certificate, Secure Shell (SSH) credential, or OAuth token.

A secret is used in internal connections between different resources. For example, applications and services can share secret data within a namespace.

Types of secrets

There are different types of secrets, and OpenShift validates that data stored in a secret conforms to the type of secret in use. Secret types include:

• kubernetes.io/service-account-token: ServiceAccount token
• kubernetes.io/basic-auth: Credentials for basic authentication
• kubernetes.io/ssh-auth: Credentials for SSH authentication
• kubernetes.io/tls: Data for a TLS client or server
• Opaque: Arbitrary user-defined data

Most secret types you create will likely be of the opaque type. The opaque secret type is used in cases where you don't want validation, meaning that the secret does not claim to conform to any conventions for key names or values.

OpenShift uses secrets for two primary reasons:

• To store credentials used by pods in a microservices architecture. A secret is useful when you want to connect two pods.
• To store TLS certificates and keys. Secrets are crucial in defining secure routes in OpenShift networking. Developers can mount a secret as a volume and create a pass-through route to an application. It's important to note that a TLS secret stores the certificate as tls.crt and the certificate key as tls.key, so the application using them must use the same naming convention.

[ Check out this guide to boosting hybrid cloud security and protecting your business. ]

Create a secret object on the controller node

You must specify the secret type (for example, generic, TLS, or docker-registry) when you create it.

Use the --from-literal flag to create a generic secret:

$ oc create secret generic topsecret \
--from-literal \
user=vcirrus-consulting \
--from-literal \
password=topsecretpassword \
secret/topsecret created

The secret looks like this after you create it:

$ oc get secret topsecret -o yaml                                                                                                                                                                                                             
---
apiVersion: v1
data:
  password: dG9wc2VjcmV0cGFzc3dvcmQ=
  user: dmNpcnJ1cy1jb25zdWx0aW5n
kind: Secret
...

It's important to know that a secret is not encrypted. It's just a ConfigMap encoded with Base64, so anyone can decode it into plain text:

$ echo dmNpcnJ1cy1jb25zdWx0aW5n | base64 --decode                                                                                                                                                                                                       
vcirrus-consulting
$ echo dG9wc2VjcmV0cGFzc3dvcmQ= | base64 --decode                                                                                                                                                                                                 
topsecretpassword

To protect data in a secret, you can encrypt etcd. Etcd is a Kubernetes data store that contains cluster information in key-value pairs.

[ Learn Kubernetes usage basics in this cheat sheet. ]

Create a generic secret from a file

Use the --from-file option to create a generic secret using the contents of a file:

$ oc create secret generic top-ssh-secret \
--from-file secure_id_ecdsa \
--from-file secure_id_ecdsa.pub
secret/top-ssh-secret created

The secure_id_ecdsa and secure_id_ecdsa.pub files contain the private and public SSH keys, respectively.

Expose secrets to pods

Skip to the bottom of list

Image

Download now

To make a secret available to a pod, you can refer to a secret as a variable or as a file in the pod's configuration. The most convenient way to update a pod with information in a secret is by using the oc set env command. This writes the environment variables obtained from a secret to a pod or deployment.

1. Create the secret

First, create a generic secret with the variables you want to be used within the pod:

$ oc create secret generic \
mysql-secret --from-literal user=janedoe \
--from-literal password=mysqlpassword \
--from-literal database=mysqlsecretdb \
--from-literal hostname=janedoe-mysql \
--from-literal root_password=janedoe-password
secret/mysql-secret created

2. Use jq to view the secret

Use the jq command to view the secret you've created as JSON:

$ oc get secret mysql-secret -o json | jq                                                                                                                                                                                                         
{
  "apiVersion": "v1",
  "data": {
    "database": "bXlzcWxzZWNyZXRkYg==",
    "hostname": "amFuZWRvZS1teXNxbA==",
    "password": "bXlzcWxwYXNzd29yZA==",
    "root_password": "amFuZWRvZS1wYXNzd29yZA==",
    "user": "amFuZWRvZQ=="
  },
  "kind": "Secret",
  "metadata": {
    "creationTimestamp": "2022-08-22T11:59:41Z",
    "name": "mysql-secret",
    "namespace": "default",
    "resourceVersion": "163733",
    "uid": "b6f43859-ad60-49dd-9be8-3d23f68cd5de"
  },
  "type": "Opaque"
}

[ Want to test your sysadmin skills? Take a skills assessment today. ]

3. Start a MySQL pod

Now start a MySQL or MariaDB pod with the name janedoe-mysql:

$ oc new-app --name janedoe-mysql --image bitnami/mysql                                                                                                                                                                                    
[…]
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose service/janedoe-mysql'
    Run 'oc status' to view your app.

4. Watch the pod fail

Watch the status of the pod. Note that the pod fails with the CrashLoopBackOff error:

$ oc get pods -w                                                                                                                                                                                                                                  
NAME                    READY STATUS             RESTARTS    AGE
janedoe-mysql-d5dd---   0/1   ContainerCreating  0           8s
janedoe-mysql-d5dd---   0/1   Error              0           2m44s
janedoe-mysql-d5dd---   0/1   Error              1 (2s ago)  2m45s
janedoe-mysql-d5dd---   0/1   CrashLoopBackOff   1 (2s ago)  2m46s

5. Investigate

The pod failed, and you can learn why by viewing the output of oc logs. In this case, the Bitnami MySQL pod is failing because an environment variable isn't set. If you're running a MariaDB pod, oc logs hints at which variable you need to set:

$ oc logs janedoe-mysql-d5ddd6877-nqntj                                                                                                                                                                                                       
mysql 12:07:49.52
mysql 12:07:49.52 Welcome to the Bitnami mysql container
mysql 12:07:49.53 Subscribe to project updates by watching https://github.com/bitnami/containers
mysql 12:07:49.53 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mysql 12:07:49.54
mysql 12:07:49.54 INFO  ==> ** Starting MySQL setup **
mysql 12:07:49.59 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql 12:07:49.60 ERROR ==> The MYSQL_ROOT_PASSWORD environment variable is empty or not set. Set the environment variable ALLOW_EMPTY_PASSWORD=yes to allow the container to be started with blank passwords. This is recommended only for development.

6. Create a secret

To satisfy the pod's requirement, you must set a root_password variable in a secret. Refer to Step 1 above to create a new secret containing the root_password variable, but include the prefix MYSQL_.

When you include the ROOT_PASSWORD variable and exclude the MYSQL_ prefix, you make your secret flexible enough to work in multiple environments. In this case, the secret could adapt well, for example, to either a MySQL or a MariaDB environment.

Using a prefix is convenient because it allows you to define a variable and use it in a specific way in your configuration. This example defined root_password, which is not significant in a MySQL environment. MySQL requires a variable with the name MYSQL_ROOT_PASSWORD.

7. Update the failing pod with a secret

The Bitnami MySQL image requires the MYSQL_ROOT_PASSWORD variable. The command below reads variables from your secret, and it can detect the correct variable because one of them is root_password combined with the MYSQL_ prefix:

$ oc set env deployment/janedoe-mysql \
--from secret/mysql-secret --prefix MYSQL_                                                                                                                                                                  
deployment.apps/janedoe-mysql updated

8. Watch the pod again and confirm it's in a running state

Now that you've updated your secret, watch the status of the test MySQL pod and verify that it's running:

$ oc get pods -w                                                                                                                                                                                                                                  
NAME                    READY STATUS             RESTARTS   AGE
janedoe-mysql-7c56---   1/1   Running            0          11s

[ Getting started with containers? Check out this no-cost course on deploying containerized applications. ]

9. Explore the pod environment

Verify the environment variables provided by your secret by listing all of them with the env command:

$ oc exec -it \
janedoe-mysql-7c567d5564-d99pp – env
[...]
MYSQL_HOSTNAME=janedoe-mysql
MYSQL_PASSWORD=mysqlpassword
MYSQL_ROOT_PASSWORD=janedoe-password
MYSQL_USER=janedoe
MYSQL_DATABASE=mysqlsecretdb
KUBERNETES_PORT_443_TCP=tcp://10.217.4.1:443
JANEDOE_MYSQL_PORT_3306_TCP_ADDR=10.217.5.245
KUBERNETES_PORT=tcp://10.217.4.1:443
JANEDOE_MYSQL_PORT_3306_TCP_PORT=3306
KUBERNETES_SERVICE_PORT_HTTPS=443
[...]

Encapsulate your data

Despite its name, a secret isn't secret until you encrypt etcd, but it's still a useful way to pass vital information into pods.

Now that you understand how to use a secret, try learning how to mount a secret as a volume (hint: try the oc set volume command).

Keep in mind that you should keep your secrets private on production servers. Read Encrypting etcd data in the OpenShift docs for more information.