Basic security principles for containers and container runtimes

There have been recent discussions around the general security of containers and container runtimes like Podman. None of the discussions resulted in the identification of a vulnerability or exploit by their definitions, but the talks did elevate the importance of basic security principles that apply to containers, and just about everything else we do with technology.

Refining the discussions to their core, three core themes emerged:

• How to handle user privileges.
• The podman container runlabel command.
• Remote Podman API execution.

In the following sections, we will review each of these themes to remind ourselves of the basic security principles that need to be followed.

User privileges

In the single-node container world (non-orchestrated with Kubernetes and the like), many people have become lax about running containers as the root user regardless of the inherent danger that practice brings. If a container is run as root, then it has full root privileges. Consider the following exercise, where we look at the first four lines of /etc/ssh/sshd_config, which is read-protected against unprivileged users.

Reading the file as an unprivileged user should fail:

$ head -n4 /etc/ssh/sshd_config
head: cannot open '/etc/ssh/sshd_config' for reading: Permission denied

Reading the file as root via sudo should succeed:

$ sudo head -n4 /etc/ssh/sshd_config
#    $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.

Reading inside the context of container run by root is no different than being root on the host itself:

$ sudo podman run -it --rm -v /etc:/host_etc alpine head -n4 /host_etc/ssh/sshd_config
#    $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.

One of Podman’s key features is the ability to run containers as an unprivileged user, often referred to as rootless. Running rootless containers should be the default for most people. Using these eliminates the unnecessary risk that accompanies handing out sudo or root access. More importantly, a rootless container is somewhat confined by the privileges of its user.

Revisiting our example, we should not be able to see the sshd_config file as a regular user:

$ podman run -it --rm -v /etc:/host_etc alpine /bin/sh
/ # head -n4 /host_etc/ssh/sshd_config
head: /host_etc/ssh/sshd_config: Permission denied
/ # whoami
root
/ #

However, this exercise shows a wrinkle. Notice that we are unable to see the file as expected, but also notice that we are considered to be root inside the container. Podman takes advantage of user namespaces, so root within the container is mapped to a non-root UID on the host. This setup allows Podman to safely install packages and run services from within the container, and not impact the host.

There are cases where running a container as root makes sense. One of the reasons might involve containers needing access to specific mounts, devices on the host, or specific filesystems. Another example involves containers that need to listen on ports less than 1024 on the host network. Revisualizing your container setup and architecture can often unburden many of these requirements. Be sure to explore your options before running things as root.

Service-oriented containers that leverage systemd internally can run as rootless with Podman. Moreover, users can use the host’s systemd services to automatically start and stop the containers.

The podman container runlabel command

Podman has the convenience-oriented subcommand podman container runlabel. This feature allows you to define commands that you want to run as metadata on the image.

The main use case for runlabel is to simplify a lengthy command for running an image. For example, consider the following LABEL that could define with the key of RUN:

RUN=`podman run -dt -p 80:8001 -p 7640:3280 -v /somedir:/foobar IMAGE sh myscript.sh`

Instead of users having to type that lengthy command to run this image, they can simply issue:

$ podman container runlabel RUN <imagename>

Podman then executes the full command as defined by the label on the image. People have pointed out, however, that runlabel could be used for malicious intent. Consider a runlabel like this one:

RUN=`rm -fa /`

Running that command blindly as root could be catastrophic. So, like many convenience functions in technology, Podman’s runlabel needs to be run by an informed user. In a lot of ways, runlabels are similar to scripts executed out of an RPM package. They can execute similar commands that cause similar issues on your system.

Container images should be treated the same way as any software you install on your system. You should only get software from trusted entities.

One way to mitigate concern around the use of a runlabel is to use the --display flag before executing the runlabel. The display flag will return the command that is associated with the image label without actually executing the command.

Here is an example of a mariadb container I use locally:

$ podman container runlabel --display RUN db
command: podman run --name demodb -P -e MYSQL_ROOT_PASSWORD=x -dt localhost/db:latest

The discussion around running content in images is much broader than Podman’s runlabel. The question narrows itself down to: How does one confidently run any image?

Consider the following tips as a series of good first steps:

• Use a reputable registry: Developing trust starts with the provenance of the image. Pull images by their fully-qualified names instead of a shortened name that might be pulled from a different registry.

• Run signed images: Limit the images you run to images that have been signed by the registry and verified upon being pulled to your local system.

• Use SELinux or AppArmor: Ensure that system security tools are running and preventing malicious container escapes.

• Run as rootless: When possible, run images as an unprivileged user.

• Inspect your image: Once the image is local, inspect it for ports, mounts, and other access it wants that you may not approve of.

• Use runlabel --display: Use the display flag before running labels on your image to see what will be executed.

Remote Podman API execution

Any time a set of API endpoints is exposed remotely (and to some degree, locally) an element of risk is introduced. In the case of Podman, being prudent with common security practices helps mitigate any risk.

The basic implementation of Podman’s remote API is varlink over a socket. This socket can be managed by a service like systemd, or it can be manually created. In Fedora, the Podman package includes a systemd service and socket file to use with Podman and varlink. That socket file is configured for a Unix socket, thereby restricting it to local use only.

Your first consideration for security, like the previous sections, is to run your socket as a non-root user—particularly if your containers have no requirements to run as a privileged user. You could set up a systemd unit file to create a Podman varlink socket, or you could just run a Podman command:

$ podman varlink --timeout=0 unix:/run/user/$(id -u)/podman/io.podman

When creating a socket like this, Podman will ensure that proper permissions are set on the socket. This fact is important because if the socket permissions are set incorrectly, it might be possible for a different user to use the socket to run commands. Moreover, do not use a TCP socket in the varlink URI (tcp vs. unix), as doing so allows unauthenticated and unencrypted use of Podman.

The bottom line is that access to the socket is the equivalent of running the Podman CLI as the socket’s user. With access to a root varlink socket, it is trivial to create a privileged container and mount in the host filesystem. Because of this issue, access to a root varlink socket is already root access to the system. The same is true of the Docker socket. By default, you have to be root in the first place to talk to a Podman varlink socket running as root, so there is no escalation of privileges here.

When doing true remote (client-to-server) connections, use the varlink bridge method for connecting. This method has a number of advantages. Firstly, it wraps all communications in SSH, thereby solving both the authentication and encryption problems. And secondly, no running socket is required for Podman on the host. This method creates a temporary socket specifically for that session.

If you wanted to talk to Podman varlink running as root, then you would have needed to log in as root over the SSH session.

Conclusions

When running containers, user privileges really do matter. With the exception of a few narrow use cases, containers should be run as unprivileged users. Before running container images, you should understand its origin and what it will do when run.

Be certain to complement your diligence around the security implications of running containers with tools like SELinux. And finally, when using the Podman remote API, use the varlink bridge to ensure both authentication and encryption when interacting with a remote system and its containers.

[New to containers? Download the Containers Primer and learn the basics of Linux containers.]