New features for running containers on NFS with rootless Podman
Back in November, we wrote Rootless Podman on NFS, which explained why you cannot use the default setup of Podman when running with an NFS home directory. Since then, we have made it easier with newer versions of Podman and have discovered some issues with users setting up these environments.
Currently, rootless Podman (by default) stores its container image content in the user’s home directory. The blog linked above explains that Podman is blocked from using multiple UIDs on the NFS server side since the NFS server does not understand user namespace. The blog recommends that the user set up a different directory for storing content. Users need to edit the storage.conf
file to make that happen.
About rootless_storage_path
We have added a new option field to storage.conf
named rootless_storage_path
to make it easier to use rootless Podman on an NFS home directory for different users.
$ man containers-storage.conf
rootless_storage_path="$HOME/.local/share/containers/storage"
Storage path for rootless users. By default, the graphroot for rootless users is set to $XDG_DATA_HOME/containers/storage, if XDG_DATA_HOME is set. Otherwise, $HOME/.local/share/containers/storage is used. This field can be used if administrators need to change the storage location for all users.
The rootless storage path supports three substations:
* `$HOME` => Replaced by the users home directory.
* `$UID` => Replaced by the users UID
* `$USER` => Replaced by the users name
A common use case for this field is to provide a local storage directory when user home directories are NFS-mounted (podman does not support container storage over NFS).
This feature lets an administrator set up a machine with NFS home directories point the user’s container content to a separate directory.
For example, see the following setting:
rootless_storage_path="$/var/lib/user/$USER/containers/storage"
If the user sally executed Podman as a normal user, Podman would attempt to create the directories /var/lib/user/sally/containers/storage
, and store the container images in that directory. It would also use these directories for all other containers. If ralph logged into the same machine and executed Podman, his storage would be /var/lib/user/ralph/containers/storage
. Obviously, these directories would need to pre-exist and be writable by the respective user. The administrator could set up the parent directories, /var/lib/users
, to be world-writable, and the Podman command would create the entire path.
Now, this leads me to my second point for this blog.
Using tmp directories for container storage
Administrators may be tempted to use /tmp
or /var/tmp
as container storage directories. Well, some of our users have reported problems when using these directories. The users reported that random files/directories disappeared from their images and containers. We diagnosed the problem and figured out that systemd
was doing tmpfile cleanup. Systemd
periodically triggers a job to clean up old content left in temporary directories. This job was erasing older files and directories stored in container-storage. The systemd-tmpfiles
does not know about these Podman directories - it just does its job.
Wrap up
The bottom line is that if an administrator wants to store his content in /var/tmp
or /tmp
, they need to modify the systemd-tmpfiles
to ignore the container content directories. Finally, on RHEL, CentOS, and Fedora, as well as many other distributions, the /tmp
is on a tmpfs, which by default is only allowed to store up to 50% the size of memory, and will be destroyed on reboot.
[ Getting started with containers? Check out this free course: Deploying containerized applications: A technical overview. ]
Dan Walsh
Daniel Walsh has worked in the computer security field for over 30 years. Dan is a Consulting Engineer at Red Hat. He joined Red Hat in August 2001. Dan leads the Red Hat Container Engineering team since August 2013, but has been working on container technology for several years. More about me
Qi Wang
I am an engineer in the containers engine team at Red Hat. I have a passion for container-related projects, where I started my career, and contribute to projects like Podman and Skopeo. More about me