Hi everyone!

I’m in the process of finally doing containers right in my NixOS installation. This is my ‘wishlist’:

  • podman containers should be run by users with minimal permissions
  • separate user per container
  • containers managed by systemd services for easier management

My current work-in-progress setup looks like this:

For each service (called $name), I have:

  • a user and corresponding group (referred to as $uid in the following)
  • a directory /srv/$name owned by $uid, in which mounted volumes are located

My containers are declared like this:

virtualisation.oci-containers.containers = {
    $name = {
        image = ...;
        ports = [ ... ];
        volumes = [
            "/srv/${name}/config:/config"
            ...
        ];
        user = $uid:$gid;
        extraOptions = [
            "--security-opt=no-new-privileges:true"
        ];
    };
};

Now for the parts I don’t fully understand yet:

  • some images allow setting environment.PUID to specify a user. Does setting this option (and not setting user=$uid in the container declaration itself) mean that the container will be run as root, and the program inside will merely use PUID when e.g. creating files? This would still allow a malicious container to run commands as root on the host, right?

  • virtualisation.oci-containers.containers creates a systemd service. Since this is not a user-service for my user $uid, I need sudo to start/stop the container. Does that mean that the systemd service is run with root permissions, but it executes the command to spawn the container as $uid? If whatever is running inside the container was malicious, is there a functional difference between the container being started ‘by root as $uid’ and it being started by me (after logging in as $uid)?

  • Is it feasible to make these systemd services user-services owned by $uid instead?

  • Are there further hardening steps I forgot about?

Thanks for your input!

  • chameleon@fedia.io
    link
    fedilink
    arrow-up
    10
    ·
    2 days ago

    PUID is indeed handled inside the container itself, it’ll run a container-provided script as whatever the container’s UID 0 happens to be first which then drops to whatever $PUID happens to be inside the container. user= is enforced by Podman itself before the container starts, but Podman will still run as root in that setup. That means Podman is running “rootful”, while if you started the container manually as $uid using the regular Podman CLI, it would be “rootless”. That is a major difference in a lot of respects, including security, and you can find quite a bit of documentation on the differences between those operating modes online; it wouldn’t fit in a comment. Rootless is generally considered the better mode, though there are some things that still require a rootful container.

    In the upcoming NixOS 25.05 or current unstable, there are some tools you can use to run containers rootless as another user more easily using a new $name.podman.user = ""; setting. From what I understand they’ll still be root-managed systemd system services that require sudo to operate, but that means privileges get dropped by systemd before running Podman, instead of dropped by Podman before running the container. This stuff is recent and I haven’t used it, I just happen to know it exists, relevant nixpkgs commit if you wanna dig into it yourself: https://github.com/NixOS/nixpkgs/commit/7d443d378b07ad55686e9ba68faf16802c030025

    • ftbd@feddit.orgOP
      link
      fedilink
      English
      arrow-up
      4
      ·
      2 days ago

      You just made my day, kind internet person! That’s exactly the holy grail setup I’ve been looking for for the last couple of months. Will try it out as soon as I can!

  • ReedReads@lemmy.zip
    link
    fedilink
    English
    arrow-up
    4
    ·
    2 days ago

    What services are you running in your pods/containers? Are they local applications like libreoffice or are they network accessible in the more traditional style? What’s the advantage to running a podman container on your machine vs a Flatpak container?

    Sorry for all the questions. This is an interesting setup and I’m just really curious.

    • ftbd@feddit.orgOP
      link
      fedilink
      English
      arrow-up
      4
      arrow-down
      1
      ·
      2 days ago

      These containers are running on various servers I have at home, not on a desktop machine. I use podman as an alternative to docker, because it’s fully libre and does not require running containers as root. To be honest, I’ve never thought about running flatpak containers for these kinds of services – do you have a setup like this that you want to share?

      • ReedReads@lemmy.zip
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        2 days ago

        That makes sense. I’ve always thought of NixOS as a desktop distro, not as a server. Guess I need to expand my thinking!

        I run Fedora Server with podman and docker side by side. I try to use podman whenever possible but sometimes it’s not worth the hassle so that’s when it becomes a docker container 😬

        • ftbd@feddit.orgOP
          link
          fedilink
          English
          arrow-up
          2
          ·
          2 days ago

          I’d say NixOS is great for servers, mostly. Only having to worry about certain things (secure boot with custom keys, FDE, partition layout, network, sshd, firejail, etc.) once, and then replicating the same setup on another machine is waaay more convenient than going “I wonder what I was thinking when setting up this machine” once in a while when looking at some machine again you haven’t touched in some time. When it comes to desktop usage, the whole thing does not feel as magical - configuring system options in e.g. KDE is still a lot of clicking around in a GUI. I still use it for my desktop machine, just so I don’t have to think about another distro.