In the last yearly update, I talked about isolating my self-hosted LLMs running in Ollama as well as Open WebUI in systemd-nspawn containers and promised a blog post about it. However, while writing that blog post, a footnote on why I am using it instead of Docker accidentally turned into a full blog post on its own. Here’s the actual post on systemd-nspawn.

Fundamentally, systemd-nspawn is a lightweight Linux namespaces-based container technology, not dissimilar to Docker. The difference is mostly in image management—instead of describing how to build images with Dockerfiles and distributing prebuilt, read-only images containing ready-to-run software, systemd-nspawn is typically used with a writable root filesystem, functioning more similarly to a virtual machine. For those of you who remember using chroot to run software on a different Linux distro, it can also be described as chroot on steroids.

I find systemd-nspawn especially useful in the following scenarios:

In this post, I’ll describe the process of setting up systemd-nspawn containers and how to use them in some common scenarios.

Update (2025-05-19): added safer Alpine container creation instructions and some more guidance on setting up Alpine Linux containers.

systemd-nspawn, like all containers, requires a separate rootfs in a directory. Since it doesn’t come with an image manager to build images for you, you’ll need to use the traditional Unix approach of creating a chroot. It is, after all, chroot on steroids.

For systemd-nspawn, the typical convention is to put these rootfses under /var/lib/machines/[hostname], where the hostname is that of the container. For this exercise, I’ll just use the name of the OS as the hostname.

Each Linux distro has its own way of creating a new rootfs, so we’ll go through some of the more common ones. I am testing all this on Debian, but there’s no reason why you wouldn’t be able to do any of this on another distro by substituting the equivalent packages. Still, I’ll probably go deeper into Debian because I am more familiar with it.

debootstrap has featured on this blog many times, mostly when doing crazy things like installing Debian by hand from rescue systems. For Debian-based distros like Ubuntu or Debian itself, this is the tool to use to create a rootfs.

On a Debian-based OS, install it with sudo apt install debootstrap. It may also be packaged on other distros, or you can just download it from Debian’s repositories:

$ curl -L https://salsa.debian.org/installer-team/debootstrap/-/archive/master/debootstrap-master.tar.gz | tar xz ... $ cd debootstrap-master/ $ ./debootstrap I: usage: [OPTION]... <suite> <target> [<mirror> [<script>]] ...

Either way you installed debootstrap, let’s create the rootfs in /var/lib/machines/debian:

$ sudo debootstrap --include=systemd,dbus,locales,bash-completion,curl bookworm /var/lib/machines/debian I: Target architecture can be executed I: Retrieving InRelease ... I: Base system installed successfully.

Here, we’ve installed bookworm, but feel free to substitute this for the latest Debian release. To avoid silly issues, I ensured that systemd and dbus are installed, along with locales to avoid weird warnings in apt, bash-completion so I don’t hate my life when typing in commands by hand, and curl so you can download stuff later. Some of these require a bit of additional configuration later, which we’ll do once we boot into the container.

First, install dnf and rpm if you are on another distro, e.g. on Debian:

$ sudo apt install dnf rpm ... Setting up dnf (4.14.0-3+deb12u1) ... Setting up rpm (4.18.0+dfsg-1+deb12u1) ... ... $ sudo mkdir -p /var/lib/rpm $ sudo rpm --initdb

Then, we make a Fedora 41 chroot as an example:

$ sudo mkdir -p /var/lib/machines/fedora/etc/dnf $ sudo tee /var/lib/machines/fedora/etc/dnf/dnf.conf > /dev/null [fedora] name=Fedora $releasever - $basearch metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch gpgkey=https://getfedora.org/static/fedora.gpg

[updates] name=Fedora $releasever - $basearch - Updates metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch gpgkey=https://getfedora.org/static/fedora.gpg $ sudo dnf --releasever=41 --best --setopt=install_weak_deps=False --repo=fedora --repo=updates --installroot=/var/lib/machines/fedora install dnf fedora-release glibc glibc-langpack-en iproute iputils less ncurses passwd systemd systemd-networkd systemd-resolved util-linux vim-default-editor ... Transaction Summary

Install 137 Packages

Total download size: 62 M Installed size: 206 M Is this ok [y/N]: y ... Complete!

Installing Alpine Linux with alpine-nspawn-install

Alpine Linux is commonly used in Docker images due to its lightweight size, so I thought I might as well try creating an Alpine Linux chroot. The usual tool for this sort of thing is alpine-chroot-install, but its default behaviour is horrifying for our use case. For example:

It is possible to run a crazy umount command afterward to clean up after alpine-chroot-install, such as:

mount | grep /var/lib/machines/alpine | cut -d' ' -f3 | sort -r | while read -r path; do sudo umount "$path"; done

However, I deem this a ticking time-bomb waiting to explode in someone’s face, so instead, I forked the script and created alpine-nspawn-install. My version removes all the crazy “features” of alpine-chroot-install that don’t make sense for systemd-nspawn, e.g. the mounting, chroot helper scripts, qemu support, etc.2, while installing a more sane list of packages by default. You can use it like this:

$ wget https://raw.githubusercontent.com/quantum5/alpine-nspawn-install/refs/heads/master/alpine-nspawn-install ...

2025-05-19 22:23:27 (47.6 MB/s) - ‘alpine-nspawn-install’ saved [15897/15897] $ sudo bash alpine-nspawn-install -d /var/lib/machines/alpine ...

Alpine container installation is complete.

Now let’s see if Alpine lives up to its reputation:

$ sudo du -sh /var/lib/machines/debian 338M /var/lib/machines/debian $ sudo du -sh /var/lib/machines/fedora 507M /var/lib/machines/fedora $ sudo du -sh /var/lib/machines/alpine 16M /var/lib/machines/alpine

Well, that’s certainly impressive… Alpine is an order of magnitude smaller than the other two, while Fedora is quite bloated with the list of packages I have installed. I am not a Fedora user and copied the package list from the Arch Linux wiki guide on systemd-nspawn, so I am not sure if it’s because I’ve installed bloat, but they all seem pretty necessary… It’s certainly bigger than what dnf claimed to be, so I took a look, and dnf created 260 MiB of stuff under /var/cache/dnf.

Also to be fair to Debian, I did install a bunch of “bloat” like locales for a more pleasant experience. Removing the “bloat” results in 305 MiB, which is slightly smaller.

First, install systemd-nspawn with sudo apt install systemd-container or your distro’s equivalent.

Then, you can get a root shell in a container by running the following command:

$ sudo systemd-nspawn -D /var/lib/machines/[name] -M [name] --private-users=pick --private-users-ownership=chown

And here is what each argument does:

$ sudo systemd-nspawn -D /var/lib/machines/debian -M debian --private-users=pick --private-users-ownership=chown Spawning container debian on /var/lib/machines/debian. Press ^] three times within 1s to kill container. Selected user namespace base 1766326272 and range 65536. root@debian:~#

And we are in the newly created Debian container. Note that this is just a root shell and isn’t a proper way to deploy the container long term. There is no init process even.

If you are going to be using a root shell for a significant amount of time and want autocomplete to avoid typing entire commands:

root@debian:# cp /etc/skel/.bashrc . root@debian:# . ~/.bashrc

If you are just using the container to run a single application, it can get really tedious to manage your own users and then sudo, so I just opt to do the management as root and having a functioning .bashrc works wonders. This is technically not best practice, but I basically never run commands in single application containers that don’t require root anyway, so it makes no real difference.

debootstrap also copied the host’s hostname into /etc/hostname. Let’s fix that:

root@debian:~# hostname > /etc/hostname

Getting back into the container

If you exited the root shell for some reason, you can go back by doing:

$ sudo systemd-nspawn -D /var/lib/machines/debian -M debian --private-users=yes Spawning container debian on /var/lib/machines/debian. Press ^] three times within 1s to kill container. Selected user namespace base 1766326272 and range 65536. root@debian:~#

Preparing the container

Running the container like this isn’t a long-term solution. You probably want to set it up to run properly in the background. However, this leaves the question of how to access the container. If you are using a systemd-based distro, you have a few options:

root@debian:~# passwd New password: Retype new password: passwd: password updated successfully

Install openssh-server and set up SSH keys to connect remotely (you’ll also need to set up networking, as explained later, or you won’t be able to connect):

root@debian:# apt install openssh-server ... Created symlink /etc/systemd/system/sshd.service → /lib/systemd/system/ssh.service. Created symlink /etc/systemd/system/multi-user.target.wants/ssh.service → /lib/systemd/system/ssh.service. Processing triggers for libc-bin (2.36-9+deb12u10) ... root@debian:# curl -q https://ssh.qt.ax -o ~/.ssh/authorized_keys % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 243 100 243 0 0 3934 0 --:--:-- --:--:-- --:--:-- 3983

Note that because we only have a root shell right now, not a proper boot, systemd (or any other init system) isn’t actually running—so don’t expect to be able to start sshd for the moment. The next step will boot the container properly.

Note that console and shell access through machinectl requires systemd to be running inside a container. This will not be possible when using a non-systemd distro, such as Alpine. If you are using Alpine, you’ll need the openssh-server route.

The default Alpine Linux install created by alpine-chroot-install is configured exclusively for chroot use and doesn’t even include a functioning init system. To run it in a systemd-nspawn container properly, we’ll need to install openrc, their init system, as well as openssh-server:

alpine:# apk add openrc openssh-server ... alpine:# rc-update add sshd alpine:~# rc-update add networking

If you are using my alpine-nspawn-install script instead, this is all done for you. All you have to do is some basic configuration:

alpine:# hostname > /etc/hostname alpine:# mkdir -p /.ssh alpine:# wget https://ssh.qt.ax -O ~/.ssh/authorized_keys ...

Alpine uses ifupdown-ng to configure networking, and the syntax should be compatible with Debian’s, so you can just copy the examples below. You’ll need to do the networking configuration before starting the container, or you will be unable to log in via ssh.

We can create /etc/systemd/nspawn/[name].nspawn to manage the container with machinectl, e.g. machinectl start [name]. Note that the /etc/systemd/nspawn directory isn’t created by default, so you may need to:

$ sudo mkdir -p /etc/systemd/nspawn

The .nspawn file is a text file in INI format that contains the equivalent of command line arguments to systemd-nspawn. For the basics, do the equivalent of --private-users=yes:

[Exec] PrivateUsers=yes

There are many other options, such as networking, resource limits, and bind mounting directories on the host system into the container. We’ll cover those later.

For non-systemd distros like Alpine, add KillSignal=SIGTERM to the [Exec] sect