Linux containers in 500 lines of code

I've used Linux containers directly and indirectly for years, but I wanted to become more familiar with them. So I wrote some code. This used to be 500 lines of code, I swear, but I've revised it some since publishing; I've ended up with about 70 lines more.

I wanted specifically to find a minimal set of restrictions to run untrusted code. This isn't how you should approach containers on anything with any exposure: you should restrict everything you can. But I think it's important to know which permissions are categorically unsafe! I've tried to back up things I'm saying with links to code or people I trust, but I'd love to know if I missed anything.

This is a noweb-style piece of literate code. References named <<x>> will be expanded to the code block named x. You can find the tangled source here. This document is an orgmode document, you can find its source here. This document and this code are licensed under the GPLv3; you can find its source here.

There are several complementary and overlapping mechanisms that make up modern Linux containers. Roughly,

These are all Linux kernel mechanisms. Seccomp, capabilities, and setrlimit are all done with system calls. cgroups is accessed through a filesystem.

There's a lot here, and the scope of each mechanism is pretty unclear. They overlap a lot and it's tricky to find the best way to limit things. User namespaces are somewhat new, and promise to unify a lot of this behavior. But unfortunately compiling the kernel with user namespaces enabled complicates things. Compiling with user namespaces changes the semantics of capabilities system-wide, which could cause more problems or at least confusion1. There have been a large number of privilege-escalation bugs exposed by user namespaces. "Understanding and Hardening Linux Containers" explains

Despite the large upsides the user namespace provides in terms of security, due to the sensitive nature of the user namespace, somewhat conflicting security models and large amount of new code, several serious vulnerabilities have been discovered and new vulnerabilities have unfortunately continued to be discovered. These deal with both the implementation of user namespaces itself or allow the illegitimate or unintended use of the user namespace to perform a privilege escalation. Often these issues present themselves on systems where containers are not being used, and where the kernel version is recent enough to support user namespaces.

It's turned off by default in Linux at the time of this writing2, but many distributions apply patches to turn it on in a limited way3.

But all of these issues apply to hosts with user namespaces compiled in; it doesn't really matter whether we use user namespaces or not, especially since I'll be preventing nested user namespaces. So I'll only use a user namespace if they're available.

(The user-namespace handling in this code was originally pretty broken. Jann Horn in particular gave great feedback. Thanks!)

This program can be used like this, to run /misc/img/bin/sh in /misc/img as root:

[lizzie@empress l-c-i-500-l]$ sudo ./contained -m ~/misc/busybox-img/ -u 0 -c /bin/sh => validating Linux version...4.7.10.201610222037-1-grsec on x86_64. => setting cgroups...memory...cpu...pids...blkio...done. => setting rlimit...done. => remounting everything with MS_PRIVATE...remounted. => making a temp directory and a bind mount there...done. => pivoting root...done. => unmounting /oldroot.oQ5jOY...done. => trying a user namespace...writing /proc/32627/uid_map...writing /proc/32627/gid_map...done. => switching to uid 0 / gid 0...done. => dropping capabilities...bounding...inheritable...done. => filtering syscalls...done. / # whoami root / # hostname 05fe5c-three-of-pentacles / # exit => cleaning cgroups...done.

So, a skeleton for it:

/* -- compile-command: "gcc -Wall -Werror -lcap -lseccomp contained.c -o contained" -- / / This code is licensed under the GPLv3. You can find its text here: https://www.gnu.org/licenses/gpl-3.0.en.html */

#define _GNU_SOURCE #include <errno.h> #include <fcntl.h> #include <grp.h> #include <pwd.h> #include <sched.h> #include <seccomp.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <time.h> #include <unistd.h> #include <sys/capability.h> #include <sys/mount.h> #include <sys/prctl.h> #include <sys/resource.h> #include <sys/socket.h> #include <sys/stat.h> #include <sys/syscall.h> #include <sys/utsname.h> #include <sys/wait.h> #include <linux/capability.h> #include <linux/limits.h>

struct child_config { int argc; uid_t uid; int fd; char *hostname; char **argv; char *mount_dir; };

<<capabilities>>

<<mounts>>

<<syscalls>>

<<resources>>

<<child>>

<<choose-hostname>>

int main (int argc, char **argv) { struct child_config config = {0}; int err = 0; int option = 0; int sockets[2] = {0}; pid_t child_pid = 0; int last_optind = 0; while ((option = getopt(argc, argv, "c:m:u:"))) { switch (option) { case 'c': config.argc = argc - last_optind - 1; config.argv = &argv[argc - config.argc]; goto finish_options; case 'm': config.mount_dir = optarg; break; case 'u': if (sscanf(optarg, "%d", &config.uid) != 1) { fprintf(stderr, "badly-formatted uid: %s\n", optarg); goto usage; } break; default: goto usage; } last_optind = optind; } finish_options: if (!config.argc) goto usage; if (!config.mount_dir) goto usage;

<<check-linux-version>>

char hostname[256] = {0}; if (choose_hostname(hostname, sizeof(hostname))) goto error; config.hostname = hostname;

<<namespaces>>

goto cleanup; usage: fprintf(stderr, "Usage: %s -u -1 -m . -c /bin/sh ~\n", argv[0]); error: err = 1; cleanup: if (sockets[0]) close(sockets[0]); if (sockets[1]) close(sockets[1]); return err; }

Since I'll be blacklisting system calls and capabilities, it's important to make sure there aren't any new ones.

fprintf(stderr, "=> validating Linux version..."); struct utsname host = {0}; if (uname(&host)) { fprintf(stderr, "failed: %m\n"); goto cleanup; } int major = -1; int minor = -1; if (sscanf(host.release, "%u.%u.", &major, &minor) != 2) { fprintf(stderr, "weird release format: %s\n", host.release); goto cleanup; } if (major != 4 || (minor != 7 && minor != 8)) { fprintf(stderr, "expected 4.7.x or 4.8.x: %s\n", host.release); goto cleanup; } if (strcmp("x86_64", host.machine)) { fprintf(stderr, "expected x86_64: %s\n", host.machine); goto cleanup; } fprintf(stderr, "%s on %s.\n", host.release, host.machine);

(This had a bug. captainjey on reddit let me know. Thanks!)

And I wasn't quite at 500 lines of code, so I thought I had some space to build nice hostnames.

int choose_hostname(char *buff, size_t len) { static const char *suits[] = { "swords", "wands", "pentacles", "cups" }; static const char *minor[] = { "ace", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten", "page", "knight", "queen", "king" }; static const char *major[] = { "fool", "magician", "high-priestess", "empress", "emperor", "hierophant", "lovers", "chariot", "strength", "hermit", "wheel", "justice", "hanged-man", "death", "temperance", "devil", "tower", "star", "moon", "sun", "judgment", "world" }; struct timespec now = {0}; clock_gettime(CLOCK_MONOTONIC, &now); size_t ix = now.tv_nsec % 78; if (ix < sizeof(major) / sizeof(*major)) { snprintf(buff, len, "%05lx-%s", now.tv_sec, major[ix]); } else { ix -= sizeof(major) / sizeof(*major); snprintf(buff, len, "%05lxc-%s-of-%s", now.tv_sec, minor[ix % (sizeof(minor) / sizeof(*minor))], suits[ix / (sizeof(minor) / sizeof(*minor))]); } return 0; }

Namespaces

clone is the system call behind fork() et al. It's also the key to all of this. Conceptually we want to create a process with different properties than its parent: it should be able to mount a different /, set its own hostname, and do other things. We'll specify all of this by passing flags to clone 4.

The child needs to send some messages to the parent, so we'll initialize a socketpair, and then make sure the child only receives access to one.

if (socketpair(AF_LOCAL, SOCK_SEQPACKET, 0, sockets)) { fprintf(stderr, "socketpair failed: %m\n"); goto error; } if (fcntl(sockets[0], F_SETFD, FD_CLOEXEC)) { fprintf(stderr, "fcntl failed: %m\n"); goto error; } config.fd = sockets[1];

But first we need to set up room for a stack. We'll execve later, which will actually set up the stack again, so this is only temporary.5

#define STACK_SIZE (1024 * 1024)

char *stack = 0; if (!(stack = malloc(STACK_SIZE))) { fprintf(stderr, "=> malloc failed, out of memory?\n"); goto error; }

We'll also prepare the cgroup for this process tree. More on this later.

if (resources(&config)) { err = 1; goto clear_resources; }

We'll namespace the mounts, pids, IPC data structures, network devices, and hostname / domain name. I'll go into these more in the code for capabilities, cgroups, and syscalls.

int flags = CLONE_NEWNS | CLONE_NEWCGROUP | CLONE_NEWPID | CLONE_NEWIPC | CLONE_NEWNET | CLONE_NEWUTS;

Stacks on x86, and almost everything else Linux runs on, grow downwards, so we'll add STACK_SIZE to get a pointer just below the end.6 We also | the flags with SIGCHLD so that we can wait on it.

if ((child_pid = clone(child, stack + STACK_SIZE, flags | SIGCHLD, &config)) == -1) { fprintf(stderr, "=> clone failed! %m\n"); err = 1; goto clear_resources; }

Close and zero the child's socket, so that if something breaks then we don't leave an open fd, possibly causing the child to or the parent to hang.

close(sockets[1]); sockets[1] = 0;

The parent process will configure the child's user namespace and then pause until the child process tree exits7.

#define USERNS_OFFSET 10000 #define USERNS_COUNT 2000

int handle_child_uid_map (pid_t child_pid, int fd) { int uid_map = 0; int has_userns = -1; if (read(fd, &has_userns, sizeof(has_userns)) != sizeof(has_userns)) { fprintf(stderr, "couldn't read from child!\n"); return -1; } if (has_userns) { char path[PATH_MAX] = {0}; for (char **file = (char *[]) { "uid_map", "gid_map", 0 }; *file; file++) { if (snprintf(path, sizeof(path), "/proc/%d/%s", child_pid, *file)

sizeof(path)) { fprintf(stderr, "snprintf too big? %m\n"); return -1; } fprintf(stderr, "writing %s...", path); if ((uid_map = open(path, O_WRONLY)) == -1) { fprintf(stderr, "open failed: %m\n"); return -1; } if (dprintf(uid_map, "0 %d %d\n", USERNS_OFFSET, USERNS_COUNT) == -1) { fprintf(stderr, "dprintf failed: %m\n"); close(uid_map); return -1; } close(uid_map); } } if (write(fd, & (int) { 0 }, sizeof(int)) != sizeof(int)) { fprintf(stderr, "couldn't write: %m\n"); return -1; } return 0; }

The child process will send a message to the parent process about whether it should set uid and gid mappings. If that works, it will setgroups, setresgid, and setresuid. Both setgroups and setresgid are necessary here since there are two separate group mechanisms on Linux9. I'm also assuming here that every uid has a corresponding gid, which is common but not necessarily universal.

int userns(struct child_config *config) { fprintf(stderr, "=> trying a user namespace..."); int has_userns = !unshare(CLONE_NEWUSER); if (write(config->fd, &has_userns, sizeof(has_userns)) != sizeof(has_userns)) { fprintf(stderr, "couldn't write: %m\n"); return -1; } int result = 0; if (read(config->fd, &result, sizeof(result)) != sizeof(result)) { fprintf(stderr, "couldn't read: %m\n"); return -1; } if (result) return -1; if (has_userns) { fprintf(stderr, "done.\n"); } else { fprintf(stderr, "unsupported? continuing.\n"); } fprintf(stderr, "=> switching to uid %d / gid %d...", config->uid, config->uid); if (setgroups(1, & (gid_t) { config->uid }) || setresgid(config->uid, config->uid, config->uid) || setresuid(config->uid, config->uid, config->uid)) { fprintf(stderr, "%m\n"); return -1; } fprintf(stderr, "done.\n"); return 0; }

And this is where