Single-tenant Titus: VM-level isolation on a multi-tenant container platform

Samartha Chandrashekar & Nick Parker You click play on a Friday night, and in the second that follows, dozens of services answer in milliseconds. Most of them run on Titus, our internal container platform. Titus is, in essence, a Kubernetes-based container hosting platform: we keep both the control plane and the data plane (the container hosts that actually run containers) patched and up to date, so application teams get a consistent and reliable container experience. They hand us a container, and we run it, with no host, cluster, or kernel to own. Titus launches millions of containers every week across tens of thousands of EC2 instances in multiple AWS regions. It is multi-tenant by default: many containers from many teams share each host. That density is what makes container compute efficient, and it is why Titus has been the default home for so much of Netflix’s compute for years. For most workloads, sharing a host works well. For a few, it does not, and those few include some of the most demanding real-time systems we run. This post is about them, and about single-tenant Titus, a mode that gives a task its own virtual machine while keeping everything else about Titus the same: the API, the deployment path, and the managed experience where the Compute/Platform team, not the application team, owns the host. We will walk through why shared-kernel isolation runs out of road for certain workloads, how we measure isolation rather than asserting it, the options we weighed (microVMs, gVisor, EC2, and more shared-kernel tuning) and why a dedicated VM inside Titus won, how the mode is built on the Titus federation layer and autoscaling, and what we have seen along three axes: isolation, fleet efficiency, and the operational cost of keeping the fleet patched.

The cost of good neighbors

Multi-tenancy is a tradeoff. Packing many containers onto a large bare-metal instance keeps utilization high, but it also means those containers share a single Linux kernel and the underlying hardware. Titus isolates them with the tools the kernel provides: cgroups cap CPU, memory, and disk; namespaces give each container its own view of the system; seccomp and AppArmor restrict the system calls a container can make; the Completely Fair Scheduler divides CPU time; the iocost controller meters disk IO; and traffic shaping meters each container’s slice of the network. Each container also gets its own identity, IAM role, and security groups, so tenants are isolated for security and resource management. These mechanisms provide isolation for what the kernel can partition. They cannot fully partition what is shared in hardware. A group of cores shares a last-level cache and a path to memory, so a container that suddenly pulls a lot of data from RAM evicts a neighbor’s cache lines and slows it down, even when both stay inside their CPU limits. There is no cgroup for memory bandwidth or cache residency, as there is for CPU shares. The interference is real, it is hardware, and it does not show up as either container exceeding its quota. A noisy neighbor and a co-located app each stay within their cgroup CPU and memory limits, but both draw on a shared last-level cache and memory bandwidth that lack cgroup fences. The noisy neighbor takes most of it, so the co-located app stalls and its tail latency rises even though no one exceeded a quota. We have written about this before. Predictive CPU isolation of containers at Netflix moved some placement decisions out of the kernel and into a service that forecasts each container’s CPU demand and re-pins containers to cores to reduce cache interference, formulated as a combinatorial optimization solved every minute. Noisy Neighbor Detection with eBPF instruments the kernel scheduler to surface per-container run-queue latency, so we can see a victim being starved in close to real time. And Mount Mayhem traced container-startup slowness to a global kernel lock amplified by modern CPU cache topology. Each of these raised the floor for everyone on the platform, and we are continuing that work. A decade of shared-kernel isolation, at Netflix and across the industry, has not made co-tenancy invisible, because a noisy neighbor is not one problem but a family of them. Almost every physical resource a host shares is its own axis of interference, and a cgroup can fence very few of them:

• CPU performance indeterminacy. Co-located threads contend for the run queue, so scheduling latency rises even while a container remains within its CPU quota. Separately, a neighbor with a large working set evicts a tenant’s last-level cache lines and saturates memory bandwidth, neither of which a CPU share can partition.
• Network bandwidth and packets per second. Per-container shaping competes for link bandwidth, and receives processing runs in softirq context, charging one container’s packet work to whatever CPU a neighbor is running on. A flood of small packets exhausts a core’s softirq budget long before any bandwidth cap is reached.
• Storage bandwidth and IOPS. A neighbor saturating a device’s queue serializes everyone’s I/O, and on cloud block storage, the per-volume and per-instance throughput ceilings are a shared budget in their own right.
• Long tail. Page-cache reclaim, NUMA and cross-socket interconnect traffic, TLB-shootdown interrupts, PCIe and GPU contention, package-level power and thermal throttling, and contention on kernel global locks (the class of problem behind Mount Mayhem) each degrade neighbors in their own way. What unites them is that they are physical, shared, and below the level a quota can reach. A platform can cap how much CPU time or memory a container is granted; it cannot hand each tenant its own slice of the memory bus, its own softirq budget, or its own portion of the last-level cache. Of these, the network dimension has bitten hardest in our experience. Per-container bandwidth shaping uses a hierarchical token bucket (HTB) queueing discipline that serializes traffic with a global lock to account for each container’s bytes and packets. Under a flood of small packets from a single container, contention for that lock can burn kernel CPU and starve co-located containers of the time they need to process their own traffic, resulting in TCP retransmits and tail-latency spikes on services that did nothing wrong. The mechanism meant to prevent noisy neighbors becomes one. The workloads most exposed to this are stateful, high-throughput streaming jobs: real-time data pipelines and stream processors that continuously move large volumes of data and maintain in-memory state. When one of these is perturbed, recovery is not cheap. A stream processor that falls behind or loses its place may have to restore from a checkpoint and pause processing for many minutes to catch up. For these jobs, intermittent interference from a neighbor is not a small slowdown that averages out; it is a stop-the-world event that pages an on-call engineer. The same teams report, accurately, that they never saw this on a plain EC2 instance.

Why we kept coming back to the VM

Full VM isolation, where a workload gets its own host and kernel, is the standard our customers are already happy with. The question was not whether VM isolation works; it was how to offer it without giving up what makes Titus worth using. We weighed four paths.

• Keep improving shared-kernel isolation. This helps every workload on the platform, needs no opt-in, and we are still investing in it (more on the roadmap below). Its limitation is structural. Each round of tuning narrows the gap without closing it, because the underlying hardware remains shared. Cache, memory bandwidth, and the kernel’s own shaping and scheduling locks are not things a container boundary can fully contain. Shared-kernel isolation is the right default and the wrong promise for a workload that cannot tolerate a bad minute.
• microVMs (Firecracker, Kata Containers). Lightweight VMs boot in about 150 milliseconds by emulating far less hardware than traditional VMs, which is why they power serverless platforms. They were the most appealing alternative on paper, and we carefully reviewed them. Three things made the cost hard to justify. First, the hardware virtualization they rely on is available only on bare-metal instances or through nested virtualization, and nested virtualization was not ready on the instance types we run at Netflix at the time, so adopting microVMs broadly would have constrained us to a narrow slice of the fleet. Second, there is no managed microVM service we could build on, which means we would have to run and maintain a virtualization layer ourselves: patching the hypervisor, live-migrating guests, and qualifying emulated devices. That is exactly the kind of operational surface we prefer not to own when we can avoid it. Third, even setting those aside, microVMs add virtualization overhead on CPU- and IO-bound work and do not support GPUs. Excellent isolation, but at a price in flexibility and operational load we were not willing to pay.
• gVisor. A user-space kernel that intercepts system calls provides sandbox-grade isolation without a hypervisor, thereby avoiding the bare-metal constraint. The catch is performance and dependency. Intercepting syscalls in user space adds overhead on IO-heavy paths, which are precisely the workloads we most need to protect, and the open-source project lags behind its internal counterpart in fixes and is effectively maintained by a single vendor. The isolation we would gain came with a throughput tax on the exact jobs we were trying to help.
• Move the workload to EC2. It is possible to run containers directly on EC2 instances. The problem is that the application team takes back the operational load that Titus exists to carry: choosing instance types, patching hosts, and sizing and scaling the fleet. For a team that came to Titus precisely to avoid those things, this is a regression in the experience, and the move between Titus and EC2 has historically been a one-way door that teams are reluctant to cross. Spectrum of isolation options from lighter and more shared to stronger isolation: shared-kernel containers, then gVisor, then microVMs (Firecracker or Kata), then a full VM as used by single-tenant Titus, which is highlighted. Each option is annotated with its launch latency, overhead, and the entity that operates it; single-tenant Titus is a dedicated EC2 instance with no additional virtualization overhead. There is also a question of what kind of isolation we actually need. microVMs and gVisor are built for security isolation between mutually distrusting tenants, the case where one tenant might actively try to attack or escape into another. Among Netflix teams, that threat model is more than the situation warrants. What these workloads need is resource and performance isolation: a guarantee that a neighbor cannot steal their CPU, cache, memory bandwidth, or network, not protection from a hostile co-tenant. A dedicated EC2 VM delivers exactly that, with hardware we do not have to emulate and a kernel we do not have to sandbox. We pay for the isolation we need rather than for a stronger security boundary. microVMs and gVisor asked for a large, bespoke investment with real risk in order to approximate something we could get directly: a VM. Using EC2 to run containers provides the VM, but takes away the managed experience. So we took the direct route and brought single-tenant VM isolation into Titus itself, under the existing API and the existing shared-responsibility model.

How single-tenant Titus works

The idea is simple to state: a single-tenant task runs on its own EC2 instance, sized to fit it, with nothing else scheduled alongside. That instance is still a Titus container host that the Compute/Platform team owns and patches. The team running the workload sees the same Titus API and the same deployment path. Nothing about how they package or ship the job changes. Side-by-side an