Virtual Oxide Labs: Emulating a Full Oxide Rack

The trouble with testing and developing for rack-scale hardware is that rack-scale is not all that easy to come by. I mean, nevermind the whole DDR5-as-a-currency thing; even if all the parts were available in abundance, any completed racks go swiftly to the front lines (AKA our customers' data centers).

As a member of the Product Assurance Team at Oxide, I find that totally unfair yet completely reasonable. And so we have to get a bit creative. We have to be able to:

Replicate diverse customer networking topologies

Represent the hardware components (we make racks, after all) including our switches and microcontrollers

And the cherry on top: we need to do that without cutting special "test version" releases. Because we want to keep our CI sleek, but also because that wouldn’t be playing fair.

Now, don’t get me wrong — we’re not without our tricks. When I joined Oxide last year, one of the first things I was introduced to was our racklettes; that is, smaller than average racks of four sleds and two switches each. These racklettes can be initialized and run just like a real rack, precisely because they are a real rack. Just not in any form factor it would be economical to sell. But for testing they work like a charm. We have four of them today, soon to be eight, where we can test any combination of hardware/software requirement.

Do you remember the game Q*bert? It was one of the first wildly popular isometric games, and looked a little like this:

Why am I showing you a ridiculously small game screenshot? Well, each cube in that image represents a 'voxel,' or a point in 3D space. It’s actually the 3D version of a pixel. Volume Element + Pixel = Voxel. Neat, right?

Anyway, for Oxide, Virtual OXidE Labs (or Voxel) means an incredible amount of fidelity packed into a single piece of hardware. It is the spiritual successor to our prior virtual labs (known as a4x2) which allowed us to hack out decent racks running in VMs.

Pay attention to that word "fidelity" by the way. For it is, as Bryan Cantrill puts it, the Prime Directive: labs should use stock Oxide software without hack or modification.

This is what Voxel gives us. A complete software emulation stack, built on illumos in Rust, where we can build and test a variety of realistic configurations:

BGP (numbered and unnumbered) or static networking with Debian and FRR

P4-powered switches via Oxide’s own SoftNPU, a software P4 switch

Microcontroller simulation and emulation for our Service Processor (SP) and Root of Trust (RoT)

And what’s more, it gives us all these features in a fairly small footprint. At home, I run Voxel labs on a Minisforum MS-A2 mini-PC with a 32-thread AMD Ryzen 9 9955HX and 96GB RAM. On that computer, I can spin up 6 sleds with ease, either in the form of one rack or in a multirack configuration (two separate racks, one management plane). All it requires is Oxide Helios and a decently fast NVMe drive. Launch, and in about 10 minutes you have a full Oxide rack with each "sled" in its own Propolis VM — alongside switches (also in Propolis) and other infrastructure.

Remember how I said we have to use stock Oxide software? That actually makes things easier. The first thing we do (following a common practice in hypervisor tooling) is to create a reusable snapshot.

To create a snapshot for Voxel, all we must do is create a "sled" (virtual machine), install our control plane (Omicron) on it, snapshot it before the point of configuration, et voilà: a ready-to-launch sled. The control plane can be installed one of two ways:

Voxel clones the Omicron repo on GitHub (or uses a locally defined source for rapid development) and uses cargo to compile

We use Tufaceous to package Omicron releases using The Update Framework (TUF) and our CI tool Buildomat

Using the second option as an example, here’s the beginning of the process where voxel image create pulls out the key pieces that make up the sled config:

$ pfexec ./target/debug/voxel image create --from-tuf ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip 5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c [voxel] TUF source ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip: system version 23.0.0-0.ci+git5d176e0be7e, omicron 5d176e0be7e [voxel] extracted 12 zones from ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip [voxel] extracted 3 measurement corpus artifacts [voxel] extracting host artifacts -> /export/home/helios/voxel-rawfix/target/debug/../../voxel-image/.tuf-host [voxel] host boot image 1258295296 bytes, phase 1 rom 33554432 bytes [voxel] extracted SP/RoT firmware -> /export/home/helios/voxel-rawfix/target/debug/../../voxel-image/.tuf-fw/ca9184058bd7

After a decent amount of unpacking and staging, it will launch a VM that everything gets written/installed to. And finally:

[voxel] ready: voxel-cp version=unknown unpacked_artifacts=18 built=2026-09-06T21:37:58 [voxel] clearing device-instance map, halting [voxel] flushing, up to 25s [voxel] stopping hypervisor [voxel] writing voxel image voxel-cp-5d176e0be7e-tuf

That image is a zfs snapshot which can be used and reused to create our rack.

Launching the rack uses a similar process, except the control plane is already installed. All we need at this point is:

An "frr" image (the Debian VMs for networking)

A control plane image (like the one we just created)

The voxel-init agent that gives each virtual sled its assignment

The log output is pretty long, but you can scroll through it if you’d like. I’ll recap after.

$ pfexec ./target/debug/voxel launch --emu [voxel] sled-agent config schema (image properties): data_links=Tagged disks=Hardcoded Writing key pair to "./cargo-bay/sprockets/test-root-a.key.pem" Writing key pair to "./cargo-bay/sprockets/test-signer-a1.key.pem" ... Writing certificate to "./cargo-bay/sprockets/test-root-a.cert.pem" Writing certificate to "./cargo-bay/sprockets/test-signer-a1.cert.pem" ... Writing pki path to "./cargo-bay/sprockets/test-sprockets-auth-0.certlist.pem" Writing pki path to "./cargo-bay/sprockets/test-alias-0.certlist.pem" ... add net fdb0:a840:2500:1::/64: gateway fdb0:a840:25ff:0::2 add net fdb0:a840:2500:5::/64: gateway fdb0:a840:25ff:0::4 [sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300 [identity] seed = 0x2aac7ae3f8cfbb1e (3074967765137734430) [identity] SP UID 13120858064df80da1642f2b, RoT UUID 00a94a8e206f3f138a790fba1b12db36 [flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/sidecar.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300/archives/sp.zip flashed 786688 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300/sp-flash.bin [sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310 [identity] seed = 0xa65abed0e20d08d1 (11987103162554976465) [identity] SP UID 72a3c3327682e0df38de478a, RoT UUID fcb600f7322f3393899f72036aa2b4fd [flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310/archives/sp.zip flashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310/sp-flash.bin [sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320 [identity] seed = 0xbab29f9c81fda354 (13452990531453690708) [identity] SP UID 0883f59f877ab40da88e8072, RoT UUID 15468fb5bffd30d3b381e8759621fb4f [flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320/archives/sp.zip flashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320/sp-flash.bin [sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330 [identity] seed = 0xea954465eef85936 (16903491980996663606) [identity] SP UID 49c0e331f4877c7d0afc71eb, RoT UUID 03ec945840463823a95d38dda9e261d1 [flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330/archives/sp.zip flashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330/sp-flash.bin [voxel] rack 0 SP fleet up on fdb0:a840:25ff:0::1 (4 SP(s): [33300, 33310, 33320, 33330]) Sep 06 22:15:47.126 INFO launching runner: deployment(voxel) persistent(true) custom_propolis_binary(Some("/export/home/helios/propolis-server")) dataset(testbed/falcon) falcon_dir(.falcon) Sep 06 22:15:47.126 INFO starting preflight for deployment voxel Sep 06 22:15:48.162 INFO creating links Sep 06 22:15:48.162 INFO creating simnet link 'voxel_ce_vn_sim0' Sep 06 22:15:48.164 INFO creating vnic link 'voxel_ce_vn_vnic0' ... Sep 06 22:15:48.249 INFO creating external links Sep 06 22:15:48.249 INFO creating external link voxel_ce_vn_vnic2 Sep 06 22:15:48.251 INFO creating external link voxel_g0_vn_vnic3 ... Sep 06 22:15:48.255 INFO creating nodes Sep 06 22:15:48.255 INFO g0: launching node Sep 06 22:15:48.257 INFO g1: launching node ... Sep 06 22:15:48.290 INFO launched instance g0 with pid 13041 on port 43736 Sep 06 22:15:48.291 INFO g0: instance ensure Sep 06 22:15:48.295 INFO launched instance g1 with pid 13042 on port 33187 Sep 06 22:15:48.295 INFO g1: instance ensure Sep 06 22:15:48.297 INFO launched instance g2 with pid 13043 on port 51110 ... Sep 06 22:15:52.423 INFO g1: instance ensure completed after 0 retries Sep 06 22:15:52.423 INFO g1: instance run Sep 06 22:15:52.479 INFO g2: instance ensure completed after 0 retries Sep 06 22:15:52.479 INFO g2: instance run Sep 06 22:15:52.534 INFO g0: instance ensure completed after 0 retries Sep 06 22:15:52.534 INFO g0: instance run Sep 06 22:16:03.961 INFO cr1: mounting /opt/cargo-bay Sep 06 22:16:03.969 INFO cr2: mounting /opt/cargo-bay Sep 06 22:16:03.994 INFO cr1: finished mounting /opt/cargo-bay Sep 06 22:16:03.995 INFO ce: mounting /opt/cargo-bay Sep 06 22:16:04.002 INFO cr2: finished mounting /opt/cargo-bay ... Sep 06 22:16:09.236 INFO ce: launch start Sep 06 22:16:09.237 INFO cr1: launch start Sep 06 22:16:09.237 INFO cr2: launch start Sep 06 22:16:12.893 INFO ce: [voxel-init] pinned enp0s10 DHCP client-id to MAC; re-DHCPing host-LAN uplink Sep 06 22:16:12.893 INFO ce: [voxel-init] NAT rack egress via enp0s10 Sep 06 22:16:12.894 INFO ce: [voxel-init] static edge IP 192.168.68.170/22 added on enp0s10 Sep 06 22:16:12.894 INFO ce: [voxel-init] router bring-up complete Sep 06 22:16:12.894 INFO ce: launch ok Sep 06 22:16:12.956 INFO cr1: [voxel-init] pinned enp0s11 DHCP client-id to MAC; re-DHCPing host-LAN uplink Sep 06 22:16:12.956 INFO cr1: [voxel-init] NAT rack egress via enp0s11 Sep 06 22:16:12.956 INFO cr1: [voxel-init] router bring-up complete Sep 06 22:16:12.956 INFO cr1: launch ok Sep 06 22:16:12.985 INFO cr2: [voxel-init] pinned enp0s11 DHCP client-id to MAC; re-DHCPing host-LAN uplink Sep 06 22:16:12.985 INFO cr2: [voxel-init] NAT rack egress via enp0s11 Sep 06 22:16:12.985 INFO cr2: [voxel-init] router bring-up complete Sep 06 22:16:12.985 INFO cr2: launch ok Sep 06 22:16:12.985 INFO g0: launch start Sep 06 22:16:12.986 INFO g1: launch start Sep 06 22:16:12.987 INFO g2: launch start Sep 06 22:16:28.581 INFO g1: [voxel-init] underlay(jumbo)=["vioif0", "vioif1"] ext-candidates=["vioif2"] Sep 06 22:16:28.581 INFO g1: [voxel-init] c2t1d0 (2FAKE001-M20): M.2 layout written Sep 06 22:16:28.581 INFO g1: [voxel-init] c2t1d0: host phase 2 seeded Sep 06 22:16:28.581 INFO g1: [voxel-init] c3t1d0 (2FAKE001-M21): M.2 layout written Sep 06 22:16:28.582 INFO g1: [voxel-init] c3t1d0: host phase 2 seeded Sep 06 22:16:28.582 INFO g1: [voxel-init] c4t1d0 (2FAKE001-U20): labeled Sep 06 22:16:28.582 INFO g1: [voxel-init] c5t1d0 (2FAKE001-U21): labeled Sep 06 22:16:28.582 INFO g1: [vox