Apache Iggy's migration journey to clustering powered by Viewstamped Replication

For about three months, from February to May 2026, a simulator was the only thing that could run Iggy's clustering code. We built the consensus, metadata, journal and message_bus crates next to the existing server, which kept shipping releases, and it was May before iggy-server-ng could boot a replica. The simulator runs a whole cluster on one thread, ten milliseconds at a time, and draws every packet delay, crash and task switch from a single 64-bit seed. We still test the cluster with it. It found both bugs in the fix from part 2's "The write we put back", and it's why every timeout in Iggy is a count of ticks.

Part 1 gave determinism as one reason we chose Viewstamped Replication: the primary of view v is replica v % n, so replicas that receive the same messages in the same order make the same decisions. That only helps if the clock, the network, the disk and the order async tasks run in are deterministic too, and most of the simulator's code is there to make them so.

Every replica in the simulator is a real shard::IggyShard running the real message pump, with one VsrConsensus group for metadata and one per partition, plus the metadata state machine and the partitions themselves. With --shell, client requests also go through the server's own request handling, login and session binding included. Everything that code touches on its way to the outside world is replaced:

The seam is the MessageBus trait, which carries the three calls that would otherwise reach the real world: sleep, spawn and realtime_micros. By default they're compio's timer, compio's spawn and the system clock, and the simulator overrides all three. The comment on sleep explains the choice:

Routed through the bus because the bus is the one dependency the shard already injects per environment; a separate clock generic would thread a second seam through every shard constructor for the same effect.

The login path checks a personal access token's expiry against realtime_micros. If it read the system clock instead, the reply a client gets would depend on the day the test ran. Consensus stamps prepares from an injected clock for the same reason, and the simulated clock starts at 2026-01-01 so those timestamps don't decode as 1970.

One call to Simulator::step() is one tick of virtual time:

1. advance time by 10 ms every pump's tick fires: heartbeats, retransmits, view changes run the pumps poll ready tasks until none is left
2. deliver packets every packet due this tick, each to a seeded shard of its replica run the pumps routing, consensus, replies and their loopback follow-ups
3. drain the outboxes everything sent this tick goes into the packet simulator
4. advance the network partitions and clogs start or end

When a replica has several shards, the seed picks which one each packet lands on. Usually that's a shard that doesn't own the packet, and the frame takes the same router hop it would take in production. Everything sent during a tick is on the simulated wire before network time moves, which keeps a reply from setting off another delivery in the same tick. None of this needs Docker, a server binary, sockets or io_uring. The only real files are the snapshots a checkpointing run writes.

Each consensus timer in Iggy holds a number of ticks, and every tick takes one off. TICK_INTERVAL in core/consensus/src/vsr_timeout.rs is 10 ms. In production the tick is one arm of the message pump's select_biased!, a bus.sleep(CONSENSUS_TICK_INTERVAL), and in the simulator the same call returns a virtual timer that advance_time fires.

The config is still written in durations. At boot each one is divided by 10 ms and rounded down, with a floor of one tick:

In production a tick takes 10 ms plus whatever it had to wait for. The pump re-arms its tick timer only after the tick's work is done, and the tick arm can wait behind a frame the pump is still processing, a journal fsync included. That makes 500 ticks of heartbeat window at least five seconds.

Because timers count ticks, the simulator decides when time passes. A five-second heartbeat window takes five seconds on a production cluster and 500 calls to step() in the simulator, each returning as soon as the pumps go quiet. Seed 211 from part 2 now passes:

$ workload-fuzz --seed 211 --ticks 4000 --plane metadata --faults none
--journal-slots 80 --crash-prob 0.02 --restart-prob 0.08 --crash-primary workload-fuzz: seed=211 ticks=4000 clients=1 replicas=3 plane=Metadata faults=None shell=false crash_prob=0.02 quiesce=true ... ran 4000 ticks; 54 replies; crashes=11 restarts=10 still down: 1 ... workload-fuzz: OK (seed=211)

That's 40 seconds of cluster time, with eleven crashes and ten restarts, in 0.24 seconds of wall time and 32 MB of memory.

The only randomness in consensus is the jitter on the prepare retransmit's backoff. (Part 2 explained why the view-change timers aren't randomized.) Each retry doubles the wait, up to 16 times the base, and adds a random amount up to the same again. That PRNG is seeded from the replica and group ids, so the jitter replays along with everything else.

Code that calls compio's timers directly can't run under the simulator, because compio panics outside its own runtime. A few storage paths still do, and two storage tests are #[ignore]d because of it: WriterLease::acquire drains through compio::runtime::time::timeout, which the deterministic executor can't drive.

SimSeeds::derive turns the run's seed into six child seeds, one each for the network, the workload, the executor, the choice of entry shard, crash scheduling and the swarm parameters. The comment on SimSeeds says why they don't share a PRNG:

A shared stream couples them: adding a draw anywhere shifts every later draw everywhere, so enabling crash injection would change the network trace, and a bug found at one seed would vanish once an unrelated draw was added.

The children are drawn from a parent PRNG in field order. An earlier version XOR-ed a different constant into the seed for each stream, and one forgotten constant is how the workload and the packet simulator ended up sharing a stream. Drawing needs no constants but makes field order part of the contract, so new streams go at the end of the struct. The workload's Action enum has the same rule, with a compile-time assert on its first three discriminants. With crash scheduling on a stream of its own, a run with both crash probabilities at zero draws nothing from it and replays bit for bit like a run with no injector.

We ran seed 211 twice for this post, and the two outputs matched apart from the paths of their data directories. In the test suite, workload_replay_is_deterministic pins that property: it runs seed 0xDEAD_BEEF twice, hashes every reply header and compares the result with 0xECC6_9F76_2D47_8F12. The constant moves whenever the draw order does, for instance when an op gains an outcome the workload can aim for, and the comment above it lists every reason it's been re-locked.

A seed replays only on the build that produced it. Draws go through rand's range reduction, which means a rand or rand_xoshiro upgrade can shift every schedule, and any change that adds a draw can too. When a seed matters, we turn it into a test that asserts the property it broke. That's how seeds 211 and 144 ended up in the suite.

The server is async Rust on compio, one thread per core. A shard runs its message pump alongside short-lived tasks for poll I/O, client requests and metadata submits, and bugs live in how those interleave. Much of our simulator is modelled on TigerBeetle's VOPR, but TigerBeetle's replicas don't run async tasks, so the VOPR has nothing to schedule at this level. Ours do, and we wrote DetExecutor for them.

DetExecutor keeps !Send futures on one thread and switches between them only at their await points. When several tasks are ready, it polls one picked uniformly at random from the executor's stream. Only advance_time moves its clock. Timers due at the same moment fire in the order they were set, and a sleep's deadline is fixed when the sleep is created, as in compio. The pump relies on that, since it builds its tick timer once and polls it many times. Every poll, timer fire and spawn is folded into an FNV-1a hash of the schedule, which determinism tests compare between runs. The hash is spelled out as FNV-1a because DefaultHasher may change between Rust releases.

A crash drops the replica's pump and every task its bus spawned where they stand, mid-await, without draining anything, and the packet simulator discards whatever is addressed to the replica. A restart builds new shards on the superblock and metadata WAL the simulator kept, under a new incarnation number derived from the seed.

Each round of polling has a budget of 100,000 polls. A task that keeps waking itself runs it out, and the run fails with the seed and the schedule hash. Whenever no task is ready, every live pump must also have an empty inbox. A frame still sitting in one means something queued it without waking the pump, a lost wakeup. In production the next tick would wake the pump and drain the frame, hiding the bug, so the simulator fails the run before that tick comes.

The partitions live in an UnsafeCell<Vec<_>>, so the borrow checker can't see a task holding a reference into it across an .await while another task reallocates the vector. #3557 removed the places where that happened. A debug-only guard now counts live partition borrows, and inserting or removing a partition asserts that none is held. Under DetExecutor a task parked on an await stays suspended while other tasks run, which shell_detects_partition_borrow_held_across_await relies on: it parks a read that holds a borrow, runs a removal and checks that the guard fires. The read it parks is synthetic, behind the simulator feature, because the production read path no longer holds a borrow across any await.

Most of the determinism work was finding code that read the outside world without saying so. Plain futures::select! takes its arm order from a thread-local PRNG the simulator can't seed, so the pump uses select_biased!. The partitions' namespace index is a BTreeMap: the pump walks it on every tick, and a HashMap's iteration order changes with each process's RandomState. Argon2 password hashes take a random salt. Under the simulator feature they use a fixed one, which keeps replicated user records the same on every run, and production keeps the random salt.

Repair and state-transfer sessions still take their nonces from Uuid::new_v4(), which reads OS randomness, at five places in core/shard/src/lib.rs. Nonces are only compared for equality, so the schedule and every decision still replay, though the bytes on the wire differ between runs.

The network is a PacketSimulator, ported with few changes. Each pair of processes has a link with a bounded queue, 64 packets by default, and each packet's delay is drawn from an exponential distribution with a floor. Loss and duplication are decided when a packet comes due. A full link evicts a random packet to make room. On each tick any path can clog for an exponentially distributed number of ticks, and the network as a whole can split. A partition is symmetric or asymmetric and takes one of three shapes: a random size, a random side for each replica, or one replica cut off. Links also filter by message type, which lets a test block only DoViewChange between two replicas. A crashed process has everything addressed to it dropped, and a partition that stood when it crashed still stands when it restarts.

--faults picks a profile. light and heavy are fixed, and swarm draws every parameter from the seed's swarm stream:

Faults cost progress, because every lost frame waits out a resend timeout. With one client on one partition, a thre