Animations for System Builders

Again, we have load spike triggering the metastable state. Writes, in this case. The overloaded peers look like they’re unavailable, which triggers a storm of application-level resends. I like this example, because etcd is widely-used quality software. Operational experience doesn’t seem to effectively shake out these kinds of issues.

When I wrote Garbage Collection and Metastability, a widely-known systems academic told me that it was a fantasy. I’ve seen it happen in the wild in ways I couldn’t share, and I can forgive people for not believing this can happen based on the hand waving in the blog.

So it’s nice to have a real world example, picking on etcd again.

At the start, we’re ticking along nicely at 500 reads/second, before spiking load to 1000 reads/s. The read spike increases the time spent in GC, increasing average tenure, and making less CPU available for the read workload. This is a self-reinforcing loop, and the system doesn’t recover without intercession.

I believe the same thing could happen on a runtime without GC, due to increasing work in the allocator, but haven’t been able to reproduce that yet.

The animations here are all generated with Kiro and KiroCrew, mostly with Opus 5.5. The actual results came from a real test environment, which captures the traces that back the animations.

Marc Brooker

The opinions on this site are my own. They do not necessarily represent those of my employer.

marcbrooker@gmail.com