Scaling and benchmarking a critical message bus using a new indexing strategy

As with any performance work, we had to benchmark in order to validate that we were actually improving the system. We also wanted extensive testing, since the message publishing logic is in the critical path for Aria.

Theodor started by writing a tool to profile various recovery scenarios. That way, we could test out different configurations, such as the number of topic partitions, the degree of interleaving, the number of readers, etc., and see how changing these variables affected the performance of the system.

With the benchmark in hand, he implemented the initial version of the topic partition index. When we profiled this implementation, we found that the min-heap was the biggest bottleneck in our code. Before AI, this is probably where we’d pick a single implementation that appeared to be better. But these days, running experiments is cheap: Theodor prompted an agent to spin up five different heap implementations and profile them overnight. The next day, we had our answer: fast_heap_unboxed gave us a 2x performance improvement to our indexed recovery code.

Theodor tested the new index with plenty of expect tests, exercised the new logic in Antithesis, and finally had a swarm of agents analyze the code with high scrutiny.

Once we devised the index data structure, we needed to figure out an efficient representation. We didn’t want to make a ring buffer for each index, since ring buffers are not really dynamically resizable. They would therefore have to be sized to accommodate the worst case scenario. Specifically, index size is inversely proportional to message size (smaller messages mean more messages can fit in the tip store, which means you need a larger index). An Aria message can be as small as 32 bytes. If the entire tip store consisted of messages that small, the corresponding index would be 2GB. Keeping a 2GB index for each topic partition would waste too much memory.

Instead, we wanted an index to dynamically shrink and grow with the number of messages in its topic partition.

What we settled on was using a shared pool of blocks that each contain 1024 entries. The indexes point to a block and insert values into it. Once all of the messages within a block have left the ring buffer, blocks can be popped off from the index and re-used for other indexes: