
A Better Benchmark Harness for SRE Agents
This report shows how to create a benchmark harness that injects realistic faults into Kubernetes microservices and has SRE agents work through them. Each agent is scored on five things: locating the fault, diagnosing t…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
This report shows how to create a benchmark harness that injects realistic faults into Kubernetes microservices and has SRE agents work through them. Each agent is scored on five things: locating the fault, diagnosing the root cause, identifying a mitigation, grounding its conclusions in evidence, and explaining its reasoning clearly. A panel of judges built on different models does the scoring, which keeps any single model’s bias out of the results. Initial comparative results demonstrate that PagerDuty’s SRE Agent consistently led the benchmark across 17 scenarios on issue localization, diagnosis, evidence grounding, mitigation recommendations, and explanation quality. The results also point to two factors that matter most: the strength of the underlying model and how much context the agent can reach. Both give teams a clear direction for improving how AI resolves incidents.
A reliable benchmark measures whether changes make an SRE agent better at resolving incidents. Building a basic SRE agent is easy. You take a powerful LLM, give it access to code and observability via MCP and put it all together in a simple harness. Job done. But to get from basic to great requires asking additional questions. How well does it work now? Is it solving the problem effectively? Is it doing it efficiently? How do you know the complexity you have added actually made things better? As you work to improve your agent, you need a reliable way to quantify the changes.
The obvious first step is to replay the agent on past incidents and compare its output with what your post-incident review says happened. We do this, and it’s useful, but it has hard limits as an evaluation method:
Injecting faults into a live system addresses each of these. The ground truth is exact because we caused it. Every agent investigates the present through its normal integrations. Every scenario can be repeated, varied, and ablated. And we choose which faults, technologies, and languages to cover instead of waiting for them to fail. Replay still has a role for signals a synthetic system can’t reproduce, such as incident-call transcripts (see Impact of transcript data), but for benchmarking we needed something new.
Another benefit of having a good benchmarking suite is for our customers: there are now many general-purpose and SRE-specific AI agents, and they want to know how PagerDuty’s SRE Agent compares. Answering that honestly means testing every agent the way a customer would run it, on the same faults and against the same ground truth. A well-designed benchmarking suite will allow us to do this as fairly as possible.
Our first question was whether we could adopt or adapt a benchmark that already exists. There are a number of public benchmark harnesses available, but they all presented a few drawbacks:
For these reasons, we created our own benchmark harness. It is a purpose-built financial application that runs on Kubernetes microservices, emits telemetry, has alerting rules on Grafana and sends those alerts to PagerDuty (or the agent’s own SaaS platform). With this standard infrastructure in place, we can integrate the stack with a variety of agents, and grant them access to metrics, logs, alerting and code to use in their RCA.
This accounted for all the basic information that is ingested by all SRE agents but does not include some of the highest value information that we have for PagerDuty’s SRE Agent:
Neither of these are easy to include in a benchmarking suite. LLMs can generate a fake incident call, but you don’t want them leaking the answer. Past incidents need to be similar enough to be useful but not the same. Memories should be clues, not solutions. Likewise, the general-purpose agents were run in default configurations, without SRE-specific instruction files, skills, or subagents that a customer might add.
We are publishing this report to outline how we are testing our agents and highlight the strengths and limitations of our work. Future reports will cover different aspects of that testing in more detail as we continue to work on it. This report is focused on the underlying principles we have adopted when building the benchmark, and the initial results we have achieved using it.
This benchmark has been undertaken knowing that there is a conflict of interest, we are both testing and building agents based on the same vision of what an SRE agent should be. To reduce this bias, we have implemented the following protections:
At the moment, we are keeping this codebase private (since we do not want it leaking into the LLMs training data), but we intend to make it available in the future to allow others to validate our results.
Our benchmark harness deploys a realistic, purpose-built financial app that runs on a number of interconnected Kubernetes services. It is a mono-repo consisting of 11 micro-services written in Go, Python, TypeScript and Elixir using Kafka, Redis, PostgreSQL and Secrets Manager, all running on EKS on AWS. They are instrumented to emit metrics and logs to Grafana Cloud. On the Grafana stack a number of alerting rules were created on telemetry such as HTTP errors, request latency, Kubernetes infrastructure health, etc. Additional services are stubbed out and will be added in future iterations using additional coding languages (e.g. Java) and technologies (e.g. MySQL) to provide a variety of scenarios and issues that can be used to test the agents. This organic growth has allowed us to test whilst we have continued to grow the benchmark.
Each service has a runbook that outlines how it works and how to debug “common faults”. These have been written by an LLM to replicate typical runbooks from effective teams and have been made available to all agents.
An external load-generation service posts requests to the financial app’s endpoints and exercises the whole system. This load-generation system does not emit MELT (metrics, events, logs and traces) data to the shared Grafana Cloud and is hidden from the agents.
We designed a number of self-contained incident scenarios, where operationally realistic faults are injected, leading alert rules to fire. These alerts are the trigger for agents to start their analyses. Faults can be injected using four different approaches:
Scenarios were classified in the following four categories:
They were often designed to have an impact on a service different from the one that had the problem, testing the SRE agent’s ability to traverse a codebase to determine the topology and call structure, identifying the causal service, not just the symptomatic one.
The ground truth was written by an LLM when the scenario was devised and the evidence produced (not the agents’ performance against that evidence) was reviewed after a couple of runs to ensure that it matched the designed scenario.
Each agent ran in parallel on the same scenario, ensuring that they all had access to the same data. Default configurations were used for all agents apart from models.
We specifically do not provide access to the k8s substrate directly since we are using Chaos Mesh and direct access would leak that information. We do export filtered logs from k8s to Grafana Cloud with all Chaos Mesh related logs filtered out to prevent this leakage. Post-run reviews at the trace level have found no evidence of the agents finding Chaos Mesh during their analysis.
Note: for PagerDuty’s SRE Agent we use the incident triggered by the first alert we receive, no matter what service it impacted. We do not use any of the PagerDuty features such as cross-service alert grouping and between runs all incidents are redacted and memories wiped to prevent them from being found using our past incident functionality, which identifies past incidents that are similar to this one and could be helpful in solving it. In all other respects, it is treated identically to all the other agents. These functionalities would create an uneven setup and, although testing has shown that they improve the results, it would defeat the purpose of this study.
For each agent being exercised, a custom adapter is created that defines how to:
Our benchmarking process runs as follows.
All successful agent responses are scored by 3 LLM judges (DeepSeek, Claude, GPT) to mitigate the risk of same family or provider bias. Agent failures are excluded from the analysis (not scored as zero). There is also a per-judge deviation check to ensure that the performance is comparable across judges. The judges are provided with:
In an incident, the root-cause is often a set of contributing factors and the mitigation of the issue may not imply resolving the issue but ensuring that there’s no impact. For example, when a change causes a problem in the system, we often roll it back to a previous version instead of trying to solve the bug during the incident. As such, the performance of an SRE Agent cannot be judged across a single axis. We split judging over the following axes to determine the performance of SRE Agents
The LLM judging has been benchmarked against a human evaluation showing agreement, but a full analysis of this work is ongoing. A contamination scanner is also run at this point to verify that there has been no answer-key leakage, GitHub leakage, and cross-scenario contamination. This has identified agents that have found other ways to diagnose the problem, e.g., through broader-scoped keys accessing the benchmarking repo directly. Contaminated agents were removed from the analysis and work was undertaken to prevent future leaks. In the latest runs, no contamination was found, but the scanner is still run to identify any regression.
The results presented here are for a single trial per agent. This is a known statistical limitation and ideally we would run a larger number of trials, but, with each scenario taking up to 1 hour to run, we have prioritised diversity of scenarios and quality of the benchmarking suite over statistical rigour at this stage. Final results will be presented using a larger number of trials.
In this benchmark study, PagerDuty SRE Agent scored first, followed by Claude Code (Opus 5.5) second, with Codex (Sol 5.6) placing third. The latest complete sweep of 17 scenarios, in equal conditions, produced the following results.
Note: All values are means per scenario. For Misdirection and Latency, lower is better; for all other metrics, higher is better. The average score is a simple average of Localization, Diagnosis, Mitigation, Evidence Grounding, Explanation Quality and (1 – Misdirection).
Earlier sweep-to-sweep comparisons suggest variability of roughly 2pp, though this sweep is a single trial. On that basis, PagerDuty’s SRE Agent led or was within noise of the lead on every quality metric. It was not the fastest.
Holding the model constant, the harness and context also mattered: on Sol 5.6, PagerDuty’s SRE Agent scored 0.77 against Codex’s 0.66. Model strength and operational context appear to add up rather than substitute for each other.
Aside from the headline result, one of the most valuable aspects of this work has been that it has identified areas of weakness in PagerDuty’s SRE Agent. The first iteration of its Deep Investigation mode we developed was a complex, multi-agent approach (see our blog on how we addressed some of the software engineering challenges). This worked well with the models that we first started developing PagerDuty’s SRE Agent with, but when we tested on the benchmark, it scored lower than a number of the other harnesses (on a subset of the above scenarios, about middle rank) and was also the slowest. Digging into it we found that:
Early runs of the benchmarking highlighted these issues and provided motivation to try a different approach. We rebuilt the agent using a new topology, which resulted in the 20pp increase that we have observed, as well
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。