Sample production logs into a golden set, write deterministic assertions, diff baseline against candidate and gate it in CI. Built on six real failures.

Your provider is retiring the model you run in production, or a cheaper one just shipped and someone has done the arithmetic. Either way, one string in a config file is about to change, and nothing in your test suite has an opinion about what the model says. Here is the process for giving it one: sample real inputs from production into a golden set, write a deterministic assertion for each way an upgrade can break you, run the current model and the candidate over the same cases and diff the pass rates, then put that diff in CI so the next upgrade is a pull request check instead of a week of support tickets.

Everything below runs from one small repo, evalshift/llm-regression-example ↗, and every failure in it is real: it comes from upgrading a three-route agent from Sonnet 5 to Sonnet 5.5. The repo ships the recorded outputs, so the diff runs offline, with no API key.

The reason this needs a process rather than an afternoon in the playground is that the bugs a model upgrade causes are mostly silent. The output still reads fine. A tool argument changed shape, an enum value moved one notch, a ticket landed in a different queue, and nobody sees it until something downstream counts it.

The agent in the example has three routes: a day planner and a unit converter that call tools, and a ticket triage classifier that returns JSON. Thirty recorded production turns, replayed on both models. Six of the thirty failed a check on the new model that the old one passed, and every one of them produced output a reviewer would have waved through.

The table is from a toy agent, but none of the rows is toy-specific. Every production agent I have looked at has an enum field, a tool argument with a required shape, and a case where the right answer is to not call a tool. The upgrade that gets one of those wrong does not throw.

The playground fails for a structural reason: the prompts you can think of are the prompts you already handle. The cases that break are the ones a user typed, so the golden set has to come from logs. Log one line per model turn, with everything the assertions will later need:

{"request_id": "planner-000", "ts": "2026-10-12T20:35:51Z", "route": "planner", "history": [], "input": "What's on my calendar today?", "output": "You have Team standup at 9:30–10:00 AM on Zoom, and a Dentist appointment ...", "tool_calls": [{"name": "get_schedule", "args": {"date": "2026-09-08"}}], "latency_ms": 1863}

The tool_calls and the output are what production did, and they become the expectations. (The dates disagree on purpose. The example agent runs on a frozen clock: its system prompt says "Today is Tuesday, 2026-09-08", so every replay resolves "today" to the same date, even for a turn logged on October 12.) Then sample. Aim for 100 to 300 lines, stratified by route or feature, because a 5% regression on a route that is 5% of traffic is invisible in an unstratified sample. Strip PII before anything touches disk. Drop repeated inputs: twenty copies of "where is my order" make the suite look twenty times more certain than it is. And tag the edge cases you already know about, so sampling can never drop them.

def build(lines: list[dict], per_route: int) -> list[dict]: by_route: dict[str, list[dict]] = defaultdict(list) seen: set[tuple[str, str]] = set() for line in sorted(lines, key=lambda x: x["ts"]): key = (line["route"], line["input"].strip().lower()) if key in seen: continue seen.add(key) by_route[line["route"]].append(line)

cases = [] for route, route_lines in sorted(by_route.items()): edge = [x for x in route_lines if "edge" in x.get("tags", [])] rest = [x for x in route_lines if x not in edge] chosen = edge + rest[: max(per_route - len(edge), 0)] chosen.sort(key=lambda x: x["ts"]) for line in chosen: cases.append( { "id": line["request_id"], "route": route, "history": [ {"role": h["role"], "content": strip_pii(h["content"])} for h in line.get("history", []) ], "input": strip_pii(line["input"]), "expect": expectation(line), } ) return cases

bashCOPY ⌘C

python build_golden.py logs/production.jsonl > golden.jsonl

expectation() writes the tools production called with their arguments, or, for the JSON route, the values of the enum fields. Review the file before committing it, then freeze it. The expectations say "do what production did", and production is a baseline, not an oracle. If you are still editing cases while you compare models, you are comparing two moving things.

Here is the opinionated part. Every regression in the table above was caught by a check that makes no model call, runs in microseconds, and could be explained to a new hire in one sentence. Embedding similarity scored the priority downgrade at 0.95 and the queue change at 0.87. An LLM judge, asked which output was better, preferred the new model on 8 of 9 tickets, which is true and useless: better prose, wrong priority.

So start with the boring checks, and encode them as data rather than prose:

Required keys of the triage JSON and their allowed values (None = free text).

JSON_SCHEMA: dict[str, set[str] | None] = { "category": {"billing", "bug", "feature_request", "account", "other"}, "priority": {"low", "medium", "high", "urgent"}, "sentiment": {"negative", "neutral", "positive"}, "summary": None, "suggested_reply": None, } SUMMARY_MAX_WORDS = 20

ISO_DATE = re.compile(r"^\d{4}-\d{2}-\d{2}

quot;) ISO_CURRENCY = {"USD", "EUR", "GBP", "JPY", "CHF"}

pythonCOPY ⌘C

What the prompts promise the tools will receive. The model is supposed to

normalise before calling; these rules say whether it did.

ARG_RULES = { "get_schedule": {"date": _is_iso_date}, "days_between": {"start_date": _is_iso_date, "end_date": _is_iso_date}, "convert_currency": { "from_currency": lambda v: v in ISO_CURRENCY, "to_currency": lambda v: v in ISO_CURRENCY, }, "convert_units": {"value": _is_number}, }

Arguments the model phrases freely. A different wording is drift worth

printing, not a failure.

FREE_TEXT_ARGS = {("search_notes", "query")}

Two kinds of check are mixed in here, and it is worth keeping them straight. Baseline checks compare the candidate against what production did: same set of tools, same arguments, same enum values. Contract checks encode what the prompt promises and hold no matter what production did: the JSON parses, the date is ISO, the summary is under twenty words. The baseline checks catch change; the contract checks catch things both models might be getting wrong.

The strict JSON parse deserves its own line. Your app almost certainly strips fences before parsing, because at some point a model added them and someone fixed it in the parser. Keep that parser. But assert on the raw output too, so a model that starts, or stops, adding fences shows up as a change rather than silently depending on a workaround.

Free text gets a judge, and only free text. The suggested reply in a triage ticket has no deterministic contract beyond its length, so a pairwise judge asking "which reply is more specific" is the right tool. Run it advisory, read what it says, and do not let it gate anything until you have audited its calls by hand. Everything above it in this section is the gate.

Run every golden case through both models at temperature 0, apply every check to every output, and compare pass rates per route and per check. Not one average: a run where triage improves and the planner regresses can average out to nothing.

Nondeterminism is real even at temperature 0. The baseline model disagreed with itself on one of nine tickets between the original capture and the replay: sentiment flipped from negative to neutral on the same input, same prompt, same model. That is why the runner takes several samples per case and scores the fraction that pass, rather than one outcome. Three samples per case is a reasonable default; the pass rate then has a floor you can reason about.

Here is the real output, trimmed to the rows that matter, with one sample per case because that is what the recorded run had:

30 cases, 1 sample(s) each: claude-sonnet-5 -> claude-sonnet-5-5

route check n baseline candidate delta converter answered 1 100.0% 0.0% -100.0 REGRESSED converter arg_format 9 100.0% 100.0% +0.0 converter tool_args 9 100.0% 100.0% +0.0 converter tool_set 10 100.0% 90.0% -10.0 REGRESSED planner arg_format 11 100.0% 90.9% -9.1 REGRESSED planner tool_args 11 100.0% 90.9% -9.1 REGRESSED planner tool_set 11 100.0% 81.8% -18.2 REGRESSED triage fields 9 88.9% 66.7% -22.2 REGRESSED triage json_schema 9 100.0% 100.0% +0.0 triage json_strict 9 11.1% 100.0% +88.9 improved triage summary_length 9 100.0% 100.0% +0.0

Cases where claude-sonnet-5-5 fails a check claude-sonnet-5 passed:

converter-008 'Convert 10 kilometres to kilograms.' tool_set extra ['convert_units'] answered no text reply claude-sonnet-5: "That's not a valid conversion—kilometres measure distance while kilograms measure mass, so there's no meani..." claude-sonnet-5-5: convert_units({"value": 10, "from_unit": "km", "to_unit": "kg"})

planner-000 "What's on my calendar today?" tool_args get_schedule.date: '2026-09-08' -> 'today' arg_format get_schedule.date='today' claude-sonnet-5: get_schedule({"date": "2026-09-08"}) claude-sonnet-5-5: get_schedule({"date": "today"})

triage-003 'I was charged twice for my subscription this month and need the duplicate refunded today — this is unacceptable.' fields priority: 'urgent' -> 'high' claude-sonnet-5: '```json { "category": "billing", "priority": "urgent", "sentiment": "negative", "summary": "Custome...' claude-sonnet-5-5: '{"category":"billing","priority":"high","sentiment":"negative","summary":"Customer was charged twice for th...'

route metric claude-sonnet-5 claude-sonnet-5-5 change converter latency_ms 1568 ms 2312 ms +47% converter cost_usd $0.0057 $0.0057 -1% planner latency_ms 1481 ms 1538 ms +4% triage latency_ms 3035 ms 2071 ms -32% triage cost_usd $0.0028 $0.0020 -27%

FAIL: 6 check(s) dropped more than 2% (worst: converter/answered -100.0 points)

The threshold is a policy choice, and it has to respect the denominator. With eleven planner cases, one failure is a nine-point drop, so a two-point threshold means zero tolerance for that route; that is fine as long as you know it. With 300 cases the same threshold tolerates six. Set it per route if the routes differ in how much they matter, and write it down before you look at the numbers, because a threshold chosen after seeing the diff is a negotiation, not a gate.

Look at the n column too. The answered check ran on one case, because production answered without a tool exactly once in thirty turns. That is a hole in the golden set, not a finding about the model, and it is the kind of thing this table is for: it tells you which rows you would act on and which rows you cannot yet trust.

The run that changes outcomes is the one attached to the pull request that bumps the model string. Trigger it on the files that can move the numbers, and nothing else, because every run spends real money:

name: llm-regression

Run only when something that can move the numbers changes: the prompts and

model names (agent.py), the assertions, the golden set, or the runner.

on: pull_request: paths:

• agent.py
• checks.py
• golden.jsonl
• run_eval.py
• .github/workflows/llm-regression.yml workflow_dispatch:

permissions: contents: read

jobs: eval: runs-on: ubuntu-latest env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} steps:

• uses: actions/checkout@v7
• uses: actions/setup-python@v6 with: python-version: "3.12"
• run: pip install -e .

The baseline is the model production runs today. Its outputs only

change when the prompt or t