Is your LLM judge flaky?

Re-run any evaluator across runs, judge models and rubric wordings, and get a reliability card.

When the verdict can't be trusted, declare a mistrial.

pip install mistrial export OPENAI_API_KEY=... # or ANTHROPIC_API_KEY, GEMINI_API_KEY, GROQ_API_KEY ... see mistrial providers mistrial run cases.jsonl -j openai:gpt-5.4-mini -j openai:gpt-5.5 --runs 5

No key? pipx run mistrial demo runs three fake judges that are built to be flaky on purpose.

You ship AI features based on a score from an LLM judge. The score gates CI, decides which prompt wins, and goes on the dashboard. But the judge is a model sampling from a distribution. Ask it the same question twice and it changes its verdict on some of your cases. Swap the model and it changes its mind on more. Reword the rubric without changing its meaning and it moves again. One eval run is one sample from a noisy process, and almost nobody measures the noise.

These are real replies. On the first case, gpt-5.4-nano wrote "the added mention of 1811 does not contradict the core requirement" on one run and "1811 is not part of the reference, so the answer is incorrect" on the next. On the second, it worked out that 239,000 miles is 384,000 km and then failed the answer for being in miles. Each of those judges reports a clean score on its own. The flakiness only shows up when you ask again and compare.

The research agrees. The same judge gives different scores to the same answer across runs (Rating Roulette, EMNLP 2025). Raw percent agreement overstates reliability by 30 to 40 points compared with chance-corrected kappa, across 21 judges (Reliability without Validity, 2026).

Eval platforms report one number and stop. LangSmith, Braintrust, Langfuse and friends run the judge once per case. Some can repeat an experiment and show a standard deviation, none report self-agreement, kappa between judges, or which cases flipped.

The statistics exist, but not as a tool. Test-retest reliability, Cohen's kappa and Krippendorff's alpha are a century old and standard in every field that uses human raters. They live in research notebooks, not in something you can run against your evaluator in a minute.

The projects that tried stop at the card. judgecheck, judgebench, Assay rejudge and judge-drift-sentinel each cover a piece of this well, and docs/COMPARISON.md credits what each does better. None of them take your existing evaluator function, run in CI, or tell you what to do when the judge is flaky.

Before "how good is my model?" you need "can I trust the number that told me?"

Point it at the evaluator you already use: a model name, or the function you run in LangSmith or openevals. It asks the judge several times, across one or more judge models, and under rewordings of the rubric, and reports the spread: self-agreement, flipped cases, kappa and alpha between judges, accuracy against your human labels, and the exact cases that move. Then it does something about it: a gate for CI, a stabilized judge with the same signature, and a list of the cases a human should look at.

The 90 second version. Every number in it is from the runs below.

explainer.mp4

Real numbers

Three current OpenAI judges on examples/tricky.jsonl, 32 cases built to be hard to judge (hedges, rounding, unit conversions, extra details, equivalent code), five runs each. 480 live calls on 25 October 2026. Every number below is from that run, and the GIF replays those replies.

mistrial · 32 cases · 3 judges · 5 runs each · 480 verdicts in 76.66s

Consistency (same judge, asked again) ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ ┃ judge ┃ self-agreement ┃ flipped cases ┃ vs human ┃ errors ┃ reading ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │ openai:gpt-5.4-nano │ 92% │ 5/32 (16%) │ 81% (κ 0.63) │ 0 │ noticeably flaky │ │ openai:gpt-5.4-mini │ 95% │ 4/32 (12%) │ 88% (κ 0.75) │ 0 │ mostly stable │ │ openai:gpt-5.5 │ 100% │ 0/32 (0%) │ 91% (κ 0.81) │ 0 │ stable │ └─────────────────────┴────────────────┴───────────────┴───────────────┴────────┴──────────────────┘

Flakiest cases (everyone's verdicts pooled) ┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━┓ ┃ case ┃ disagreement ┃ openai:gpt-5.4-nano ┃ openai:gpt-5.4-mini ┃ openai:gpt-5.5 ┃ human ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━┩ │ arigato │ 40% │ F F F F F │ F T T T T │ T T T T T │ T │ │ austen-wrong-year │ 40% │ T F F F T │ T T T T F │ F F F F F │ F │ │ largest-country-hedge │ 40% │ F F F F F │ T F F F F │ T T T T T │ F │ │ moon-miles │ 33% │ F F F F F │ F F F F F │ T T T T T │ T │ │ percent-unit │ 33% │ T T T T T │ F F F F F │ F F F F F │ F │ └───────────────────────┴──────────────┴─────────────────────┴─────────────────────┴────────────────┴───────┘

What the run says, in order:

On the 32 tricky cases, gpt-5.5 and the three gpt-6 judges never flip. So we wrote examples/hard.jsonl: 64 cases that pit precision against reasonableness. Rounding that is close enough or not, unit conversions, hedges that commit to the right answer and hedges that don't, extra facts that are right and extra facts that are wrong, equivalent code, multi-step answers with a wrong step and a right total. Six judges, five runs each, 1,920 live verdicts on 26 October 2026.

Every gpt-6 judge flipped, and the rationales show what they flip on:

The pattern: weak judges flip everywhere, strong judges flip exactly where a reasonable person could go either way, at 1 to 3% of cases. On an eval of a few hundred cases that is still several verdicts that depend on which run you got. And the bigger number is between tiers: Avogadro without "per mole" passes on every gpt-5 judge and fails on every gpt-6 judge; "32 degrees Fahrenheit" for the freezing point fails on both 5.4 judges and passes on everything above. Kappa between gpt-5.4-nano and gpt-6.1-sol is 0.60. Krippendorff's alpha across all six is 0.81, across the three gpt-6 judges 0.97.

One more thing the run caught: OpenAI's moderation rejected one of gpt-6-luna's 320 calls, on a question about sum(xs) / len(xs), as a flagged prompt. The parser audit recorded it and marked that judge "fragile". A pipeline without error accounting would have counted that as a fail or skipped it silently.

Both sets ship with the package. examples/tricky.jsonl is where cheap judges fall apart; examples/hard.jsonl is where the expensive ones do.

You get the card in the terminal, a single-file report.html you can send to anyone, JSON for machines, and markdown for a pull request.

Every run is saved as JSON under .mistrial/runs/ and every response is cached on disk, so the probes below can read a saved run, re-rendering is free, and --fresh is explicit. Beyond the card: the probes for the ways judges go wrong in production, the fixes for the flakiness they find, and the CI setup to keep it that way.

Cases file

One case per line. JSONL, JSON or CSV.

{"id": "capital-france", "inputs": "What is the capital of France?", "outputs": "Paris.", "reference_outputs": "Paris", "label": true} {"id": "seven-times-eight", "inputs": "What is 7 × 8?", "outputs": "54", "reference_outputs": "56", "label": false}

inputs and outputs are required. reference_outputs and the human label are optional but unlock the correctness rubric and the accuracy column. Common aliases work: input, question, answer, expected, ground_truth.

Three kinds, mixed freely with repeated --judge:

All LLM judges use the same built-in rubric and ask for a JSON verdict. mistrial providers lists the providers with their env vars and base URLs; <PROVIDER>_BASE_URL overrides a base URL, which is how you point ollama: at another machine or openai: at a proxy.

GPT vs Claude vs a local model, five runs each

mistrial run cases.jsonl --judge openai:gpt-5.4-mini --judge anthropic:claude-sonnet-5-5 --judge ollama:llama3.2 --runs 5

Your own evaluator

Any function that takes inputs, outputs, reference_outputs and returns a score, a bool, or {"score": ...}:

my_evals.py

def correctness(inputs, outputs, reference_outputs): return {"key": "correctness", "score": outputs.strip() == reference_outputs.strip()}

mistrial run cases.jsonl --judge my_evals:correctness --runs 5

openevals and LangSmith

create_llm_as_judge returns exactly that kind of function, so an openevals judge needs no adapter:

my_evals.py

from openevals.llm import create_llm_as_judge from openevals.prompts import CORRECTNESS_PROMPT judge = create_llm_as_judge(prompt=CORRECTNESS_PROMPT, model="openai:gpt-5.4-mini")

mistrial run cases.jsonl --judge my_evals:judge --judge openai:gpt-5.4-mini --runs 5

Any LangSmith custom evaluator with the same signature works the same way. Export your dataset as JSONL and point at it.

--rubric correctness (boolean, needs reference_outputs) and --rubric quality (1 to 5, no reference). Or --rubric my_prompt.txt with {inputs}, {outputs}, {reference_outputs} placeholders; the model must reply with JSON {"score": ..., "reasoning": ...}. Run mistrial rubrics to list them.

Each probe answers one way judges go wrong in production. Most read a saved run, so they cost nothing after the run that produced it. Everything below works on the demo judges too, so you can try each one before spending a token: swap openai:gpt-5.4-mini for demo:flaky.

mistrial run examples/tricky.jsonl --judge openai:gpt-5.4-mini --runs 5 --pass-mark 0.5

or on a saved run

mistrial gate-risk .mistrial/runs/20261025-101500.json --pass-mark 0.5 --runs-in-ci 1 --max-false-alarm 0.05

Gate risk (pass mark 0.5, 1 run per case in CI) ┃ judge ┃ expected score ┃ 90% range ┃ false alarm ┃ safe mark ┃ fix ┃ │ openai:gpt-5.4-nano │ 0.38 │ 0.34–0.44 │ 100% │ 0.34 │ already below the mark on average │ │ openai:gpt-5.4-mini │ 0.46 │ 0.41–0.50 │ 90% │ 0.41 │ already below the mark on average │ │ openai:gpt-5.5 │ 0.50 │ 0.50–0.50 │ 0% │ 0.50 │ fine as is │

Each case's observed verdicts are treated as its distribution. The gate score is re-sampled 20,000 times and "false alarm" is how often it lands below your pass mark with nothing changed. The fix column gives the pass mark the judge clears 95% of the time, or the runs-per-case majority vote that gets the alarm rate under 5%. Pure Monte Carlo, so it is right for 10 cases as well as 1,000.

mistrial rubric-check examples/tricky.jsonl --judge openai:gpt-5.4-nano --judge openai:gpt-5.5 --rubric correctness

Rubric sensitivity (same meaning, different words) ┃ judge ┃ cases ┃ verdicts that changed ┃ alpha across wordings ┃ per wording vs v0 ┃ reading ┃ │ openai:gpt-5.4-nano │ 32 │ 5 (16%) │ 0.81 │ v1 12%, v2 12%, v3 6% │ measures the wording │ │ openai:gpt-5.5 │ 32 │ 0 (0%) │ 1.00 │ v1 0%, v2 0%, v3 0% │ robust │

Rubric reversal (ask for INCORRECT instead of correct) ┃ judge ┃ cases ┃ verdicts inverted ┃ unchanged ┃ reading ┃ │ openai:gpt-5.4-nano │ 32 │ 28 (88%) │ 4 │ mostly reads the rubric │ │ openai:gpt-5.5 │ 32 │ 32 (100%) │ 0 │ reads the rubric │

The built-in rubrics ship with three rewordings that mean the same thing and a reversed form. The judge runs under each; changed is the share of cases whose verdict depends on which wording you used, and inverted is the share that flipped when the criterion flipped. Bring your own wordings with --variant my_rubric_v2.txt. Gates: --max-changed 0.10, --min-inversion 0.90. Function judges have no rubric, so they are skipped.

mistrial run examples/tricky.jsonl --judge openai:gpt-5.4-nano --judge openai:gpt-5.5 --runs 3 mistrial migrate .mistrial/runs/20261025-101500.json --pass-mark 0.5

agree on 81% of cases · kappa 0.62 (substantial) · shift +0.12 your pass mark 0.5 on openai:gpt-5.4-nano is about 0.62 on openai:gpt-5.5 6 cases get a different verdict: arigato: F → T moon-miles: F → T percent-unit: T → F …

Both judges on the