The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale

By Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Ratna Kavuri, Rohan Gosain, Veli Balin, Minsu Jang, Louis Garcia Introduction Any team generating text with an LLM at scale hits the same problem. You cannot evaluate everything manually. You don't know which outputs are great, which are mediocre, and which are harmful, because you simply cannot read them all. The standard solution is LLM-as-a-Judge, a second model that scores the first model’s output as a proxy for human raters. It is fast, cheap, and easy to deploy. However, it introduces a new and critical question: Who evaluates the evaluator? Furthermore, if the judge drifts, quietly degrading and starting to approve bad output, nothing downstream will flag it. This post is about how we solve that problem, by building, aligning, and continuously monitoring LLM judges so they stay trustworthy over time. Our core principle is simple: An LLM judge must never operate alone. It has to be anchored in human judgment. We keep a human-in-the-loop (HITL) at every phase of the lifecycle. Humans define what “good” means, create the labeled benchmark datasets to train and validate the judge, monitor its behavior while it runs in the system, and review and approve any updates before they’re rolled out. We believe: A reliable LLM judge is not a static artifact. It has a lifecycle. It needs to be built against trusted human labels, aligned with human preference and reasoning, run under human-set policy, and maintained as the data and usage pattern shift. Each of these stages has human owners. In the sections below, we walk through these four phases, the key design decisions in each, and the lessons we’d share with a team building their own LLM judge, using one concrete task as our running example: recommendation explanations. A recommendation explanation is a short natural-language message that accompanies a recommended item, and it shapes how users perceive and trust a recommender system. In our work, an explanation is human-readable evidence for why a title was recommended based on what the member has previously watched, shown on the title’s detail page, which is the screen a member reaches by selecting a title from the homepage. We focus on similarity-based explanations, which connect a recommended title to one reference title the member has interacted with via shared attributes such as genre and tone. For example, A funny, heartfelt holiday romance about love and new beginnings, much like “My Secret Santa.” Quality matters here because a misleading or poorly grounded explanation erodes the very trust the explanation is meant to build. Everything that follows comes out of a series of controlled, user-facing online experiments we ran on the explanation feature, including a month-long experiment on the Netflix mobile app. In these tests we used an LLM to generate explanations, hundreds of thousands a week across a constantly changing catalog. Manually reviewing them at this scale isn’t feasible, so we deployed a second LLM to evaluate them. What counts as a good explanation is specific to our task. However, the way we build, align, and monitor the LLM judge for evaluating explanations should be reusable and generalizable across different tasks. The Lifecycle at a Glance Figure 1 summarizes the four phases of our LLM-judge lifecycle and the data flow connecting them. Figure 1. The four-phase lifecycle of an LLM judge, and the data that flows between the phases. We start with Birth: Benchmark Data Creation, where human experts craft hard or near-boundary examples and human raters label them with pass/fail decisions and written rationales. We augment this with LLM-generated gray-area examples to build a richer benchmark dataset. Next, in the second phase of Training: LLM Judge Development, we tune the LLM judge to align with human labels and rationales from the Birth phase, using self-reflection to identify mismatches between the judge and humans, and updating the rubrics based on those mismatches and anchor examples. We iterate in this phase until our metrics show that the judge is sufficiently aligned with human raters. In the third phase, Deployment, we integrate the judge into the explanation generation system as both a gate and a critic. The LLM generator writes explanations, and the LLM judge scores them and either passes them through or blocks them and supplies a rationale that the generator can use for self-refinement and retries. Passed explanations are served to members, and low-quality explanations are gated. Finally, in the last phase, HITL Alignment & Drift Monitoring, we keep the judge aligned with humans over time. “Human-in-the-loop” here means a person is directly involved in the decision, not just reviewing outcomes after the fact. We benchmark the judge against trained raters, and a human approves every change before it is rolled out to the system. During each test we sampled a set of explanations each week (both passed and gated, with a bias toward newer items), had human raters review them with labels and rationales, and compared the judge against these human ratings. If drift exceeded a threshold, we triggered retraining and benchmark data augmentation. This closes the loop by feeding new human-labeled data and updated signals back into the benchmark and development phases. System components Four automated components appear across all phases, alongside the human roles that define, validate, and supervise them.

• Generator: produces the explanation text, optionally using prior judge rationales and traces as part of its prompt for revision.
• Judge: evaluates an explanation and returns a pass/fail label together with a written rationale. In the deployed pipeline, it acts as both a guardrail (gating bad outputs) and a critic (providing feedback for self-refinement).
• Reflector: revises the Judge’s rubric when the Judge makes mistakes. It updates the rubric based on mismatched examples, human rationales, and anchor examples. It tunes the rubric and does not directly evaluate explanations.
• Meta-Judge: checks whether the Judge failed an explanation for the right reason by comparing the Judge’s rationale against a human rater’s rationale.
• Human experts and raters: the experts who define the quality criteria; the raters whose labels and written rationales the Judge is tuned and measured against; the reviewers who monitor the Judge while it is running and explicitly approve any changes before they were rolled out. Phase I, Birth: Establishing Ground Truth Everything downstream depends on knowing what “good” means, so this is the most human-intensive phase. Our internal writing experts define the must-have criteria an explanation must satisfy, author detailed labeling guidelines, and hand-craft tricky adversarial examples. Those same guidelines do double duty. They’re both the rulebook for human raters and the seed rubric for the LLM judges we tune later. Why we built our own benchmark Off-the-shelf LLM-judge benchmarks focus on general-purpose text evaluation and miss what makes our task specific. Explanations are short, item-specific, and context-dependent (each one is tied to a target title and one or two reference titles the member has watched). We also care about alignment to our own editorial bar, not just generic language quality. As a result, we built our own custom benchmark from three complementary sources:
• Expert-crafted adversarial cases: hand-authored pass/fail examples with known failure modes, the kind of edge cases nothing else reliably produces.
• Boundary examples: LLM-synthesized explanations that are deliberately pushed near the pass/fail boundary, then rated by humans. These capture the ambiguous cases that sampling from live traffic rarely surfaces on its own.
• Real sampled explanations: explanations sampled from the output of our LLM generator during test runs. Two details that mattered
• The benchmark is not frozen. A monitoring pipeline (Phase IV) appends a fresh batch of human-rated explanations on a weekly cadence, so over the course of a test the benchmark tracked the current catalog and new usage patterns instead of going stale.
• Every “fail” label carries a free-text rationale explaining why the explanation failed. This looks like a small bookkeeping detail, but it is actually critical for the next phase, since those rationales are what we use to tune and audit the judge’s reasoning. Phase II, Training: Aligning the Judge to Humans The failure mode: right answer, wrong reason There is a failure mode that does not show up in label-agreement metrics at all. A judge can agree with humans on the label, passing when they would pass and failing when they would fail, while reaching the right verdict for the wrong reason. On a dashboard it looks perfectly aligned. But as soon as you ask it to explain a rejection, which our pipeline does to steer self-revision, it can return misleading guidance. We therefore care about not just label agreement but also reasoning alignment. In our setup, the Judge is always asked to output:
• a label (e.g., PASS/FAIL), and
• a short natural-language rationale for that label. Here is an illustrative example. The member watched a musical, a family fantasy with a young heroine and a well-known ensemble cast. We recommend an adventure movie that is also a family fantasy with a young heroine, which is why the recommender paired them. But it shares neither the songs nor the ensemble cast. The label (FAIL) is correct, and the stated reason is reasonable at first glance. But it turns out to be incomplete in a way that simple label agreement cannot detect, as we show below. Reasoning-Aligned Rubric Tuning We tune each criterion’s LLM Judge with a procedure we call Reasoning-Aligned Rubric Tuning (RART). As introduced in the system components above, the Judge, Meta-Judge, and Reflector are distinct components. The Judge scores an explanation against a rubric and gives a rationale, the Meta-Judge compares that rationale against a human-written one, and the Reflector edits the rubric text itself. RART is how the three are combined into a training loop. Every Judge is a fixed prompt template parameterized by a rubric, so tuning it does not mean retraining the underlying model. Instead we iteratively refine the rubric text until the Judge’s behavior matches human labels and human rationales. At iteration i, the Judge is conditioned on the current rubric text and maps an explanation to a pair, a label and a rationale. R_i is the current rubric text, y_J(x) the label, and r_J(x) the rationale. At a high level, each RART iteration proceeds as follows.
• Score the training data with the current rubric.
• Evaluate weighted alignment metrics on a held-out validation set. If all per-criterion metrics exceed their thresholds, stop.
• Otherwise, construct an error-driven focus set of misaligned examples and hand it to the Reflector, which proposes a revised rubric. The key difference from a standard loop is how we define a “mistake.” Most self-improvement loops look only at label disagreements. RART adds a second class of error. The Meta-Judge examines the cases where the Judge and a human both said fail and compares the Judge’s stated reason against the human’s rationale. When the two diverge, that is a right answer, wrong reason error, and it is also fed to the Reflector. Concretely, the focus set handed to the Reflector is the union of two error classes:
• Label mismatches: examples where the Judge’s label differs from the human’s.
• Right label, wrong reason: examples where the Judge and the human both said FAIL, but the Meta-Judge finds their two rationales don’t match. So the Reflector revises the rubric from both ordinary label errors and right label, wrong reason errors. That second term is the core of RART. The Judge is pushed not only to match human decisions but to justify them in a way the Meta-Judge recognizes as aligned with human rea