The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations

A Netflix case study arguing that an LLM judge running in production is best understood as a lifecycle, not a static artifact. The judge evaluates user-facing recommendation explanations — hundreds of thousands per week, served to millions of members — and must be built, trained, deployed, and continuously maintained as the catalog and user base drift.

The framework has four phases. Birth builds a domain-specific, human-labeled benchmark (~900 examples at launch, near-balanced, each fail label carrying a free-text rationale). Training refines each criterion's rubric via Reasoning-Aligned Rubric Tuning (RART), which uses a meta-judge to score the judge's reasoning against human rationales on agreed-fail examples — a right-verdict-but-wrong-reason case is treated as an error because the same reason is later fed to the generator as a revision instruction. Deployment puts one judge in two roles: a quality gate and the critic in a generate→judge→revise loop with a bounded retry budget (drop, don't serve, flagged explanations). Monitoring runs a weekly 300-example human-in-the-loop sample, requiring the judge to stay within two standard deviations of the average human rater (a relative, difficulty-adaptive threshold); drift triggers re-tuning behind a human review gate.

A five-week A/B test over tens of millions of members found judge-aligned explanations shifted viewing toward novel content and increased successful browse-to-play sessions versus a no-explanation control, with no quality-related takedowns. Takeaways: invest in a rationale-annotated benchmark before the judge, reuse one tuned judge across roles, and plan monitoring from day one.