Agreement Is Not a Criterion: What Matching Human Labels Certifies About an LLM Judge, Why a Shared Shortcut Passes the Test, and Why Every External Criterion Found So Far Validates a Use Rather Than a Judge

Abstract

An automatic judge is usually accepted on one piece of evidence: it agrees with human preference labels about as often as humans agree with each other. That number is then read as permission to use the judge for ranking models, filtering data, selecting samples and supplying reward. This paper separates three claims the ritual runs together - that the judge can stand in for the labelers on a given item population, that it tracks the quality construct the labels were meant to measure, and that decisions made with it turn out well in a stated use - and argues that agreement evidence bears directly only on the first. A stylised two-loading model makes the reason exact: agreement with a single reference identifies one linear combination of a judge's sensitivity to the construct and to the surface features the labelers also respond to, so a judge that shares the labelers' shortcuts can match a judge that tracks quality. The published record is then read through that separation. Human labels carry documented surface sensitivities that judges reproduce; judges carry failures of their own that natural validation sets never exercise; and agreement is concentrated on the easy, wide-gap items where decisions matter least. In the corners where an external criterion exists - verified correctness, downstream policy outcomes, randomised business outcomes and verifiable task success - the criterion has repeatedly exposed failures that agreement did not show, and where it confirms an offline score the confirmation does not transfer across uses: a benchmark score that predicts best-of-N selection well gives only a rough signal for PPO. Validity, on this evidence, is a property of a judge in a use. The paper consolidates these published results, states a use-indexed validation procedure as an algorithm, and proposes the tests that would settle what remains open.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 76 papers by Pranay Mahendrakar, or his ORCID record.