Automatic Benchmarks Measure an Asymmetry: Privileged Information, Panel-Relative Difficulty, and Two Self-Biases That Pull in Opposite Directions

Abstract

Automatic benchmark construction is presented as a response to saturation: a language model proposes topics, writes items, and supplies answer keys, and the resulting datasets are reported to be harder and more novel than human-built ones. The standard objection is that a model cannot examine above its own ceiling and that its benchmarks will flatter it. This paper argues that both halves of that objection are aimed slightly off target, and that the published record supports a sharper and less comfortable reading. Three points are developed. First, privileged information does two separable jobs in these pipelines - supplying a trustworthy key, and creating an access gap between the evaluator and the candidates. The anchor paper states both, and treats the resources that discharge them as interchangeable instances of one principle. They are not interchangeable, because they differ in whether the asymmetry survives a change in the candidate's deployment configuration: where the gap is a withheld interpreter rather than a withheld corpus, the measured difficulty is a fact about what the candidate was permitted to use. Second, difficulty in this literature is defined as one minus the best accuracy over a chosen model panel, so it is a relation between a dataset and a panel; the same construction appears in expert-written frontier benchmarks, which filter submissions by whether frontier models fail them, so panel-relativity is a property of the objective rather than of the generator. Optimising that objective also concentrates the items on which the answer key is least secure, which is visible both as a negative association between invalidity and discrimination in generated suites and as a 15.4 percent expert disagreement rate in a frontier benchmark written and audited by domain experts. Third, "self-bias" names two mechanisms that have opposite signs and opposite predicted trends in generator capability: generation-side bias, whose largest published component is the generator repeating its own labelling errors at test time, and selection-side penalty, under which items chosen because a panel fails them disproportionately penalise that panel. A pipeline whose generator also sits in the panel driving item selection contains both, and the net sign has not been measured. No experiments are reported here. Six studies that would settle the open parts are named, and the case against this paper's own position is stated in full.

The literature search, drafting and citation verification for this paper were carried out with AI assistance under the author's direction. Every citation was machine-verified against the arXiv API and Crossref before inclusion, and every quantitative claim was read back against the cited source's own text. The author is responsible for the final text and for all claims made in it.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 23 papers by Pranay Mahendrakar, or his ORCID record.