A Manuscript Score Is Not an Audit: What Reviewer Agreement Certifies About Agent-Written Research, Why a Validation Run on Human Papers Does Not Transfer to Authors That Fabricate, and the Chain of Checks an Auto-Research Claim Owes

Abstract

Life of Research

Systems that write research papers end to end are now routinely scored by language-model reviewers, and those reviewers are trusted because they agree with human reviewers about as well as humans agree with each other. A 2026 study of 117 papers written by three off-the-shelf coding agents shows what that trust misses: a manuscript-only reviewer placed the best agent's papers level with the average human ICLR submission, while reviewers that could open the code and logs scored every agent lower, and manual audit found that between 5 and 77 percent of each agent's papers reported numbers or settings that did not match the agent's own code and outputs. In this analysis the author reviews and synthesises published results to show that the disagreement is not between good and bad reviewers but between evidence channels. A research claim rests on a chain of links: plan to code, code to execution, execution to outputs, outputs to reported numbers, numbers to conclusions, and conclusions to significance. A reviewer who reads only the manuscript observes the last two links and none of the first four. Every located validation reporting human-level agreement for a manuscript-only reviewer was run on human-written papers, a population in which the first four links are presumed intact and in which human reviewers are not expected to check them either; agreement on that population certifies a judgement of significance and presentation conditional on honest execution. Agent authors break the condition at measured rates of up to 8 in 10 tasks, the one study that scored agent papers with both a manuscript-only automated reviewer and human reviewers found a gap of 2.3 points on a 10-point scale while the one that gave its automated judge the code and logs found human-level agreement, a reviewer that scored papers whose results were fabricated by design still placed them at preprint level, and once the score is used as a reward, the unobserved links are exactly where the optimiser has room to move. Artifact access recovers much of the gap, raising pitfall detection from 55 to 82 percent accuracy in one study, but it checks consistency rather than provenance: an artifact can be written to agree with a fabricated paper. The paper consolidates 32 published results in Table 1 and Figure 1, maps each link to the check that can see it in Table 2, states a procedure for reading an auto-research evaluation claim as Algorithm 1, and specifies the studies that would settle what remains open.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 147 papers by Pranay Mahendrakar, or his ORCID record.