A Checklist Is Not a Simplification: Where the Published Gains of Decomposed LLM Evaluation Come From, Why a Controlled Null on Holistic Criteria Does Not Reach Instance-Specific or Verifiable Decompositions, and the Ablations a Decomposition Claim Owes

Abstract

Life of Research

Automatic evaluation with a language model as judge is often improved, in published work, by splitting the judgment into parts: sub-criteria, checklist questions, rubric items or atomic claims, each scored separately and then combined. A 2026 controlled study reports that on standard summarization and dialogue benchmarks this splitting gives no gain over a direct judge given the same human labels, and that an earlier method's reported advantage came from a regressor trained on those labels. Other work reports large gains from decomposition in instruction following, factuality and reinforcement-learning rewards. In this analysis the author reviews and synthesises published results to show that the two bodies of evidence are compatible, because "decomposition" names at least six interventions that travel together. A decomposed judge may receive supervision from human labels through its aggregator, produce finer-grained scores that change how ties are counted, receive information the direct judge lacks (a reference answer, retrieved evidence, or criteria written for the instance), make parts of the judgment checkable by an external verifier, become more consistent across evaluator models without becoming more valid, and change how calls and arithmetic are executed. The narrow claim the method was proposed for, that answering simpler questions makes the same judge better at the same judgment, is the one the controlled evidence supports least. In two of the three studies whose ablations separate information from decomposition, the information term is the larger: in one rubric study, removing the reference answer costs about four times as much correlation with human labelers as the skill-specific decomposition adds, and in a checklist study a fixed generic checklist does little while checklists written for each query by a stronger model do a great deal. Read at the resolution of its tables, the controlled null is itself dataset-dependent, with decomposition ahead by 0.03 to 0.07 Spearman on one benchmark for each of three judge models and behind by up to 0.13 on the other. The paper consolidates the measured values in a table, states an attribution procedure as an algorithm, and specifies the ablations a claim that decomposition helps should report.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 144 papers by Pranay Mahendrakar, or his ORCID record.