What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable
Abstract
A language-model agent is a model wrapped in a harness that builds its context, exposes its tools, decides when to retry and when to stop. Several 2026 studies set out to measure how much the harness matters relative to the model, holding the model and the tasks fixed, and their answers do not stack: within-model gaps of 0 to 8 points on one terminal benchmark alongside a 40-fold spread in tokens per solved task, 8.5 to 13 points on a coding subset, and up to 28 points on a question-answering benchmark. In this analysis the author reviews and synthesises published results to show that these are not noisy estimates of one quantity. A harness effect is a contrast between two systems conditional on a control set - what was held fixed and what was left free - and a readout, and each study fixed something different: native budget controls, a shared step cap, an attempt count, or nothing at all. The paper argues four points. A cap is not a spend, so a matched ceiling leaves consumption free and lets a harness win by spending. Accuracy is purchasable with attempts, time and retries, so a harness claim with no declared budget excludes no observation. Model and harness interact, so a harness main effect is a property of the model grid. And run-to-run variance is of the same order as many reported gaps. The paper consolidates the measured values in a table and a figure, separates three questions that call for three different controls, states a decision procedure as an algorithm, and names the studies that would settle what remains open.
About the author
Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 86 papers by Pranay Mahendrakar, or his ORCID record.
Questions about this paper
Who wrote "What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable"?
Pranay Mahendrakar wrote "What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable", published 12 Oct 2026. Pranay Mahendrakar is an Indian AI specialist and LLM engineer based in Bengaluru, India. He is the Managing Director of SonyTech, Nodal Coordinator at IIRS-ISRO, and an instructor at Tutorials Point. His work covers large language models, natural language processing, computer vision and retrieval-augmented generation. He publishes open-access research papers and is the author of three books: Just AI With Pranay, Multiverse of AI and It's Me LLM.
Is "What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable" free to read?
Yes. "What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable" by Pranay Mahendrakar is open access under a Creative Commons Attribution 4.0 licence, with the full PDF available from Zenodo at https://doi.org/10.5281/zenodo.23306459. There is no paywall and no account required.
How do I cite "What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable"?
Cite the DOI: Mahendrakar, P. (2026). What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable. Zenodo. https://doi.org/10.5281/zenodo.23306459 A BibTeX entry is provided on this page.