A Hard-Task Gain Is Not a Change of Shape: When Language-Model Progress Is One Number, Why a Difficulty-Specific Gain Is Defined Only Relative to a Scale, a Difficulty Coordinate, an Overlap in Ability and a System Boundary, and What a Claim of Non-Scalar Progress Has to Carry

Abstract

Life of Research

Progress in language models is usually reported as one number: a time horizon, a latent ability on a stitched scale, or an average benchmark score. Whether one number is enough is now disputed. Cross-sectional analyses find that three principal components explain about 97 percent of the variance in a model-by-benchmark matrix and that a single stitched capability predicts most benchmark scores well, and a July 2026 study shows that a one-dimensional Rasch model with rising ability reproduces the apparent migration of gains toward harder agentic tasks. The same study then reports a residual gain on the hardest competitive-programming problems of about +0.40 logits that the scalar model does not absorb, while other work documents crossovers, rank reversals and capability dimensions that progress at different rates. In this analysis the author reviews and synthesises published results to show that most of this disagreement is about the reference against which "shape" is measured, not about the models. A difficulty-specific gain is defined only relative to four choices: the response scale, because an interaction that keeps its sign can be removed by a monotone rescaling and only a crossover cannot; the difficulty coordinate, because difficulty estimated from the models being compared manufactures patterns that exogenous labels do not; the range in which the groups overlap in ability, outside which a functional form rather than the data carries the claim; and the system boundary, because scaffolds, inference budgets and model eras move together. Read against these four, the published record supports a narrower statement than either side makes: progress is close to scalar within one population, scale and boundary; the best-identified hard-item residual is real, scale-relative, and identified without a functional form only in the lower-ability range where pre- and post-era models overlap; and the clearest non-removable departures from one number appear across skill dimensions and between system configurations, including one crossover along puzzle complexity between reasoning and non-reasoning models at matched compute, rather than in era-on-era comparisons of deployed models along difficulty. The paper consolidates 41 published results in a table and a figure, states a decision procedure for reading a claim of non-scalar progress, and specifies six measurements that would settle the open part.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 144 papers by Pranay Mahendrakar, or his ORCID record.