Calibrated Is Not Informative: What a Fall in Expected Calibration Error Shows About LLM Confidence, Why Verbalised Scores Fail on Resolution as Well as Level, and the Recalibration Baseline the Ensembling Results Omit

Abstract

Life of Research

A common finding in work on language-model confidence is that a model asked to state its own confidence in one pass is badly overconfident, and that some added procedure - sampling several answers, a separate critique pass, a longer chain of thought - repairs it. The repair is usually scored by expected calibration error (ECE). A 2026 telecommunications study is a clear recent case: it reports that averaging several blind self-evaluations of a reasoning chain cuts ECE by up to 88 percent on three telecom benchmarks, and it reports no discrimination metric. In this analysis the author reviews and synthesises published results to show that such claims mix two failures that need different fixes. One is level: the scores sit too high on average. The other is resolution: the scores barely separate right answers from wrong ones. ECE mostly measures level. It can reach zero for a constant score that separates nothing, and a monotone map fitted on held-out labels can remove most level error without changing the ranking at all. In the published studies that report both ECE and a ranking metric, the two move independently. Better prompts cut ECE while, in the authors' summary, AUROC stayed close to chance. Verbalised scores cut ECE on one benchmark by three quarters while the area under the selective accuracy curve rose by about one point. Sampling several answers raised AUROC from 54.8 to 92.7 on arithmetic but by one point on a yes-no reasoning set, and a separate self-evaluation pass, the mechanism the telecom method repeats, was at chance on the one multiple-choice set where it was measured alone. Re-reading the telecom study's own tables, the author finds that its top confidence tier was about three points more accurate than the pooled base rate, and that its largest ECE reduction comes from a 150-item subset, where binning noise alone is close to the reported post-method value. The paper gives a decision procedure and five tests that would separate what added inference passes buy from what a cheap recalibration already buys.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 110 papers by Pranay Mahendrakar, or his ORCID record.