preprint open access

A Second Sample Is Not a Second Signal: What the One-Sample Semantic Entropy Result Measures, Why the Sampling Premium Is Mostly a Disagreement About the Baseline, and the Matched Comparison That Would Settle It

Abstract

Life of Research

Semantic entropy detects confabulations in language models by drawing several answers, grouping them by meaning and measuring how spread the meanings are. Because each extra answer costs a full generation, a line of work asks how few samples will do. A 2025 Bayesian estimator matches the standard estimator's AUROC with 53 percent of its samples and, in its authors' words, is useful "even with just one sample", a result they call counterintuitive because with one sample no comparison of meanings takes place. In this analysis the author reviews and synthesises published results to show that the one-sample result is a likelihood result, and that the wider question it raises, how much a second sample adds, has no settled answer because the literature has not fixed what the second sample is compared against. Under one protocol, ten samples beat the log-probability of a low-temperature answer by 3 to 27 AUROC points depending on model and dataset. Under another, the log-probability of the greedy answer outperforms the sampling-based measures in 13 of 18 settings. Laid side by side on TriviaQA and SVAMP, the two papers' mean discrete semantic-entropy scores differ by less than one point, while their mean single-sequence scores differ by about 17 and 27 points, so most of the disagreement about the sampling premium is a disagreement about the baseline. Correctness labels move the comparison further: two 2025 studies show that the choice of grading function can reorder methods, and one finds that a single judge leaves a standard deviation as large as typical differences between methods. The author separates the conditions under which extra samples carry information a single pass lacks (long answers, several valid answers, reasoning chains) from errors that no number of samples can expose, and proposes a matched-budget protocol that would measure the premium directly.

Pranay Mahendrakar, AI specialist

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 166 papers by Pranay Mahendrakar, or his ORCID record.

Questions about this paper

Who wrote "A Second Sample Is Not a Second Signal"?

Pranay Mahendrakar wrote "A Second Sample Is Not a Second Signal: What the One-Sample Semantic Entropy Result Measures, Why the Sampling Premium Is Mostly a Disagreement About the Baseline, and the Matched Comparison That Would Settle It", published 20 Nov 2026. Pranay Mahendrakar is an Indian AI specialist and LLM engineer based in Bengaluru, India. He is the Managing Director of SonyTech, Nodal Coordinator at IIRS-ISRO, and an instructor at Tutorials Point. His work covers large language models, natural language processing, computer vision and retrieval-augmented generation. He publishes open-access research papers and is the author of three books: Just AI With Pranay, Multiverse of AI and It's Me LLM.

Is "A Second Sample Is Not a Second Signal" free to read?

Yes. "A Second Sample Is Not a Second Signal" by Pranay Mahendrakar is open access under a Creative Commons Attribution 4.0 licence, with the full PDF available from Zenodo at https://doi.org/10.5281/zenodo.24398949. There is no paywall and no account required.

How do I cite "A Second Sample Is Not a Second Signal"?

Cite the DOI: Mahendrakar, P. (2026). A Second Sample Is Not a Second Signal: What the One-Sample Semantic Entropy Result Measures, Why the Sampling Premium Is Mostly a Disagreement About the Baseline, and the Matched Comparison That Would Settle It. Zenodo. https://doi.org/10.5281/zenodo.24398949 A BibTeX entry is provided on this page.

Related research by Pranay Mahendrakar

← All papers by Pranay Mahendrakar