The Self-Verification Gap: Why Working Hallucination Detectors Are External, and What Real-Time Correction Would Actually Require

Abstract

Large language models generate fluent text that is sometimes false, and a substantial literature now proposes to have the model notice and repair those errors while it writes. Parts of the problem are settled. Semantic entropy detects confabulations across datasets and tasks without prior knowledge of the task (Farquhar et al., 2024). Search-augmented verification agrees with crowdsourced annotators 72% of the time on roughly 16,000 individual facts, at more than 20 times lower cost (Wei et al., 2024). Probes on hidden states separate true from false statements at 71-83% balanced accuracy in distribution (Azaria and Mitchell, 2023). The tension is that every one of these signals is external to anything the model reports about itself, and that the four properties a deployed detector needs - low cost, transfer under distribution shift, span-level localisation, and a false-positive rate low enough to act on - have never been demonstrated together in a single system. This paper states five constraints such a system must satisfy, argues that no published system satisfies more than two, and proposes five experiments that would settle the open questions. Every figure is credited to the study that reported it, and the proposed experiments specify the evidence that would show whether such a system can be built.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 72 papers by Pranay Mahendrakar, or his ORCID record.