A Blind Score Is Not a Reliance Measure: What Answering Without the Audio Shows About Audio-Language Benchmarks, Why the Partial-Input Debate Already Settled Half of It, and the Counterfactual-Audio Test the Other Half Needs

Abstract

Audio-language models are now routinely tested with the audio taken away, and the results are read in opposite directions. One 2026 audit reports that eight models keep 60 to 72 percent of their accuracy on three benchmarks when the audio is omitted, and concludes that the scores largely reflect text priors; the authors of one of those benchmarks replaced the audio with noise, saw accuracy fall, and concluded that the models are listening. In this analysis the author reviews and synthesises published results to show that both readings are correct and that they are about different quantities. A blind score estimates how answerable the items are without the audio, for one answerer under one replacement of the audio. It does not measure whether the full model's correct answers depend on the audio, and it does not settle whether the benchmark measures what it claims, which depends on whether its stated construct includes the knowledge that answers the item. Natural language inference research learned the first two points about hypothesis-only baselines between 2018 and 2022; multimodal evaluation is re-running that debate without citing its resolution. Re-expressing the 2026 audit's own table, the author shows that correcting for guessing lowers the retained share to 29 to 45 percent, that the blind score on one benchmark ranges from 8 to 72 percent of the above-chance score across models, and that ranking models by what the audio added is unrelated to ranking them by score on one benchmark (rho = -0.24) but not on the other two. The paper gives a decision procedure for what a blind result licenses and specifies the paired counterfactual-audio test that would identify reliance item by item.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 92 papers by Pranay Mahendrakar, or his ORCID record.