A Failure to Reject Is Not a Finding: Closed-Set Accuracy, Specialized Open-Set Recognition, and the Semantic-Distance Parameter the Deflationary Debate Leaves Free

Abstract

A widely cited result reports that the open-set performance of an image classifier is strongly correlated with its closed-set accuracy, and that a well-trained baseline scoring rule is competitive with specialized open-set machinery. The result is often read as showing that specialized open-set recognition adds nothing. Its authors did not claim that. They wrote that their findings gave them insufficient evidence to reject the question their title poses, which is a failure to reject a null and not a positive finding, and they introduced a new benchmark precisely because the existing ones lacked a clear definition of the semantic class whose absence the task is supposed to detect. This paper separates three claims that the phrase "a good closed-set classifier is all you need" is used to make - a correlational claim about models, a comparative claim about method rankings, and an eliminative claim about the research programme - and sets out the different falsifiers each would need. Refereed 2024 and 2025 results meet the falsifier for the comparative claim; the falsifier for the correlational claim is met only by one unrefereed preprint and one observation its own authors attribute to a confound; and no source found here defends the eliminative claim, including the authors of the result it is attributed to. It then argues that the residual empirical disagreement is not adjudicable in its current form, because the quantity that decides it is a free parameter: the semantic distance between the closed set and the unknown set, together with which of several non-equivalent benchmark constructions of "unknown" is in force. Refereed work identifies granularity and open-to-closed similarity as understudied confounders and reports that the best scoring rule depends on them; two 2025 method papers state in their own abstracts that their gains concentrate on the semantically controlled benchmark; and an object-detection benchmark reports that method rankings change when unknown objects are absent from training rather than merely unlabelled. The paper also notes that the deflationary and anti-deflationary results are not always reported on the same metric, and that the field is closing the question by taxonomic absorption rather than by settling it. No experiments are reported. What is not known is stated flatly, including that no published comparison found here holds closed-set accuracy fixed while varying the open-set mechanism, which is the comparison the correlational claim would need.

The literature search, drafting and citation verification for this paper were carried out with AI assistance under the author's direction. Every citation was machine-verified against the arXiv API and Crossref before inclusion, and every quantitative claim was read back against the cited source's own text, either its abstract or, where a claim is drawn from a paper's body, the located passage. The author is responsible for the final text and for all claims made in it.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 66 papers by Pranay Mahendrakar, or his ORCID record.