preprint open access

Problem Choice Without a Referee: An Automated Novelty Check Certifies an Empty Search, a Rediscovery Benchmark Credits the Hypothesis Humans Found Next, and Every Located Referee of Worth Independent of Proposer and Field Needs the Objective Given

Abstract

Life of Research

(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0.

Pipelines that claim to automate scientific discovery gate their search on a judgement that a proposed idea or problem is novel and worth pursuing. The best-controlled evidence on that judgement points two ways at once: in a blind study involving more than one hundred NLP researchers, expert reviewers rated language-model research ideas as more novel than expert-written ones, and when forty-three researchers executed randomly assigned ideas from the same study, the model ideas lost that advantage and fell further than expert ideas on every metric, novelty included. This paper asks what, if anything, can referee a proposed research problem before the field has worked on it. It distinguishes four kinds of referee: an absence check (was a match retrieved?), a judgement before execution (does a reader rate it novel or promising?), uptake (did humans later publish it?), and outcome (did executing it work against an objective fixed in advance?). Assembling published measurements for each, it argues that the absence check fails mainly at retrieval, with comparison a smaller and less-measured source of error, and is benchmarkable without labels; that model and expert judges disagree about which questions are non-obvious, and human judges have documented biases of their own, including lower merit scores for highly novel proposals; that uptake benchmarks supply the research background and score the rediscovered hypothesis, so by construction they credit only what humans went on to publish, as several of their authors concede; and that, of the referees located, the only one that scores worth and is independent of both the proposer and the field's present and future judgement is the outcome referee, which requires the objective to be supplied. Choosing the objective is what problem discovery means, so at that step the only referees of worth on offer are the field's own judgement, now or later. That conclusion is partly definitional, and the observation that current systems take the research question as given has been made before; the contribution is the decomposition, the measurements that locate each referee's failure, the finding that the absence check can be audited without labels, and a refutation condition that a domain-general criterion of problem worth could meet. The claim is scoped to research-idea and research-question generation; it does not say that such systems cannot discover anything.

Pranay Mahendrakar, AI specialist

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 161 papers by Pranay Mahendrakar, or his ORCID record.

Questions about this paper

Who wrote "Problem Choice Without a Referee"?

Pranay Mahendrakar wrote "Problem Choice Without a Referee: An Automated Novelty Check Certifies an Empty Search, a Rediscovery Benchmark Credits the Hypothesis Humans Found Next, and Every Located Referee of Worth Independent of Proposer and Field Needs the Objective Given", published 2 Oct 2026. Pranay Mahendrakar is an Indian AI specialist and LLM engineer based in Bengaluru, India. He is the Managing Director of SonyTech, Nodal Coordinator at IIRS-ISRO, and an instructor at Tutorials Point. His work covers large language models, natural language processing, computer vision and retrieval-augmented generation. He publishes open-access research papers and is the author of three books: Just AI With Pranay, Multiverse of AI and It's Me LLM.

Is "Problem Choice Without a Referee" free to read?

Yes. "Problem Choice Without a Referee" by Pranay Mahendrakar is open access under a Creative Commons Attribution 4.0 licence, with the full PDF available from Zenodo at https://doi.org/10.5281/zenodo.23093043. There is no paywall and no account required.

How do I cite "Problem Choice Without a Referee"?

Cite the DOI: Mahendrakar, P. (2026). Problem Choice Without a Referee: An Automated Novelty Check Certifies an Empty Search, a Rediscovery Benchmark Credits the Hypothesis Humans Found Next, and Every Located Referee of Worth Independent of Proposer and Field Needs the Objective Given. Zenodo. https://doi.org/10.5281/zenodo.23093043 A BibTeX entry is provided on this page.

Related research by Pranay Mahendrakar

← All papers by Pranay Mahendrakar