A Recall Score Is Not a Use Forecast: What Factual-Memory Benchmarks Predict About Whether a Conversational Memory System Uses What It Stores, Why the Published Transfer Failure Runs Through the Harness and the Write Policy, and the Within-Harness Test That Would Settle It
Abstract
Life of Research
Long-term memory systems for conversational assistants are compared almost entirely on question-answering benchmarks that ask for a fact stated somewhere in a long history. A 2026 benchmark for what its authors call memory utilization reports that strength on those benchmarks does not reliably carry over to tasks that require combining scattered, implicit and distractor-laden evidence into a long answer. In this analysis the author reviews and synthesises published results to show that this conclusion bundles three claims that the evidence supports to very different degrees. The first, that retrieving the right evidence is necessary for good use but not sufficient, is well supported and is not new: it was already visible inside the recall benchmarks, which report correct retrieval followed by a wrong answer on 15 to 19 percent of questions, summaries retrieved with high recall that did not help, and models that recall a stated preference but fail to apply it unprompted. The second, that a system's rank on a recall benchmark does not predict its rank on use, rests on comparisons with scores the systems' developers reported for themselves. Against those, the five systems shared with the utilization benchmark largely reverse order (Spearman -0.6 by the author's arithmetic); against a shared harness maintained by the developer of one of them, MemOS, the three shared systems keep most of their order, and harnesses run by third parties place the system at the bottom of the utilization ranking low on recall as well. The third claim is implicit and is the one the evidence most undermines: a recall score is not a stable property of a system. The same system's published LoCoMo score spans 10 to 21 points across harnesses, and two harnesses using the same answering model rank four shared systems with zero rank correlation. The paper separates four places where use can fail after access succeeds, consolidates the measured values in a table, and states a within-harness transfer test as an algorithm.
About the author
Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 144 papers by Pranay Mahendrakar, or his ORCID record.
Questions about this paper
Who wrote "A Recall Score Is Not a Use Forecast"?
Pranay Mahendrakar wrote "A Recall Score Is Not a Use Forecast: What Factual-Memory Benchmarks Predict About Whether a Conversational Memory System Uses What It Stores, Why the Published Transfer Failure Runs Through the Harness and the Write Policy, and the Within-Harness Test That Would Settle It", published 3 Nov 2026. Pranay Mahendrakar is an Indian AI specialist and LLM engineer based in Bengaluru, India. He is the Managing Director of SonyTech, Nodal Coordinator at IIRS-ISRO, and an instructor at Tutorials Point. His work covers large language models, natural language processing, computer vision and retrieval-augmented generation. He publishes open-access research papers and is the author of three books: Just AI With Pranay, Multiverse of AI and It's Me LLM.
Is "A Recall Score Is Not a Use Forecast" free to read?
Yes. "A Recall Score Is Not a Use Forecast" by Pranay Mahendrakar is open access under a Creative Commons Attribution 4.0 licence, with the full PDF available from Zenodo at https://doi.org/10.5281/zenodo.23899944. There is no paywall and no account required.
How do I cite "A Recall Score Is Not a Use Forecast"?
Cite the DOI: Mahendrakar, P. (2026). A Recall Score Is Not a Use Forecast: What Factual-Memory Benchmarks Predict About Whether a Conversational Memory System Uses What It Stores, Why the Published Transfer Failure Runs Through the Harness and the Write Policy, and the Within-Harness Test That Would Settle It. Zenodo. https://doi.org/10.5281/zenodo.23899944 A BibTeX entry is provided on this page.