A Clean Answer Is Not a Clean Agent: What an Unlearning Certificate Covers Once the Model Calls Tools, Why the Agentic Leaks Divide Into Residue, Scope, Reacquisition and Exposure, and the Source-by-Observer Audit That Would Assign Each to Its Owner

Abstract

Life of Research

Unlearning benchmarks judge forgetting by questioning a bare model and reading its answer. A September 2026 benchmark argues that a certificate issued this way does not survive deployment: wrapped in a tool-using agent, the same models expose the target through reasoning text, tool arguments, tool observations and elicited summaries, and when the target sits in the prompt or a retrieval store the agent discloses it on 22 to 86 percent of queries while weight-level probes report nothing. Concurrent studies report that tool access lets an unlearned model fetch what it no longer recalls, and that adding memory or tools can reorder which unlearning method looks best. In this analysis the author reviews and synthesises published results to show that these findings are four different failures that share one name. The first is residue: the target is still in the weights and the agent poses a different question, which the non-removal literature had already shown and which strong model-level probes in the same benchmark's weight case study already detect. The second is scope: the target was never in the weights, so no weight method could clear it, exact retraining included. The third is reacquisition: the agent fetches the target from a source outside the deployment that it can reach, such as the open web or a third-party service, which even pretraining-data filtering does not prevent and which only controls on what the agent seeks, reads or repeats, or on its ability to process the topic, have been shown to reduce. The fourth is exposure: a channel leaks only to whoever is allowed to read it, so a leak in a tool observation belongs to the store and the trace policy rather than to the unlearning method. The paper also shows that the agent harness can hide failure as well as reveal it, through policy collapse that reads as forgetting. It states a certificate as a triple of source, observer and audience, consolidates 34 published results in a table, maps four deployment purposes onto the failures that matter for each, gives an audit procedure that assigns every leak to the party able to fix it, and specifies six studies that would settle the open part.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 121 papers by Pranay Mahendrakar, or his ORCID record.