Attribution Is Scored on a Finished Trace: What Failure-Attribution Benchmarks Measure, What Cascade Containment Would Need, and Why Neither Result Bounds the Other

Abstract

A family of recent proposals aims to stop errors from spreading through LLM-based multi-agent systems: genealogy-graph governance over message dependencies, cross-channel causal monitoring, propagation-aware remediation of contaminated state, infection-aware safeguarding, and corrector placement on the communication graph. Most of these mechanisms have to decide where a spreading error came from before they can decide what to cut. A separate and largely disjoint literature measures exactly that: automated failure attribution, the task of naming the agent and the step responsible for a failed run. Its reported accuracies at step granularity are low, and the obvious inference is that containment is built on a primitive that does not work. This paper argues that the inference is not available, in either direction, and sets out why. The attribution benchmarks score a method on a completed trace whose failure is already known to have occurred and whose ground truth is a single decisive step; a containment mechanism must act on a prefix, without knowing that a failure will occur, and its output feeds an intervention rather than a developer. Recent benchmark work reports that how much of the trace a method is shown, and whether the ground truth is allowed to be multi-valued, both move the measured accuracy substantially, which means the pessimistic numbers are not a stable property of the task. The paper also separates the containment proposals by what they localise and notes one that localises nothing at all; and it observes that the most widely used taxonomy of why these systems fail defines its first category as failures that occur during execution but reflect pre-execution design choices, for which the step a method points at is a symptom by construction. It closes with the measurements that would make the question decidable, and with the older fault-localisation literature that already ran this experiment once.

The literature search, drafting and citation verification for this paper were carried out with AI assistance under the author's direction. Every citation was machine-verified against the arXiv API and Crossref before inclusion, and every quantitative claim was read back against the cited source's own text, either its abstract or, where a claim is drawn from a paper's body, the located passage. The author is responsible for the final text and for all claims made in it.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 66 papers by Pranay Mahendrakar, or his ORCID record.