Told the Answer Is Not Taught to Reason: What On-Policy Self-Distillation From Privileged Context Transfers to a Language Model, Why Its Gains and Losses Divide Along Whether the Student Can Ever Hold What the Teacher Was Told, and the Controls That Would Separate Three Signals Under One Loss

Abstract

Life of Research

On-policy self-distillation trains a language model on its own samples against the token distribution of the same model shown something the student never sees: a reference solution, a final answer, execution feedback or a demonstration. Early 2026 reports credited the recipe with matching reinforcement learning at a fraction of the sampling cost. Later 2026 studies report the opposite on thinking models: accuracy falls by up to 17 percent relative on competition mathematics, by up to 40 percent in another study, and agents trained this way can end below their untrained base. In this analysis the author reviews and synthesises published results to show that the disagreement is not about the method but about three signals that one loss carries at once. A mode signal comes from the teacher's prompt format and reasoning mode; a behaviour signal comes from the confidence, brevity and absence of checking that knowing the answer licenses; an information signal comes from the content of what the teacher was told. On mathematics with students trained in non-thinking mode, ablations that shuffle or delete the privileged content show that the reported gain does not come from that content, and that the gain measured in the student's own non-thinking mode is largely the mode signal. Marker counts and per-token loss attributions show that much of the reported harm is the behaviour signal. The paper argues, using the imitation-gap results from robot and control learning, that the behaviour signal is safe to transfer only when the student can come to hold the information - in its weights, because the information recurs across instances, or in its own context before the step that needs it. Per-problem answers to unseen problems meet neither condition. The paper consolidates 34 published values in a table and a figure, states a reading procedure, sorts 17 published repairs by the signal each removes, and names six measurements that would settle what remains open.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 144 papers by Pranay Mahendrakar, or his ORCID record.