A Residual Is Not a Third Factor: What the Gap Between a Perfect-Retention Ensemble and Joint Training Measures in Continual Learning, Why Chunk Size, Repetition and the Combiner Move It, and the Controls That Would Make Co-observation a Mechanism

Abstract

Life of Research

Continual learning is scored against two reference points: naive sequential training below and joint training on all the data above. The usual reading is that the distance between them is made of forgetting and lost plasticity, so a learner that solved both would reach the upper line. A 2026 study disputes this. On vision benchmarks split into identically distributed chunks, a perfect-retention ensemble of sequential checkpoints still falls 2.2 to 3.9 linear-probe points short of incrementally joint training at the fourth chunk, and its authors name the residual a third facet, co-observation. In this analysis the author reviews and synthesises published results to show that the residual is real but that its name runs ahead of its evidence. Re-expressing the study's own tables, the author shows that the residual's share of the naive-to-joint gap ranges from 18 to 41 percent, smaller than the plasticity term in both self-supervised settings, that the claim that a 30 percent replay buffer closes half of it holds at the last chunk for Barlow Twins (60 percent) but not for supervised training (33 percent) or I-JEPA (14 percent), and that under the study's fixed per-chunk iteration budget each sequential member sees every example of its chunk about 100 times in one burst in supervised CIFAR-100 (about 90 in supervised ImageNet-100, and several hundred in the self-supervised settings, whose budget matches 400 joint epochs). On identically distributed chunks, seeing data together is the same thing as fitting each model on more data, so the residual is confounded with per-stage sample size and repetition, with the warm-start generalisation loss that its members also carry, and with the concatenate-and-project rule that combines them. Other studies point the same way: the sequential-to-joint gap shrinks with chunk size, falls to about one point for self-supervised pretraining on 320,000-image chunks, and closes to 0.02 in loss for language-model pretraining with 5 percent replay, while divide-and-conquer theory loses nothing in rate when local fits are regularised for the whole sample. The paper consolidates the measured values in a table, gives an attribution procedure, and specifies five controls that would turn the residual into a mechanism or dissolve it.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 144 papers by Pranay Mahendrakar, or his ORCID record.