Which Loss Prices the English Share? What Target-Language Validation Loss Can and Cannot See in Continued Pretraining for Language Adaptation, Why Two Training Regimes Disagree About What English Buys, and the Instruments a Mixture Choice Needs

Abstract

Life of Research

When an English-centric language model is adapted to a new language by continued pretraining, practitioners usually keep a share of English in the training mixture, and a growing family of mixture laws chooses that share by predicting validation loss. A 2025 study found that adding English left target-language validation perplexity unchanged while deciding whether few-shot ability in the target language emerged at all, and the result invites the reading that perplexity cannot predict emergent abilities. In this analysis the author reviews and synthesises published results to show that the evidence supports a narrower and more useful conclusion. A loss measured on held-out text from the target corpus registers only what is expressed in that text. It cannot rank mixtures, because what the English share changes is expressed elsewhere, and even within one run it can miss an early collapse of an ability it does not measure. The same study's loss on few-shot answer labels moved within the first few steps of training, thousands of steps before accuracy did; and an independent 2026 study of bilingual pretraining from scratch, over roughly a thousand runs, found that target loss chose good checkpoints within a run but could not rank bilingual against monolingual training. The two regimes then disagree about what English supplies. In continued pretraining it acts mainly as a damper on early parameter drift that protects in-context learning, and a curriculum that drops it after the first tenth of training, or weight averaging without it, recovers most of the downstream benefit. Trained from scratch on scarce target data, English supplies knowledge and diversity the target corpus lacks; measured as equivalent unique target data, mixing was worth two to three times on validation loss and two to thirteen times on benchmark accuracy. The paper consolidates the measured values in two tables and a figure, states the mixture choice as a decision procedure keyed to the instrument each comparison needs, and names the experiments that would separate the accounts.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 103 papers by Pranay Mahendrakar, or his ORCID record.