A Named Answer Is Not a Correction: The Moral Self-Correction Threshold Moved From 22B to 3.8B on Benchmarks a Context-Blind Policy Can Pass, and the Controls That Would Separate Steering From Correcting

Abstract

Life of Research

Instructing a language model to avoid stereotypes lowers its measured social bias, and a 2023 study reported that this capacity for "moral self-correction" emerges at 22 billion parameters. A 2024 study reported the same capacity in a 3.8-billion-parameter model with safety post-training. In this analysis the author reviews and synthesises published results to show that the disagreement about the threshold is a disagreement about an instrument that cannot see what the word "correction" claims. Three findings carry the argument. First, in both studies the intervention is a single instruction appended to the question; no prior answer is revised. Second, the benchmarks reward a fixed target answer: on the ambiguous half of BBQ, which the 3.8B study and several others use alone, "unknown" is always correct, and on Winogender, as the 3.8B study scores it, the fair outcome is a gender-neutral pronoun, so a policy that ignores the context entirely scores perfectly. Third, where the matched control has been reported, the gain is often bought with it: a fairness instruction that named the unknown option, added to a stereotype-biased GPT-3 prompt, raised ambiguous-question accuracy from 2.6 to 96.6 percent while its accuracy on questions whose context names the answer fell from 97.3 to 51.5. The paper also shows that in two of the 2023 study's three experiments the scale at which correction appears is the scale at which measurable bias first appears. It separates three claims a gain can support, steerability, discrimination and revision, gives a decision procedure for classifying published results, and names the controls that would settle which of the three a model has.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 98 papers by Pranay Mahendrakar, or his ORCID record.