A Missing Slice Is a Missing Measurement: What Language-Level Audits of Safety Datasets Establish, Why the Multi-Turn Residual and Cross-Lingual Transfer Break the Step From Data Gaps to Model Behaviour, and the Contrasts That Would Price Native Data

Abstract

Life of Research

Model providers cite multilingual safety benchmarks as evidence that their models are safe for speakers of many languages, and a 2026 audit shows that such claims often dissolve when a single language is inspected: across 25 language slices of 20 safety datasets in Hausa, Swahili and French, it finds overstated provenance, unreported agreement, gated or unreleased data, recycled text, and no native coverage of self-harm or sexual content in either African language. The audit then reads the persistence of multi-turn jailbreaks in low-resource languages, after single-turn translation attacks were largely closed, as the outcome its data gaps would predict. In this analysis the author reviews and synthesises published results to show that the first finding is well supported and the second step is not. A thin or translated slice is first a defect of measurement: translated attack prompts that lose their meaning lower the attack success a benchmark records, so a poor slice tends to make a model look safer, and judges, comprehension failures and deflections distort low-resource scores in ways a slice count does not show. The multi-turn residual is not shown to be larger in Kiswahili than in English in the study the audit relies on: English multi-turn harmful-response rates there run from 52.7 to 83.6 percent, against 41.8 to 70.9 percent for Kiswahili, though the English condition replays conversations that had already succeeded in English and is favoured by construction, and the 2023 Swahili figure the audit cites (83.5 percent) was obtained with an English jailbreak template that already succeeded 72.1 percent of the time in English. Where per-language behaviour gaps do exist, the clearest predictors reported are properties of the model rather than of any safety dataset: how well the model aligns the language with English, which tracked how far English-only preference tuning reduced toxicity in one study, and, on the evidence of a training study that gave every language the same translated safety data, how much of the language pretraining supplied. A 61-configuration study finds 22 configurations least safe in English. The paper consolidates 30 published results in a table and a figure, separates four accounts of a per-language safety gap, gives a reading procedure for a per-language safety claim, and specifies the native-versus-translated, matched-data and difference-in-differences contrasts that would establish what native safety data buys.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 144 papers by Pranay Mahendrakar, or his ORCID record.