Why English-only safety tests miss multilingual failures
Large language models are often aligned and evaluated primarily in English. That creates a blind spot: a model may refuse a harmful request in English but respond unsafely after the request is translated, paraphrased, or adapted to a local cultural context. The problem is especially serious for low-resource languages, where both safety data and expert red-teaming are limited.
In How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates, Max Neuwinger, Hanna Yukhymenko, and Imanol Schlag of ETH Zurich present a way to expand multilingual safety data without manually writing every harmful example. Their release, MSAFETEMP, is intended for both safety evaluation and alignment training across dozens of languages.
The core idea: separate harmful intent from the surrounding text
The authors’ key design choice is to convert a harmful English prompt into a structured template. A language model extracts the short phrases that carry the harmful meaning and replaces them with typed placeholders. The remaining sentence can then be translated or paraphrased without asking a safety-trained model to directly reproduce the full harmful request.
After translation, the pipeline fills the placeholders in several ways:
- Translated variants: generic values preserve the original harmful scenario in the target language.
- Benign minimal pairs: safe values preserve much of the same wording while creating requests that should be answered, allowing the researchers to measure over-refusal.
- Culturally localized variants: values are selected from language-specific “cultural cards” containing relevant groups, institutions, events, and concepts.
Each seed produces a main template and two paraphrases. The generated variants pass separate validation stages for meaning preservation, language quality, coherence, harmfulness, and cultural fit. The authors deliberately use different model families for generation and judging, although all-automated validation remains an important caveat.
What MSAFETEMP contains
The training track starts with 2,500 harmful prompts from WildGuardMix and expands them into a validated collection covering 44 languages. The evaluation track uses 200 HarmBench behaviors and spans 60 languages. The paper reports more than 1.8 million validated training prompts and 216,486 evaluation prompts before discussing the final filtering and track-specific details.
The scale comes from controlled variation rather than a large collection of new human-written seeds. Three templates per seed and language can receive up to five fills each, producing as many as 15 harmful variants where generation succeeds. The authors report substantial entity diversity and find no meaningful degradation in diversity between higher- and lower-resource language groups.
A small human study offers encouraging but limited evidence about quality. Fifteen expert raters covering 11 languages assessed 600 judgments. They gave the prompts average naturalness and cultural-fit scores of 4.43 and 4.46 out of 5. However, raters endorsed only 65% of the benign labels, compared with 91% of the harmful labels. Some supposedly benign controls still appeared harmful in context, which means over-refusal estimates based on those pairs may be inflated.
English results were not reliable predictors
The evaluation results support the authors’ central warning. Across the tested open models, multilingual safety varied substantially by model and language. On HarmBench, for example, Olmo-3.1-32B-Instruct had an attack success rate of 0% on the original English prompts but 13.7% on translated variants. The two Apertus models also became more vulnerable after translation, while Llama-3.3-70B-Instruct was more vulnerable on its English originals than on its translated or localized versions.
The WildGuard test split showed a similar pattern for several models: translated variants broke Olmo, Gemma, and both Apertus models between 2.4 and 4.4 times as often as the original English prompts. Qwen3.5-27B and GPT-OSS-20B remained near or below 1% attack success across the main arms, but GPT-OSS did so with high over-refusal. In other words, a low attack rate can conceal a model that refuses too many harmless requests.
Cultural localization exposed a different kind of gap. It did not consistently increase the overall attack success rate; for most leading models, localized prompts actually produced lower rates than translated prompts. But the failures were different. The translated and localized arms agreed on fewer than half of the behaviors they broke, and the authors report that roughly 45% of arm-specific failures appeared only with local entities. Localization therefore changes the failure surface even when it does not raise the average failure rate.
Training reduced attacks, but calibration remained difficult
The authors also fine-tuned several open models with MSAFETEMP data. Supervised fine-tuning substantially reduced multilingual attack success across model families and improved safety on languages that were absent from the training data. This result suggests cross-lingual transfer rather than simple memorization of the languages represented in training.
The tradeoff was over-refusal and some loss of general capability, particularly on GSM8K mathematics results. Direct preference optimization, applied after supervised fine-tuning, recovered much of the lost helpfulness while retaining low attack success. The recovery was strong for some models but incomplete for others, so the paper does not establish a universal recipe for balancing safety and utility.
Limitations and practical implications
MSAFETEMP is a useful research artifact, not a replacement for native-speaker red-teaming. The authors acknowledge that most labels and cultural judgments come from automated models. Human validation covered only a small subset of languages and participants, and inherited labels may miss cases where localization makes a prompt more severe. The dataset also uses single-turn, fixed-template prompts and does not fully represent multi-turn attacks.
For practitioners, the message is straightforward: evaluate safety by language, culture, and helpfulness together. Translate existing benchmarks, but do not stop there. Add local entities, paraphrases, matched benign controls, and multiple fills per template. Track attack success and over-refusal jointly, and validate sensitive examples with native speakers before using them for training or deployment decisions.
The authors’ broader contribution is a scalable way to make those evaluations possible. Their results do not show that templates solve multilingual alignment, but they demonstrate why English refusal rates are insufficient and provide a practical framework for finding—and training against—failures that English-only testing can miss.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.