LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Gaps
Building safe Large Language Models (LLMs) across multiple languages is
essential in ensuring both safe access and linguistic diversity. To this end,
we introduce M-ALERT, a multilingual benchmark that evaluates the safety of
LLMs in five languages: English, French, German, Italian, and Spanish. M-ALERT
includes 15k high-quality prompts per language, totaling 75k, following the
detailed ALERT taxonomy. Our extensive experiments on 10 state-of-the-art LLMs
highlight the importance of language-specific safety analysis, revealing that
models often exhibit significant inconsistencies in safety across languages and
categories. For instance, Llama3.2 shows high unsafety in the category
crime_tax for Italian but remains safe in other languages. Similar differences
can be observed across all models. In contrast, certain categories, such as
substance_cannabis and crime_propaganda, consistently trigger unsafe responses
across models and languages. These findings underscore the need for robust
multilingual safety practices in LLMs to ensure safe and responsible usage
across diverse user communities.
676931b8bc5af30a79d40a76
Felix Friedrich
admin_assigned
2024-12-23T11:35:28.332Z
62e7dd4036a8e8a82700041c
https://cdn-avatars.huggingface.co/v1/production/uploads/62e7dd4036a8e8a82700041c/Dgk9mXYLVd4LpiNLWjn-q.jpeg
Felix Friedrich
user
felfri
676931b8bc5af30a79d40a77
Simone Tedeschi
claimed_verified
2024-12-23T11:09:30.612Z
61b85aa99ba538c73a7dc78b
https://cdn-avatars.huggingface.co/v1/production/uploads/61b85aa99ba538c73a7dc78b/gWxtQAvOYn7cXgE_nAy0p.jpeg
Simone Tedeschi
user
sted97
676931b8bc5af30a79d40a78
Patrick Schramowski
admin_assigned
2024-12-23T11:35:22.391Z
62d021a3dd7bdfc5e5c61c5c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d021a3dd7bdfc5e5c61c5c/bnQW2SqirfGaQmI84HW_c.jpeg
Patrick Schramowski
user
PSaiml
676931b8bc5af30a79d40a79
Manuel Brack
admin_assigned
2024-12-23T11:35:34.064Z
62fa1d95e8c9c532aa75331c
https://cdn-avatars.huggingface.co/v1/production/uploads/62fa1d95e8c9c532aa75331c/WFfk_n8gOj845pSkfdazA.jpeg
Manuel Brack
user
mbrack
676931b8bc5af30a79d40a7a
Roberto Navigli
admin_assigned
2024-12-23T11:35:40.306Z
63845c7c99292a80134bc784
https://cdn-avatars.huggingface.co/v1/production/uploads/63845c7c99292a80134bc784/eKOPJMjm8F-JUKT_mi7Dv.png
Roberto Navigli
user
navigli
676931b8bc5af30a79d40a7b
Huu Nguyen
claimed_verified
2025-03-02T20:18:58.316Z
5fc6879e1c5ee87b1164876d
https://cdn-avatars.huggingface.co/v1/production/uploads/5fc6879e1c5ee87b1164876d/Tjnm_lv0Bq0gPbFOTDH6E.jpeg
Huu Nguyen
user
huu-ontocord
676931b8bc5af30a79d40a7c
Bo Li
676931b8bc5af30a79d40a7d
Kristian Kersting
LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Gaps
Building safe Large Language Models (LLMs) across multiple languages is
essential in ensuring both safe access and linguistic diversity. To this end,
we introduce M-ALERT, a multilingual benchmark that evaluates the safety of
LLMs in five languages: English, French, German, Italian, and Spanish. M-ALERT
includes 15k high-quality prompts per language, totaling 75k, following the
detailed ALERT taxonomy. Our extensive experiments on 10 state-of-the-art LLMs
highlight the importance of language-specific safety analysis, revealing that
models often exhibit significant inconsistencies in safety across languages and
categories. For instance, Llama3.2 shows high unsafety in the category
crime_tax for Italian but remains safe in other languages. Similar differences
can be observed across all models. In contrast, certain categories, such as
substance_cannabis and crime_propaganda, consistently trigger unsafe responses
across models and languages. These findings underscore the need for robust
multilingual safety practices in LLMs to ensure safe and responsible usage
across diverse user communities.
M-ALERT evaluates the safety of multilingual LLMs across multiple languages, highlighting significant inconsistencies in safety performance and emphasizing the need for robust multilingual safety measures.
Large Language Models LLMs multilingual benchmark M-ALERT high-quality prompts ALERT taxonomy unsafety crime_tax substance_cannabis crime_propaganda language-specific safety analysis multilingual safety practices