title: LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Gaps
summary: Building safe Large Language Models (LLMs) across multiple languages is
  essential in ensuring both safe access and linguistic diversity. To this end,
  we introduce M-ALERT, a multilingual benchmark that evaluates the safety of
  LLMs in five languages: English, French, German, Italian, and Spanish. M-ALERT
  includes 15k high-quality prompts per language, totaling 75k, following the
  detailed ALERT taxonomy. Our extensive experiments on 10 state-of-the-art LLMs
  highlight the importance of language-specific safety analysis, revealing that
  models often exhibit significant inconsistencies in safety across languages and
  categories. For instance, Llama3.2 shows high unsafety in the category
  crime_tax for Italian but remains safe in other languages. Similar differences
  can be observed across all models. In contrast, certain categories, such as
  substance_cannabis and crime_propaganda, consistently trigger unsafe responses
  across models and languages. These findings underscore the need for robust
  multilingual safety practices in LLMs to ensure safe and responsible usage
  across diverse user communities.
paper_authors._id: 676931b8bc5af30a79d40a76
paper_authors.name: Felix Friedrich
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:35:28.332Z
paper_authors._id: 62e7dd4036a8e8a82700041c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e7dd4036a8e8a82700041c/Dgk9mXYLVd4LpiNLWjn-q.jpeg
paper_authors.fullname: Felix Friedrich
paper_authors.type: user
paper_authors.user: felfri
paper_authors._id: 676931b8bc5af30a79d40a77
paper_authors.name: Simone Tedeschi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:09:30.612Z
paper_authors._id: 61b85aa99ba538c73a7dc78b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61b85aa99ba538c73a7dc78b/gWxtQAvOYn7cXgE_nAy0p.jpeg
paper_authors.fullname: Simone Tedeschi
paper_authors.type: user
paper_authors.user: sted97
paper_authors._id: 676931b8bc5af30a79d40a78
paper_authors.name: Patrick Schramowski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:35:22.391Z
paper_authors._id: 62d021a3dd7bdfc5e5c61c5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d021a3dd7bdfc5e5c61c5c/bnQW2SqirfGaQmI84HW_c.jpeg
paper_authors.fullname: Patrick Schramowski
paper_authors.type: user
paper_authors.user: PSaiml
paper_authors._id: 676931b8bc5af30a79d40a79
paper_authors.name: Manuel Brack
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:35:34.064Z
paper_authors._id: 62fa1d95e8c9c532aa75331c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fa1d95e8c9c532aa75331c/WFfk_n8gOj845pSkfdazA.jpeg
paper_authors.fullname: Manuel Brack
paper_authors.type: user
paper_authors.user: mbrack
paper_authors._id: 676931b8bc5af30a79d40a7a
paper_authors.name: Roberto Navigli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:35:40.306Z
paper_authors._id: 63845c7c99292a80134bc784
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63845c7c99292a80134bc784/eKOPJMjm8F-JUKT_mi7Dv.png
paper_authors.fullname: Roberto Navigli
paper_authors.type: user
paper_authors.user: navigli
paper_authors._id: 676931b8bc5af30a79d40a7b
paper_authors.name: Huu Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:18:58.316Z
paper_authors._id: 5fc6879e1c5ee87b1164876d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fc6879e1c5ee87b1164876d/Tjnm_lv0Bq0gPbFOTDH6E.jpeg
paper_authors.fullname: Huu Nguyen
paper_authors.type: user
paper_authors.user: huu-ontocord
paper_authors._id: 676931b8bc5af30a79d40a7c
paper_authors.name: Bo Li
paper_authors._id: 676931b8bc5af30a79d40a7d
paper_authors.name: Kristian Kersting
paper_title: LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Gaps
paper_summary: Building safe Large Language Models (LLMs) across multiple languages is
  essential in ensuring both safe access and linguistic diversity. To this end,
  we introduce M-ALERT, a multilingual benchmark that evaluates the safety of
  LLMs in five languages: English, French, German, Italian, and Spanish. M-ALERT
  includes 15k high-quality prompts per language, totaling 75k, following the
  detailed ALERT taxonomy. Our extensive experiments on 10 state-of-the-art LLMs
  highlight the importance of language-specific safety analysis, revealing that
  models often exhibit significant inconsistencies in safety across languages and
  categories. For instance, Llama3.2 shows high unsafety in the category
  crime_tax for Italian but remains safe in other languages. Similar differences
  can be observed across all models. In contrast, certain categories, such as
  substance_cannabis and crime_propaganda, consistently trigger unsafe responses
  across models and languages. These findings underscore the need for robust
  multilingual safety practices in LLMs to ensure safe and responsible usage
  across diverse user communities.