title: Language Models' Factuality Depends on the Language of Inquiry
summary: Multilingual language models (LMs) are expected to recall factual knowledge
  consistently across languages, yet they often fail to transfer knowledge
  between languages even when they possess the correct information in one of the
  languages. For example, we find that an LM may correctly identify Rashed Al
  Shashai as being from Saudi Arabia when asked in Arabic, but consistently fails
  to do so when asked in English or Swahili. To systematically investigate this
  limitation, we introduce a benchmark of 10,000 country-related facts across 13
  languages and propose three novel metrics: Factual Recall Score, Knowledge
  Transferability Score, and Cross-Lingual Factual Knowledge Transferability
  Score-to quantify factual recall and knowledge transferability in LMs across
  different languages. Our results reveal fundamental weaknesses in today's
  state-of-the-art LMs, particularly in cross-lingual generalization where models
  fail to transfer knowledge effectively across different languages, leading to
  inconsistent performance sensitive to the language used. Our findings emphasize
  the need for LMs to recognize language-specific factual reliability and
  leverage the most trustworthy information across languages. We release our
  benchmark and evaluation framework to drive future research in multilingual
  knowledge transfer.
paper_authors._id: 67bff526ca6e3c22b6e89d71
paper_authors.name: Tushar Aggarwal
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-27T18:59:26.826Z
paper_authors._id: 65d2f1e0fe21569868393411
paper_authors.avatarUrl: /avatars/1401020e76d958bef3f33e7449773694.svg
paper_authors.fullname: Tushar Aggarwal
paper_authors.type: user
paper_authors.user: AggarwalTushar
paper_authors._id: 67bff526ca6e3c22b6e89d72
paper_authors.name: Kumar Tanmay
paper_authors._id: 67bff526ca6e3c22b6e89d73
paper_authors.name: Ayush Agrawal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T09:13:56.625Z
paper_authors._id: 61a7cbb0fcbbebe775bf17fd
paper_authors.avatarUrl: /avatars/8b54907c6a1ea90a1242f26e03e117af.svg
paper_authors.fullname: Ayush Agrawal
paper_authors.type: user
paper_authors.user: ayush1801
paper_authors._id: 67bff526ca6e3c22b6e89d74
paper_authors.name: Kumar Ayush
paper_authors._id: 67bff526ca6e3c22b6e89d75
paper_authors.name: Hamid Palangi
paper_authors._id: 67bff526ca6e3c22b6e89d76
paper_authors.name: Paul Pu Liang
paper_title: Language Models' Factuality Depends on the Language of Inquiry
paper_summary: Multilingual language models (LMs) are expected to recall factual knowledge
  consistently across languages, yet they often fail to transfer knowledge
  between languages even when they possess the correct information in one of the
  languages. For example, we find that an LM may correctly identify Rashed Al
  Shashai as being from Saudi Arabia when asked in Arabic, but consistently fails
  to do so when asked in English or Swahili. To systematically investigate this
  limitation, we introduce a benchmark of 10,000 country-related facts across 13
  languages and propose three novel metrics: Factual Recall Score, Knowledge
  Transferability Score, and Cross-Lingual Factual Knowledge Transferability
  Score-to quantify factual recall and knowledge transferability in LMs across
  different languages. Our results reveal fundamental weaknesses in today's
  state-of-the-art LMs, particularly in cross-lingual generalization where models
  fail to transfer knowledge effectively across different languages, leading to
  inconsistent performance sensitive to the language used. Our findings emphasize
  the need for LMs to recognize language-specific factual reliability and
  leverage the most trustworthy information across languages. We release our
  benchmark and evaluation framework to drive future research in multilingual
  knowledge transfer.
paper_ai_summary: Multilingual language models show inconsistent factual knowledge recall and transferability across languages, highlighting the need for improved cross-lingual generalization.
paper_ai_keywords: multilingual language models factual knowledge knowledge transferability cross-lingual generalization Factual Recall Score Knowledge Transferability Score Cross-Lingual Factual Knowledge Transferability Score