Language Models' Factuality Depends on the Language of Inquiry
Multilingual language models (LMs) are expected to recall factual knowledge
consistently across languages, yet they often fail to transfer knowledge
between languages even when they possess the correct information in one of the
languages. For example, we find that an LM may correctly identify Rashed Al
Shashai as being from Saudi Arabia when asked in Arabic, but consistently fails
to do so when asked in English or Swahili. To systematically investigate this
limitation, we introduce a benchmark of 10,000 country-related facts across 13
languages and propose three novel metrics: Factual Recall Score, Knowledge
Transferability Score, and Cross-Lingual Factual Knowledge Transferability
Score-to quantify factual recall and knowledge transferability in LMs across
different languages. Our results reveal fundamental weaknesses in today's
state-of-the-art LMs, particularly in cross-lingual generalization where models
fail to transfer knowledge effectively across different languages, leading to
inconsistent performance sensitive to the language used. Our findings emphasize
the need for LMs to recognize language-specific factual reliability and
leverage the most trustworthy information across languages. We release our
benchmark and evaluation framework to drive future research in multilingual
knowledge transfer.
67bff526ca6e3c22b6e89d71
Tushar Aggarwal
extracted_confirmed
2025-02-27T18:59:26.826Z
65d2f1e0fe21569868393411
/avatars/1401020e76d958bef3f33e7449773694.svg
Tushar Aggarwal
user
AggarwalTushar
67bff526ca6e3c22b6e89d72
Kumar Tanmay
67bff526ca6e3c22b6e89d73
Ayush Agrawal
claimed_verified
2025-02-27T09:13:56.625Z
61a7cbb0fcbbebe775bf17fd
/avatars/8b54907c6a1ea90a1242f26e03e117af.svg
Ayush Agrawal
user
ayush1801
67bff526ca6e3c22b6e89d74
Kumar Ayush
67bff526ca6e3c22b6e89d75
Hamid Palangi
67bff526ca6e3c22b6e89d76
Paul Pu Liang
Language Models' Factuality Depends on the Language of Inquiry
Multilingual language models (LMs) are expected to recall factual knowledge
consistently across languages, yet they often fail to transfer knowledge
between languages even when they possess the correct information in one of the
languages. For example, we find that an LM may correctly identify Rashed Al
Shashai as being from Saudi Arabia when asked in Arabic, but consistently fails
to do so when asked in English or Swahili. To systematically investigate this
limitation, we introduce a benchmark of 10,000 country-related facts across 13
languages and propose three novel metrics: Factual Recall Score, Knowledge
Transferability Score, and Cross-Lingual Factual Knowledge Transferability
Score-to quantify factual recall and knowledge transferability in LMs across
different languages. Our results reveal fundamental weaknesses in today's
state-of-the-art LMs, particularly in cross-lingual generalization where models
fail to transfer knowledge effectively across different languages, leading to
inconsistent performance sensitive to the language used. Our findings emphasize
the need for LMs to recognize language-specific factual reliability and
leverage the most trustworthy information across languages. We release our
benchmark and evaluation framework to drive future research in multilingual
knowledge transfer.
Multilingual language models show inconsistent factual knowledge recall and transferability across languages, highlighting the need for improved cross-lingual generalization.
multilingual language models factual knowledge knowledge transferability cross-lingual generalization Factual Recall Score Knowledge Transferability Score Cross-Lingual Factual Knowledge Transferability Score