BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented
  Generation
Large language models excel at creative generation but continue to struggle
with the issues of hallucination and bias. While retrieval-augmented generation
(RAG) provides a framework for grounding LLMs' responses in accurate and
up-to-date information, it still raises the question of bias: which sources
should be selected for inclusion in the context? And how should their
importance be weighted? In this paper, we study the challenge of cross-lingual
RAG and present a dataset to investigate the robustness of existing systems at
answering queries about geopolitical disputes, which exist at the intersection
of linguistic, cultural, and political boundaries. Our dataset is sourced from
Wikipedia pages containing information relevant to the given queries and we
investigate the impact of including additional context, as well as the
composition of this context in terms of language and source, on an LLM's
response. Our results show that existing RAG systems continue to be challenged
by cross-lingual use cases and suffer from a lack of consistency when they are
provided with competing information in multiple languages. We present case
studies to illustrate these issues and outline steps for future research to
address these challenges. We make our dataset and code publicly available at
https://github.com/manestay/bordIRlines.
66feb92281382f012cce3e67
Bryan Li
claimed_verified
2024-10-07T07:28:35.326Z
618d8e2cfc15c4a6739b17e6
/avatars/3f02c0d1fad478722f180d9559ae215d.svg
Bryan Li
user
manestay
66feb92281382f012cce3e68
Samar Haider
66feb92281382f012cce3e69
Fiona Luo
claimed_verified
2024-10-04T07:31:47.983Z
66f48b642f66635def4bb0c9
/avatars/86494d96ad1018d6cd577b1704b52c9b.svg
Fiona Luo
user
flluo
66feb92281382f012cce3e6a
Adwait Agashe
66feb92281382f012cce3e6b
Chris Callison-Burch
BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented
  Generation
Large language models excel at creative generation but continue to struggle
with the issues of hallucination and bias. While retrieval-augmented generation
(RAG) provides a framework for grounding LLMs' responses in accurate and
up-to-date information, it still raises the question of bias: which sources
should be selected for inclusion in the context? And how should their
importance be weighted? In this paper, we study the challenge of cross-lingual
RAG and present a dataset to investigate the robustness of existing systems at
answering queries about geopolitical disputes, which exist at the intersection
of linguistic, cultural, and political boundaries. Our dataset is sourced from
Wikipedia pages containing information relevant to the given queries and we
investigate the impact of including additional context, as well as the
composition of this context in terms of language and source, on an LLM's
response. Our results show that existing RAG systems continue to be challenged
by cross-lingual use cases and suffer from a lack of consistency when they are
provided with competing information in multiple languages. We present case
studies to illustrate these issues and outline steps for future research to
address these challenges. We make our dataset and code publicly available at
https://github.com/manestay/bordIRlines.
RAG systems face challenges in cross-lingual settings, showing inconsistency when handling multilingual, competing context information, particularly in geopolitical domains.
retrieval-augmented generation RAG large language models LLMs geopolitical disputes multilingual cross-lingual competing context information Wikipedia pages