title: BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented
    Generation
summary: Large language models excel at creative generation but continue to struggle
  with the issues of hallucination and bias. While retrieval-augmented generation
  (RAG) provides a framework for grounding LLMs' responses in accurate and
  up-to-date information, it still raises the question of bias: which sources
  should be selected for inclusion in the context? And how should their
  importance be weighted? In this paper, we study the challenge of cross-lingual
  RAG and present a dataset to investigate the robustness of existing systems at
  answering queries about geopolitical disputes, which exist at the intersection
  of linguistic, cultural, and political boundaries. Our dataset is sourced from
  Wikipedia pages containing information relevant to the given queries and we
  investigate the impact of including additional context, as well as the
  composition of this context in terms of language and source, on an LLM's
  response. Our results show that existing RAG systems continue to be challenged
  by cross-lingual use cases and suffer from a lack of consistency when they are
  provided with competing information in multiple languages. We present case
  studies to illustrate these issues and outline steps for future research to
  address these challenges. We make our dataset and code publicly available at
  https://github.com/manestay/bordIRlines.
paper_authors._id: 66feb92281382f012cce3e67
paper_authors.name: Bryan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:28:35.326Z
paper_authors._id: 618d8e2cfc15c4a6739b17e6
paper_authors.avatarUrl: /avatars/3f02c0d1fad478722f180d9559ae215d.svg
paper_authors.fullname: Bryan Li
paper_authors.type: user
paper_authors.user: manestay
paper_authors._id: 66feb92281382f012cce3e68
paper_authors.name: Samar Haider
paper_authors._id: 66feb92281382f012cce3e69
paper_authors.name: Fiona Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:31:47.983Z
paper_authors._id: 66f48b642f66635def4bb0c9
paper_authors.avatarUrl: /avatars/86494d96ad1018d6cd577b1704b52c9b.svg
paper_authors.fullname: Fiona Luo
paper_authors.type: user
paper_authors.user: flluo
paper_authors._id: 66feb92281382f012cce3e6a
paper_authors.name: Adwait Agashe
paper_authors._id: 66feb92281382f012cce3e6b
paper_authors.name: Chris Callison-Burch
paper_title: BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented
    Generation
paper_summary: Large language models excel at creative generation but continue to struggle
  with the issues of hallucination and bias. While retrieval-augmented generation
  (RAG) provides a framework for grounding LLMs' responses in accurate and
  up-to-date information, it still raises the question of bias: which sources
  should be selected for inclusion in the context? And how should their
  importance be weighted? In this paper, we study the challenge of cross-lingual
  RAG and present a dataset to investigate the robustness of existing systems at
  answering queries about geopolitical disputes, which exist at the intersection
  of linguistic, cultural, and political boundaries. Our dataset is sourced from
  Wikipedia pages containing information relevant to the given queries and we
  investigate the impact of including additional context, as well as the
  composition of this context in terms of language and source, on an LLM's
  response. Our results show that existing RAG systems continue to be challenged
  by cross-lingual use cases and suffer from a lack of consistency when they are
  provided with competing information in multiple languages. We present case
  studies to illustrate these issues and outline steps for future research to
  address these challenges. We make our dataset and code publicly available at
  https://github.com/manestay/bordIRlines.
paper_ai_keywords: retrieval-augmented generation (RAG) large language models (LLMs) hallucination bias cross-lingual RAG geopolitical disputes Wikipedia pages additional context language composition source composition