title: RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen
    Reference Content
summary: Large Language Models (LLMs) are trained on vast amounts of data, most of
  which is automatically scraped from the internet. This data includes
  encyclopedic documents that harbor a vast amount of general knowledge (e.g.,
  Wikipedia) but also potentially overlap with benchmark datasets used for
  evaluating LLMs. Consequently, evaluating models on test splits that might have
  leaked into the training set is prone to misleading conclusions. To foster
  sound evaluation of language models, we introduce a new test dataset named
  RepLiQA, suited for question-answering and topic retrieval tasks. RepLiQA is a
  collection of five splits of test sets, four of which have not been released to
  the internet or exposed to LLM APIs prior to this publication. Each sample in
  RepLiQA comprises (1) a reference document crafted by a human annotator and
  depicting an imaginary scenario (e.g., a news article) absent from the
  internet; (2) a question about the document's topic; (3) a ground-truth answer
  derived directly from the information in the document; and (4) the paragraph
  extracted from the reference document containing the answer. As such, accurate
  answers can only be generated if a model can find relevant content within the
  provided document. We run a large-scale benchmark comprising several
  state-of-the-art LLMs to uncover differences in performance across models of
  various types and sizes in a context-conditional language modeling setting.
  Released splits of RepLiQA can be found here:
  https://huggingface.co/datasets/ServiceNow/repliqa.
paper_authors._id: 66723ae8cbf550c42aee3a88
paper_authors.name: Joao Monteiro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:06:22.413Z
paper_authors._id: 63347fed495073b88704cf02
paper_authors.avatarUrl: /avatars/cb5545adaf1cfe0166119c586f4ac9c8.svg
paper_authors.fullname: João Monteiro
paper_authors.type: user
paper_authors.user: joaomonteirof
paper_authors._id: 66723ae8cbf550c42aee3a89
paper_authors.name: Pierre-Andre Noel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:06:28.571Z
paper_authors._id: 646cc16b94eb019a96e1fb2e
paper_authors.avatarUrl: /avatars/31f168a6c1ec45eb0c784d9119c1b9bf.svg
paper_authors.fullname: Pierre-Andre Noel
paper_authors.type: user
paper_authors.user: PierreAndreNoel
paper_authors._id: 66723ae8cbf550c42aee3a8a
paper_authors.name: Etienne Marcotte
paper_authors._id: 66723ae8cbf550c42aee3a8b
paper_authors.name: Sai Rajeswar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:55:24.845Z
paper_authors._id: 642f99079b2484d7d857341b
paper_authors.avatarUrl: /avatars/01965cc5a5dbe9c08025a51973462a6a.svg
paper_authors.fullname: Sai Rajeswar
paper_authors.type: user
paper_authors.user: rajeswarsai
paper_authors._id: 66723ae8cbf550c42aee3a8c
paper_authors.name: Valentina Zantedeschi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:07:05.607Z
paper_authors._id: 6454b7601a543cf97b1add68
paper_authors.avatarUrl: /avatars/485ca04a5353046ebf3972357a17f976.svg
paper_authors.fullname: Valentina Zantedeschi
paper_authors.type: user
paper_authors.user: vzantedeschi
paper_authors._id: 66723ae8cbf550c42aee3a8d
paper_authors.name: David Vazquez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T09:00:33.588Z
paper_authors._id: 62f6691f329d4d014d1b4087
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660315866736-noauth.jpeg
paper_authors.fullname: David Vazquez
paper_authors.type: user
paper_authors.user: dvazquez
paper_authors._id: 66723ae8cbf550c42aee3a8e
paper_authors.name: Nicolas Chapados
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:07:15.270Z
paper_authors._id: 631f54aa5ba8c026340b13cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
paper_authors.fullname: Nicolas Chapados
paper_authors.type: user
paper_authors.user: nicolaschapados
paper_authors._id: 66723ae8cbf550c42aee3a8f
paper_authors.name: Christopher Pal
paper_authors._id: 66723ae8cbf550c42aee3a90
paper_authors.name: Perouz Taslakian
paper_title: RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen
    Reference Content
paper_summary: Large Language Models (LLMs) are trained on vast amounts of data, most of
  which is automatically scraped from the internet. This data includes
  encyclopedic documents that harbor a vast amount of general knowledge (e.g.,
  Wikipedia) but also potentially overlap with benchmark datasets used for
  evaluating LLMs. Consequently, evaluating models on test splits that might have
  leaked into the training set is prone to misleading conclusions. To foster
  sound evaluation of language models, we introduce a new test dataset named
  RepLiQA, suited for question-answering and topic retrieval tasks. RepLiQA is a
  collection of five splits of test sets, four of which have not been released to
  the internet or exposed to LLM APIs prior to this publication. Each sample in
  RepLiQA comprises (1) a reference document crafted by a human annotator and
  depicting an imaginary scenario (e.g., a news article) absent from the
  internet; (2) a question about the document's topic; (3) a ground-truth answer
  derived directly from the information in the document; and (4) the paragraph
  extracted from the reference document containing the answer. As such, accurate
  answers can only be generated if a model can find relevant content within the
  provided document. We run a large-scale benchmark comprising several
  state-of-the-art LLMs to uncover differences in performance across models of
  various types and sizes in a context-conditional language modeling setting.
  Released splits of RepLiQA can be found here:
  https://huggingface.co/datasets/ServiceNow/repliqa.
paper_ai_summary: A new test dataset called RepLiQA is introduced to evaluate language models by ensuring the test data has not been exposed to the models during training, focusing on question-answering and topic retrieval tasks.
paper_ai_keywords: Large Language Models LLMs RepLiQA question-answering topic retrieval context-conditional language modeling