RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen
  Reference Content
Large Language Models (LLMs) are trained on vast amounts of data, most of
which is automatically scraped from the internet. This data includes
encyclopedic documents that harbor a vast amount of general knowledge (e.g.,
Wikipedia) but also potentially overlap with benchmark datasets used for
evaluating LLMs. Consequently, evaluating models on test splits that might have
leaked into the training set is prone to misleading conclusions. To foster
sound evaluation of language models, we introduce a new test dataset named
RepLiQA, suited for question-answering and topic retrieval tasks. RepLiQA is a
collection of five splits of test sets, four of which have not been released to
the internet or exposed to LLM APIs prior to this publication. Each sample in
RepLiQA comprises (1) a reference document crafted by a human annotator and
depicting an imaginary scenario (e.g., a news article) absent from the
internet; (2) a question about the document's topic; (3) a ground-truth answer
derived directly from the information in the document; and (4) the paragraph
extracted from the reference document containing the answer. As such, accurate
answers can only be generated if a model can find relevant content within the
provided document. We run a large-scale benchmark comprising several
state-of-the-art LLMs to uncover differences in performance across models of
various types and sizes in a context-conditional language modeling setting.
Released splits of RepLiQA can be found here:
https://huggingface.co/datasets/ServiceNow/repliqa.
66723ae8cbf550c42aee3a88
Joao Monteiro
admin_assigned
2024-06-19T21:06:22.413Z
63347fed495073b88704cf02
/avatars/cb5545adaf1cfe0166119c586f4ac9c8.svg
João Monteiro
user
joaomonteirof
66723ae8cbf550c42aee3a89
Pierre-Andre Noel
admin_assigned
2024-06-19T21:06:28.571Z
646cc16b94eb019a96e1fb2e
/avatars/31f168a6c1ec45eb0c784d9119c1b9bf.svg
Pierre-Andre Noel
user
PierreAndreNoel
66723ae8cbf550c42aee3a8a
Etienne Marcotte
66723ae8cbf550c42aee3a8b
Sai Rajeswar
claimed_verified
2024-06-27T09:55:24.845Z
642f99079b2484d7d857341b
/avatars/01965cc5a5dbe9c08025a51973462a6a.svg
Sai Rajeswar
user
rajeswarsai
66723ae8cbf550c42aee3a8c
Valentina Zantedeschi
admin_assigned
2024-06-19T21:07:05.607Z
6454b7601a543cf97b1add68
/avatars/485ca04a5353046ebf3972357a17f976.svg
Valentina Zantedeschi
user
vzantedeschi
66723ae8cbf550c42aee3a8d
David Vazquez
claimed_verified
2025-05-08T09:00:33.588Z
62f6691f329d4d014d1b4087
https://cdn-avatars.huggingface.co/v1/production/uploads/1660315866736-noauth.jpeg
David Vazquez
user
dvazquez
66723ae8cbf550c42aee3a8e
Nicolas Chapados
admin_assigned
2024-06-19T21:07:15.270Z
631f54aa5ba8c026340b13cf
https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
Nicolas Chapados
user
nicolaschapados
66723ae8cbf550c42aee3a8f
Christopher Pal
66723ae8cbf550c42aee3a90
Perouz Taslakian
RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen
  Reference Content
Large Language Models (LLMs) are trained on vast amounts of data, most of
which is automatically scraped from the internet. This data includes
encyclopedic documents that harbor a vast amount of general knowledge (e.g.,
Wikipedia) but also potentially overlap with benchmark datasets used for
evaluating LLMs. Consequently, evaluating models on test splits that might have
leaked into the training set is prone to misleading conclusions. To foster
sound evaluation of language models, we introduce a new test dataset named
RepLiQA, suited for question-answering and topic retrieval tasks. RepLiQA is a
collection of five splits of test sets, four of which have not been released to
the internet or exposed to LLM APIs prior to this publication. Each sample in
RepLiQA comprises (1) a reference document crafted by a human annotator and
depicting an imaginary scenario (e.g., a news article) absent from the
internet; (2) a question about the document's topic; (3) a ground-truth answer
derived directly from the information in the document; and (4) the paragraph
extracted from the reference document containing the answer. As such, accurate
answers can only be generated if a model can find relevant content within the
provided document. We run a large-scale benchmark comprising several
state-of-the-art LLMs to uncover differences in performance across models of
various types and sizes in a context-conditional language modeling setting.
Released splits of RepLiQA can be found here:
https://huggingface.co/datasets/ServiceNow/repliqa.
A new test dataset called RepLiQA is introduced to evaluate language models by ensuring the test data has not been exposed to the models during training, focusing on question-answering and topic retrieval tasks.
Large Language Models LLMs RepLiQA question-answering topic retrieval context-conditional language modeling