title: BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive
    Retrieval
summary: Existing retrieval benchmarks primarily consist of information-seeking
  queries (e.g., aggregated questions from search engines) where keyword or
  semantic-based retrieval is usually sufficient. However, many complex
  real-world queries require in-depth reasoning to identify relevant documents
  that go beyond surface form matching. For example, finding documentation for a
  coding question requires understanding the logic and syntax of the functions
  involved. To better benchmark retrieval on such challenging queries, we
  introduce BRIGHT, the first text retrieval benchmark that requires intensive
  reasoning to retrieve relevant documents. BRIGHT is constructed from the 1,398
  real-world queries collected from diverse domains (such as economics,
  psychology, robotics, software engineering, earth sciences, etc.), sourced from
  naturally occurring or carefully curated human data. Extensive evaluation
  reveals that even state-of-the-art retrieval models perform poorly on BRIGHT.
  The leading model on the MTEB leaderboard [38 ], which achieves a score of 59.0
  nDCG@10,2 produces a score of nDCG@10 of 18.0 on BRIGHT. We further demonstrate
  that augmenting queries with Chain-of-Thought reasoning generated by large
  language models (LLMs) improves performance by up to 12.2 points. Moreover,
  BRIGHT is robust against data leakage during pretraining of the benchmarked
  models as we validate by showing similar performance even when documents from
  the benchmark are included in the training data. We believe that BRIGHT paves
  the way for future research on retrieval systems in more realistic and
  challenging settings. Our code and data are available at
  https://brightbenchmark.github.io.
paper_authors._id: 669a08cdf21b09fdce917195
paper_authors.name: Hongjin Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:22:55.865Z
paper_authors._id: 632d53951538d4798a73c849
paper_authors.avatarUrl: /avatars/b7d0a895e669bcd1303c4716b5401c36.svg
paper_authors.fullname: Hongjin SU
paper_authors.type: user
paper_authors.user: multi-train
paper_authors._id: 669a08cdf21b09fdce917196
paper_authors.name: Howard Yen
paper_authors._id: 669a08cdf21b09fdce917197
paper_authors.name: Mengzhou Xia
paper_authors._id: 669a08cdf21b09fdce917198
paper_authors.name: Weijia Shi
paper_authors._id: 669a08cdf21b09fdce917199
paper_authors.name: Niklas Muennighoff
paper_authors._id: 669a08cdf21b09fdce91719a
paper_authors.name: Han-yu Wang
paper_authors._id: 669a08cdf21b09fdce91719b
paper_authors.name: Haisu Liu
paper_authors._id: 669a08cdf21b09fdce91719c
paper_authors.name: Quan Shi
paper_authors._id: 669a08cdf21b09fdce91719d
paper_authors.name: Zachary S. Siegel
paper_authors._id: 669a08cdf21b09fdce91719e
paper_authors.name: Michael Tang
paper_authors._id: 669a08cdf21b09fdce91719f
paper_authors.name: Ruoxi Sun
paper_authors._id: 669a08cdf21b09fdce9171a0
paper_authors.name: Jinsung Yoon
paper_authors._id: 669a08cdf21b09fdce9171a1
paper_authors.name: Sercan O. Arik
paper_authors._id: 669a08cdf21b09fdce9171a2
paper_authors.name: Danqi Chen
paper_authors._id: 669a08cdf21b09fdce9171a3
paper_authors.name: Tao Yu
paper_title: BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive
    Retrieval
paper_summary: Existing retrieval benchmarks primarily consist of information-seeking
  queries (e.g., aggregated questions from search engines) where keyword or
  semantic-based retrieval is usually sufficient. However, many complex
  real-world queries require in-depth reasoning to identify relevant documents
  that go beyond surface form matching. For example, finding documentation for a
  coding question requires understanding the logic and syntax of the functions
  involved. To better benchmark retrieval on such challenging queries, we
  introduce BRIGHT, the first text retrieval benchmark that requires intensive
  reasoning to retrieve relevant documents. BRIGHT is constructed from the 1,398
  real-world queries collected from diverse domains (such as economics,
  psychology, robotics, software engineering, earth sciences, etc.), sourced from
  naturally occurring or carefully curated human data. Extensive evaluation
  reveals that even state-of-the-art retrieval models perform poorly on BRIGHT.
  The leading model on the MTEB leaderboard [38 ], which achieves a score of 59.0
  nDCG@10,2 produces a score of nDCG@10 of 18.0 on BRIGHT. We further demonstrate
  that augmenting queries with Chain-of-Thought reasoning generated by large
  language models (LLMs) improves performance by up to 12.2 points. Moreover,
  BRIGHT is robust against data leakage during pretraining of the benchmarked
  models as we validate by showing similar performance even when documents from
  the benchmark are included in the training data. We believe that BRIGHT paves
  the way for future research on retrieval systems in more realistic and
  challenging settings. Our code and data are available at
  https://brightbenchmark.github.io.
paper_ai_keywords: text retrieval benchmark intensive reasoning Chain-of-Thought large language models (LLMs) MTEB leaderboard nDCG@10 nDCG@0 data leakage pretraining