title: FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on
    Technical Documents
summary: We introduce FreshStack, a reusable framework for automatically building
  information retrieval (IR) evaluation benchmarks from community-asked questions
  and answers. FreshStack conducts the following steps: (1) automatic corpus
  collection from code and technical documentation, (2) nugget generation from
  community-asked questions and answers, and (3) nugget-level support, retrieving
  documents using a fusion of retrieval techniques and hybrid architectures. We
  use FreshStack to build five datasets on fast-growing, recent, and niche topics
  to ensure the tasks are sufficiently challenging. On FreshStack, existing
  retrieval models, when applied out-of-the-box, significantly underperform
  oracle approaches on all five topics, denoting plenty of headroom to improve IR
  quality. In addition, we identify cases where rerankers do not clearly improve
  first-stage retrieval accuracy (two out of five topics). We hope that
  FreshStack will facilitate future work toward constructing realistic, scalable,
  and uncontaminated IR and RAG evaluation benchmarks. FreshStack datasets are
  available at: https://fresh-stack.github.io.
paper_authors._id: 6801a85879ba651f02e104f5
paper_authors.name: Nandan Thakur
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:41:03.081Z
paper_authors._id: 60196690dd31fde3c1062960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1612277330660-noauth.jpeg
paper_authors.fullname: Nandan Thakur
paper_authors.type: user
paper_authors.user: nthakur
paper_authors._id: 6801a85879ba651f02e104f6
paper_authors.name: Jimmy Lin
paper_authors._id: 6801a85879ba651f02e104f7
paper_authors.name: Sam Havens
paper_authors._id: 6801a85879ba651f02e104f8
paper_authors.name: Michael Carbin
paper_authors._id: 6801a85879ba651f02e104f9
paper_authors.name: Omar Khattab
paper_authors._id: 6801a85879ba651f02e104fa
paper_authors.name: Andrew Drozdov
paper_title: FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on
    Technical Documents
paper_summary: We introduce FreshStack, a reusable framework for automatically building
  information retrieval (IR) evaluation benchmarks from community-asked questions
  and answers. FreshStack conducts the following steps: (1) automatic corpus
  collection from code and technical documentation, (2) nugget generation from
  community-asked questions and answers, and (3) nugget-level support, retrieving
  documents using a fusion of retrieval techniques and hybrid architectures. We
  use FreshStack to build five datasets on fast-growing, recent, and niche topics
  to ensure the tasks are sufficiently challenging. On FreshStack, existing
  retrieval models, when applied out-of-the-box, significantly underperform
  oracle approaches on all five topics, denoting plenty of headroom to improve IR
  quality. In addition, we identify cases where rerankers do not clearly improve
  first-stage retrieval accuracy (two out of five topics). We hope that
  FreshStack will facilitate future work toward constructing realistic, scalable,
  and uncontaminated IR and RAG evaluation benchmarks. FreshStack datasets are
  available at: https://fresh-stack.github.io.
paper_ai_summary: FreshStack framework builds IR evaluation benchmarks using community Q&A, demonstrating significant opportunities for improving retrieval models and identifying limitations of reranking techniques.
paper_ai_keywords: information retrieval (IR) evaluation benchmarks automatic corpus collection nugget generation hybrid architectures first-stage retrieval rerankers RAG evaluation benchmarks