FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on
  Technical Documents
We introduce FreshStack, a reusable framework for automatically building
information retrieval (IR) evaluation benchmarks from community-asked questions
and answers. FreshStack conducts the following steps: (1) automatic corpus
collection from code and technical documentation, (2) nugget generation from
community-asked questions and answers, and (3) nugget-level support, retrieving
documents using a fusion of retrieval techniques and hybrid architectures. We
use FreshStack to build five datasets on fast-growing, recent, and niche topics
to ensure the tasks are sufficiently challenging. On FreshStack, existing
retrieval models, when applied out-of-the-box, significantly underperform
oracle approaches on all five topics, denoting plenty of headroom to improve IR
quality. In addition, we identify cases where rerankers do not clearly improve
first-stage retrieval accuracy (two out of five topics). We hope that
FreshStack will facilitate future work toward constructing realistic, scalable,
and uncontaminated IR and RAG evaluation benchmarks. FreshStack datasets are
available at: https://fresh-stack.github.io.
6801a85879ba651f02e104f5
Nandan Thakur
claimed_verified
2025-05-26T08:41:03.081Z
60196690dd31fde3c1062960
https://cdn-avatars.huggingface.co/v1/production/uploads/1612277330660-noauth.jpeg
Nandan Thakur
user
nthakur
6801a85879ba651f02e104f6
Jimmy Lin
6801a85879ba651f02e104f7
Sam Havens
6801a85879ba651f02e104f8
Michael Carbin
6801a85879ba651f02e104f9
Omar Khattab
6801a85879ba651f02e104fa
Andrew Drozdov
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on
  Technical Documents
We introduce FreshStack, a reusable framework for automatically building
information retrieval (IR) evaluation benchmarks from community-asked questions
and answers. FreshStack conducts the following steps: (1) automatic corpus
collection from code and technical documentation, (2) nugget generation from
community-asked questions and answers, and (3) nugget-level support, retrieving
documents using a fusion of retrieval techniques and hybrid architectures. We
use FreshStack to build five datasets on fast-growing, recent, and niche topics
to ensure the tasks are sufficiently challenging. On FreshStack, existing
retrieval models, when applied out-of-the-box, significantly underperform
oracle approaches on all five topics, denoting plenty of headroom to improve IR
quality. In addition, we identify cases where rerankers do not clearly improve
first-stage retrieval accuracy (two out of five topics). We hope that
FreshStack will facilitate future work toward constructing realistic, scalable,
and uncontaminated IR and RAG evaluation benchmarks. FreshStack datasets are
available at: https://fresh-stack.github.io.
FreshStack framework builds IR evaluation benchmarks using community Q&A, demonstrating significant opportunities for improving retrieval models and identifying limitations of reranking techniques.
information retrieval (IR) evaluation benchmarks automatic corpus collection nugget generation hybrid architectures first-stage retrieval rerankers RAG evaluation benchmarks