title: DeepResearchGym: A Free, Transparent, and Reproducible Evaluation
    Sandbox for Deep Research
summary: Deep research systems represent an emerging class of agentic information
  retrieval methods that generate comprehensive and well-supported reports to
  complex queries. However, most existing frameworks rely on dynamic commercial
  search APIs, which pose reproducibility and transparency challenges in addition
  to their cost. To address these limitations, we introduce DeepResearchGym, an
  open-source sandbox that combines a reproducible search API with a rigorous
  evaluation protocol for benchmarking deep research systems. The API indexes
  large-scale public web corpora, namely ClueWeb22 and FineWeb, using a
  state-of-the-art dense retriever and approximate nearest neighbor search via
  DiskANN. It achieves lower latency than popular commercial APIs while ensuring
  stable document rankings across runs, and is freely available for research use.
  To evaluate deep research systems' outputs, we extend the Researchy Questions
  benchmark with automatic metrics through LLM-as-a-judge assessments to measure
  alignment with users' information needs, retrieval faithfulness, and report
  quality. Experimental results show that systems integrated with DeepResearchGym
  achieve performance comparable to those using commercial APIs, with performance
  rankings remaining consistent across evaluation metrics. A human evaluation
  study further confirms that our automatic protocol aligns with human
  preferences, validating the framework's ability to help support controlled
  assessment of deep research systems. Our code and API documentation are
  available at https://www.deepresearchgym.ai.
paper_authors._id: 6837bc8d0b39c9653de4d06f
paper_authors.name: João Coelho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:42:45.049Z
paper_authors._id: 6244451c9fdefb55a0b900cc
paper_authors.avatarUrl: /avatars/ca2b46ddb5d905501d827920582b5438.svg
paper_authors.fullname: Joao Coelho
paper_authors.type: user
paper_authors.user: jmvcoelho
paper_authors._id: 6837bc8d0b39c9653de4d070
paper_authors.name: Jingjie Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:18:58.772Z
paper_authors._id: 6310a812a23f0327bce68778
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a812a23f0327bce68778/dp9JherZV-RYiSRKnvxsh.jpeg
paper_authors.fullname: Ethan Ning
paper_authors.type: user
paper_authors.user: ethanning
paper_authors._id: 6837bc8d0b39c9653de4d071
paper_authors.name: Jingyuan He
paper_authors._id: 6837bc8d0b39c9653de4d072
paper_authors.name: Kangrui Mao
paper_authors._id: 6837bc8d0b39c9653de4d073
paper_authors.name: Abhijay Paladugu
paper_authors._id: 6837bc8d0b39c9653de4d074
paper_authors.name: Pranav Setlur
paper_authors._id: 6837bc8d0b39c9653de4d075
paper_authors.name: Jiahe Jin
paper_authors._id: 6837bc8d0b39c9653de4d076
paper_authors.name: Jamie Callan
paper_authors._id: 6837bc8d0b39c9653de4d077
paper_authors.name: João Magalhães
paper_authors._id: 6837bc8d0b39c9653de4d078
paper_authors.name: Bruno Martins
paper_authors._id: 6837bc8d0b39c9653de4d079
paper_authors.name: Chenyan Xiong
paper_title: DeepResearchGym: A Free, Transparent, and Reproducible Evaluation
    Sandbox for Deep Research
paper_summary: Deep research systems represent an emerging class of agentic information
  retrieval methods that generate comprehensive and well-supported reports to
  complex queries. However, most existing frameworks rely on dynamic commercial
  search APIs, which pose reproducibility and transparency challenges in addition
  to their cost. To address these limitations, we introduce DeepResearchGym, an
  open-source sandbox that combines a reproducible search API with a rigorous
  evaluation protocol for benchmarking deep research systems. The API indexes
  large-scale public web corpora, namely ClueWeb22 and FineWeb, using a
  state-of-the-art dense retriever and approximate nearest neighbor search via
  DiskANN. It achieves lower latency than popular commercial APIs while ensuring
  stable document rankings across runs, and is freely available for research use.
  To evaluate deep research systems' outputs, we extend the Researchy Questions
  benchmark with automatic metrics through LLM-as-a-judge assessments to measure
  alignment with users' information needs, retrieval faithfulness, and report
  quality. Experimental results show that systems integrated with DeepResearchGym
  achieve performance comparable to those using commercial APIs, with performance
  rankings remaining consistent across evaluation metrics. A human evaluation
  study further confirms that our automatic protocol aligns with human
  preferences, validating the framework's ability to help support controlled
  assessment of deep research systems. Our code and API documentation are
  available at https://www.deepresearchgym.ai.
paper_ai_summary: DeepResearchGym provides an open-source evaluation framework for deep research systems using a reproducible search API and LLM-as-a-judge assessments.
paper_ai_keywords: agentic information retrieval deep research systems search API reproducibility transparency open-source sandbox ClueWeb22 FineWeb dense retriever approximate nearest neighbor search DiskANN Researchy Questions benchmark LLM-as-a-judge retrieval faithfulness report quality human evaluation