title: ResearchBench: Benchmarking LLMs in Scientific Discovery via
    Inspiration-Based Task Decomposition
summary: Large language models (LLMs) have demonstrated potential in assisting
  scientific research, yet their ability to discover high-quality research
  hypotheses remains unexamined due to the lack of a dedicated benchmark. To
  address this gap, we introduce the first large-scale benchmark for evaluating
  LLMs with a near-sufficient set of sub-tasks of scientific discovery:
  inspiration retrieval, hypothesis composition, and hypothesis ranking. We
  develop an automated framework that extracts critical components - research
  questions, background surveys, inspirations, and hypotheses - from scientific
  papers across 12 disciplines, with expert validation confirming its accuracy.
  To prevent data contamination, we focus exclusively on papers published in
  2024, ensuring minimal overlap with LLM pretraining data. Our evaluation
  reveals that LLMs perform well in retrieving inspirations, an
  out-of-distribution task, suggesting their ability to surface novel knowledge
  associations. This positions LLMs as "research hypothesis mines", capable of
  facilitating automated scientific discovery by generating innovative hypotheses
  at scale with minimal human intervention.
paper_authors._id: 67e63581e8a16f38741f4baa
paper_authors.name: Yujie Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T08:58:09.374Z
paper_authors._id: 661980b46a43b2760d8d551f
paper_authors.avatarUrl: /avatars/e4743d55303fe3a88688c29dd4c67a69.svg
paper_authors.fullname: Yujie Liu
paper_authors.type: user
paper_authors.user: yujieliu
paper_authors._id: 67e63581e8a16f38741f4bab
paper_authors.name: Zonglin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T08:58:40.651Z
paper_authors._id: 646a11791556443f24b582e9
paper_authors.avatarUrl: /avatars/b54d416ddca2f124492ba51bc4b95fd2.svg
paper_authors.fullname: Zonglin Yang
paper_authors.type: user
paper_authors.user: ZonglinY
paper_authors._id: 67e63581e8a16f38741f4bac
paper_authors.name: Tong Xie
paper_authors._id: 67e63581e8a16f38741f4bad
paper_authors.name: Jinjie Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T08:58:47.888Z
paper_authors._id: 62a7362fd1e7a011fd4e31a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a7362fd1e7a011fd4e31a7/ZY_mwH-sI0o05SHpLqwc7.png
paper_authors.fullname: Jinjie Ni
paper_authors.type: user
paper_authors.user: jinjieni
paper_authors._id: 67e63581e8a16f38741f4bae
paper_authors.name: Ben Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T08:58:54.480Z
paper_authors._id: 64fe3c6b4c8924c4fed6d97b
paper_authors.avatarUrl: /avatars/5e0a49372af19aae9ec5ee84b299d111.svg
paper_authors.fullname: Ben Gao
paper_authors.type: user
paper_authors.user: bgao22182
paper_authors._id: 67e63581e8a16f38741f4baf
paper_authors.name: Yuqiang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T08:59:07.268Z
paper_authors._id: 65cb61ba0390fce629de99d1
paper_authors.avatarUrl: /avatars/9b4b4556ffd8de215dc37b02366d781b.svg
paper_authors.fullname: Yuqiang Li
paper_authors.type: user
paper_authors.user: yuqiangli
paper_authors._id: 67e63581e8a16f38741f4bb0
paper_authors.name: Shixiang Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T08:59:12.700Z
paper_authors._id: 6436403bf3b08e267d9f0329
paper_authors.avatarUrl: /avatars/a5d9c3d47073e71e4cea124d9c17356d.svg
paper_authors.fullname: SHIXIANG TANG
paper_authors.type: user
paper_authors.user: tangshixiang
paper_authors._id: 67e63581e8a16f38741f4bb1
paper_authors.name: Wanli Ouyang
paper_authors._id: 67e63581e8a16f38741f4bb2
paper_authors.name: Erik Cambria
paper_authors._id: 67e63581e8a16f38741f4bb3
paper_authors.name: Dongzhan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T08:59:22.218Z
paper_authors._id: 6538b861613fe158bd581e35
paper_authors.avatarUrl: /avatars/6817dbfe903675721fd227058b0a91ac.svg
paper_authors.fullname: Dongzhan Zhou
paper_authors.type: user
paper_authors.user: schrodingers-tiger
paper_title: ResearchBench: Benchmarking LLMs in Scientific Discovery via
    Inspiration-Based Task Decomposition
paper_summary: Large language models (LLMs) have demonstrated potential in assisting
  scientific research, yet their ability to discover high-quality research
  hypotheses remains unexamined due to the lack of a dedicated benchmark. To
  address this gap, we introduce the first large-scale benchmark for evaluating
  LLMs with a near-sufficient set of sub-tasks of scientific discovery:
  inspiration retrieval, hypothesis composition, and hypothesis ranking. We
  develop an automated framework that extracts critical components - research
  questions, background surveys, inspirations, and hypotheses - from scientific
  papers across 12 disciplines, with expert validation confirming its accuracy.
  To prevent data contamination, we focus exclusively on papers published in
  2024, ensuring minimal overlap with LLM pretraining data. Our evaluation
  reveals that LLMs perform well in retrieving inspirations, an
  out-of-distribution task, suggesting their ability to surface novel knowledge
  associations. This positions LLMs as "research hypothesis mines", capable of
  facilitating automated scientific discovery by generating innovative hypotheses
  at scale with minimal human intervention.
paper_ai_summary: LLMs demonstrate proficiency in inspiration retrieval using a newly developed benchmark, showing potential in automated scientific hypothesis generation.
paper_ai_keywords: large language models LLMs scientific discovery benchmark sub-tasks inspiration retrieval hypothesis composition hypothesis ranking research questions background surveys inspirations hypotheses automated framework out-of-distribution task automated scientific discovery innovative hypotheses