title: Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with
    100+ NLP Researchers
summary: Recent advancements in large language models (LLMs) have sparked optimism
  about their potential to accelerate scientific discovery, with a growing number
  of works proposing research agents that autonomously generate and validate new
  ideas. Despite this, no evaluations have shown that LLM systems can take the
  very first step of producing novel, expert-level ideas, let alone perform the
  entire research process. We address this by establishing an experimental design
  that evaluates research idea generation while controlling for confounders and
  performs the first head-to-head comparison between expert NLP researchers and
  an LLM ideation agent. By recruiting over 100 NLP researchers to write novel
  ideas and blind reviews of both LLM and human ideas, we obtain the first
  statistically significant conclusion on current LLM capabilities for research
  ideation: we find LLM-generated ideas are judged as more novel (p < 0.05) than
  human expert ideas while being judged slightly weaker on feasibility. Studying
  our agent baselines closely, we identify open problems in building and
  evaluating research agents, including failures of LLM self-evaluation and their
  lack of diversity in generation. Finally, we acknowledge that human judgements
  of novelty can be difficult, even by experts, and propose an end-to-end study
  design which recruits researchers to execute these ideas into full projects,
  enabling us to study whether these novelty and feasibility judgements result in
  meaningful differences in research outcome.
paper_authors._id: 66df43c2cf19a8918418b1c1
paper_authors.name: Chenglei Si
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:22:12.135Z
paper_authors._id: 6144c0aa8a3e54bb3ecef8ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631895645405-noauth.jpeg
paper_authors.fullname: Chenglei Si
paper_authors.type: user
paper_authors.user: CLS
paper_authors._id: 66df43c2cf19a8918418b1c2
paper_authors.name: Diyi Yang
paper_authors._id: 66df43c2cf19a8918418b1c3
paper_authors.name: Tatsunori Hashimoto
paper_title: Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with
    100+ NLP Researchers
paper_summary: Recent advancements in large language models (LLMs) have sparked optimism
  about their potential to accelerate scientific discovery, with a growing number
  of works proposing research agents that autonomously generate and validate new
  ideas. Despite this, no evaluations have shown that LLM systems can take the
  very first step of producing novel, expert-level ideas, let alone perform the
  entire research process. We address this by establishing an experimental design
  that evaluates research idea generation while controlling for confounders and
  performs the first head-to-head comparison between expert NLP researchers and
  an LLM ideation agent. By recruiting over 100 NLP researchers to write novel
  ideas and blind reviews of both LLM and human ideas, we obtain the first
  statistically significant conclusion on current LLM capabilities for research
  ideation: we find LLM-generated ideas are judged as more novel (p < 0.05) than
  human expert ideas while being judged slightly weaker on feasibility. Studying
  our agent baselines closely, we identify open problems in building and
  evaluating research agents, including failures of LLM self-evaluation and their
  lack of diversity in generation. Finally, we acknowledge that human judgements
  of novelty can be difficult, even by experts, and propose an end-to-end study
  design which recruits researchers to execute these ideas into full projects,
  enabling us to study whether these novelty and feasibility judgements result in
  meaningful differences in research outcome.
paper_ai_summary: LLM-generated research ideas are perceived as more novel than those from human experts but are deemed slightly less feasible, based on blind evaluations by NLP researchers.
paper_ai_keywords: large language models LLMs research agents NLP researchers research ideation self-evaluation diversity in generation feasibility novelty