title: LAB-Bench: Measuring Capabilities of Language Models for Biology
    Research
summary: There is widespread optimism that frontier Large Language Models (LLMs) and
  LLM-augmented systems have the potential to rapidly accelerate scientific
  discovery across disciplines. Today, many benchmarks exist to measure LLM
  knowledge and reasoning on textbook-style science questions, but few if any
  benchmarks are designed to evaluate language model performance on practical
  tasks required for scientific research, such as literature search, protocol
  planning, and data analysis. As a step toward building such benchmarks, we
  introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of
  over 2,400 multiple choice questions for evaluating AI systems on a range of
  practical biology research capabilities, including recall and reasoning over
  literature, interpretation of figures, access and navigation of databases, and
  comprehension and manipulation of DNA and protein sequences. Importantly, in
  contrast to previous scientific benchmarks, we expect that an AI system that
  can achieve consistently high scores on the more difficult LAB-Bench tasks
  would serve as a useful assistant for researchers in areas such as literature
  search and molecular cloning. As an initial assessment of the emergent
  scientific task capabilities of frontier language models, we measure
  performance of several against our benchmark and report results compared to
  human expert biology researchers. We will continue to update and expand
  LAB-Bench over time, and expect it to serve as a useful tool in the development
  of automated research systems going forward. A public subset of LAB-Bench is
  available for use at the following URL:
  https://huggingface.co/datasets/futurehouse/lab-bench
paper_authors._id: 6695cb23d4ca2767b9d5a15f
paper_authors.name: Jon M. Laurent
paper_authors._id: 6695cb23d4ca2767b9d5a160
paper_authors.name: Joseph D. Janizek
paper_authors._id: 6695cb23d4ca2767b9d5a161
paper_authors.name: Michael Ruzo
paper_authors._id: 6695cb23d4ca2767b9d5a162
paper_authors.name: Michaela M. Hinks
paper_authors._id: 6695cb23d4ca2767b9d5a163
paper_authors.name: Michael J. Hammerling
paper_authors._id: 6695cb23d4ca2767b9d5a164
paper_authors.name: Siddharth Narayanan
paper_authors._id: 6695cb23d4ca2767b9d5a165
paper_authors.name: Manvitha Ponnapati
paper_authors._id: 6695cb23d4ca2767b9d5a166
paper_authors.name: Andrew D. White
paper_authors._id: 6695cb23d4ca2767b9d5a167
paper_authors.name: Samuel G. Rodriques
paper_title: LAB-Bench: Measuring Capabilities of Language Models for Biology
    Research
paper_summary: There is widespread optimism that frontier Large Language Models (LLMs) and
  LLM-augmented systems have the potential to rapidly accelerate scientific
  discovery across disciplines. Today, many benchmarks exist to measure LLM
  knowledge and reasoning on textbook-style science questions, but few if any
  benchmarks are designed to evaluate language model performance on practical
  tasks required for scientific research, such as literature search, protocol
  planning, and data analysis. As a step toward building such benchmarks, we
  introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of
  over 2,400 multiple choice questions for evaluating AI systems on a range of
  practical biology research capabilities, including recall and reasoning over
  literature, interpretation of figures, access and navigation of databases, and
  comprehension and manipulation of DNA and protein sequences. Importantly, in
  contrast to previous scientific benchmarks, we expect that an AI system that
  can achieve consistently high scores on the more difficult LAB-Bench tasks
  would serve as a useful assistant for researchers in areas such as literature
  search and molecular cloning. As an initial assessment of the emergent
  scientific task capabilities of frontier language models, we measure
  performance of several against our benchmark and report results compared to
  human expert biology researchers. We will continue to update and expand
  LAB-Bench over time, and expect it to serve as a useful tool in the development
  of automated research systems going forward. A public subset of LAB-Bench is
  available for use at the following URL:
  https://huggingface.co/datasets/futurehouse/lab-bench
paper_ai_summary: LAB-Bench, a new dataset, evaluates AI systems on practical biology research tasks, providing a benchmark for assessing scientific reasoning and practical capabilities beyond textbook-style questions.
paper_ai_keywords: Large Language Models LLM-augmented systems literature search protocol planning data analysis multiple choice questions recall reasoning over literature interpretation of figures database access and navigation DNA and protein sequence comprehension automated research systems