title: BoxingGym: Benchmarking Progress in Automated Experimental Design and
    Model Discovery
summary: Understanding the world and explaining it with scientific theories is a
  central aspiration of artificial intelligence research. Proposing theories,
  designing experiments to test them, and then revising them based on data are
  fundamental to scientific discovery. Despite the significant promise of
  LLM-based scientific agents, no benchmarks systematically test LLM's ability to
  propose scientific models, collect experimental data, and revise them in light
  of new data. We introduce BoxingGym, a benchmark with 10 environments for
  systematically evaluating both experimental design (e.g. collecting data to
  test a scientific theory) and model discovery (e.g. proposing and revising
  scientific theories). To enable tractable and quantitative evaluation, we
  implement each environment as a generative probabilistic model with which a
  scientific agent can run interactive experiments. These probabilistic models
  are drawn from various real-world scientific domains ranging from psychology to
  ecology. To quantitatively evaluate a scientific agent's ability to collect
  informative experimental data, we compute the expected information gain (EIG),
  an information-theoretic quantity which measures how much an experiment reduces
  uncertainty about the parameters of a generative model. A good scientific
  theory is a concise and predictive explanation. Therefore, to quantitatively
  evaluate model discovery, we ask a scientific agent to explain their model and
  then assess whether this explanation enables another scientific agent to make
  reliable predictions about this environment. In addition to this
  explanation-based evaluation, we compute standard model evaluation metrics such
  as prediction errors. We find that current LLMs, such as GPT-4o, struggle with
  both experimental design and model discovery. We find that augmenting the
  LLM-based agent with an explicit statistical model does not reliably improve
  these results.
paper_authors._id: 677b9534c57d565bd3535ff2
paper_authors.name: Kanishk Gandhi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-06T08:33:22.650Z
paper_authors._id: 63e6a880f2e9a8f22c5a1630
paper_authors.avatarUrl: /avatars/53b57690fe052ce6882bbfc87b11567c.svg
paper_authors.fullname: Kanishk Gandhi
paper_authors.type: user
paper_authors.user: obiwan96
paper_authors._id: 677b9534c57d565bd3535ff3
paper_authors.name: Michael Y. Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T08:36:16.321Z
paper_authors._id: 667357d37376c7227771c14c
paper_authors.avatarUrl: /avatars/77253b6f8b1eb97f9ab09097e83436ed.svg
paper_authors.fullname: Michael Y Li
paper_authors.type: user
paper_authors.user: michaelyli
paper_authors._id: 677b9534c57d565bd3535ff4
paper_authors.name: Lyle Goodyear
paper_authors._id: 677b9534c57d565bd3535ff5
paper_authors.name: Louise Li
paper_authors._id: 677b9534c57d565bd3535ff6
paper_authors.name: Aditi Bhaskar
paper_authors._id: 677b9534c57d565bd3535ff7
paper_authors.name: Mohammed Zaman
paper_authors._id: 677b9534c57d565bd3535ff8
paper_authors.name: Noah D. Goodman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T08:36:39.125Z
paper_authors._id: 67321274c1f20c742bcf7a8d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
paper_authors.fullname: Noah D. Goodman
paper_authors.type: user
paper_authors.user: ngoodman
paper_title: BoxingGym: Benchmarking Progress in Automated Experimental Design and
    Model Discovery
paper_summary: Understanding the world and explaining it with scientific theories is a
  central aspiration of artificial intelligence research. Proposing theories,
  designing experiments to test them, and then revising them based on data are
  fundamental to scientific discovery. Despite the significant promise of
  LLM-based scientific agents, no benchmarks systematically test LLM's ability to
  propose scientific models, collect experimental data, and revise them in light
  of new data. We introduce BoxingGym, a benchmark with 10 environments for
  systematically evaluating both experimental design (e.g. collecting data to
  test a scientific theory) and model discovery (e.g. proposing and revising
  scientific theories). To enable tractable and quantitative evaluation, we
  implement each environment as a generative probabilistic model with which a
  scientific agent can run interactive experiments. These probabilistic models
  are drawn from various real-world scientific domains ranging from psychology to
  ecology. To quantitatively evaluate a scientific agent's ability to collect
  informative experimental data, we compute the expected information gain (EIG),
  an information-theoretic quantity which measures how much an experiment reduces
  uncertainty about the parameters of a generative model. A good scientific
  theory is a concise and predictive explanation. Therefore, to quantitatively
  evaluate model discovery, we ask a scientific agent to explain their model and
  then assess whether this explanation enables another scientific agent to make
  reliable predictions about this environment. In addition to this
  explanation-based evaluation, we compute standard model evaluation metrics such
  as prediction errors. We find that current LLMs, such as GPT-4o, struggle with
  both experimental design and model discovery. We find that augmenting the
  LLM-based agent with an explicit statistical model does not reliably improve
  these results.
paper_ai_summary: BoxingGym benchmark evaluates LLMs' ability to design experiments and propose scientific models across various domains using generative probabilistic models and information-theoretic metrics.
paper_ai_keywords: LLM-based scientific agents BoxingGym experimental design model discovery generative probabilistic models expected information gain (EIG) information-theoretic quantity scientific agent explanation-based evaluation model evaluation metrics prediction errors