title: Planetarium: A Rigorous Benchmark for Translating Text to Structured
    Planning Languages
summary: Many recent works have explored using language models for planning problems.
  One line of research focuses on translating natural language descriptions of
  planning tasks into structured planning languages, such as the planning domain
  definition language (PDDL). While this approach is promising, accurately
  measuring the quality of generated PDDL code continues to pose significant
  challenges. First, generated PDDL code is typically evaluated using planning
  validators that check whether the problem can be solved with a planner. This
  method is insufficient because a language model might generate valid PDDL code
  that does not align with the natural language description of the task. Second,
  existing evaluation sets often have natural language descriptions of the
  planning task that closely resemble the ground truth PDDL, reducing the
  challenge of the task. To bridge this gap, we introduce \benchmarkName, a
  benchmark designed to evaluate language models' ability to generate PDDL code
  from natural language descriptions of planning tasks. We begin by creating a
  PDDL equivalence algorithm that rigorously evaluates the correctness of PDDL
  code generated by language models by flexibly comparing it against a ground
  truth PDDL. Then, we present a dataset of 132,037 text-to-PDDL pairs across
  13 different tasks, with varying levels of difficulty. Finally, we evaluate
  several API-access and open-weight language models that reveal this task's
  complexity. For example, 87.6% of the PDDL problem descriptions generated by
  GPT-4o are syntactically parseable, 82.2% are valid, solve-able problems,
  but only 35.1% are semantically correct, highlighting the need for a more
  rigorous benchmark for this problem.
paper_authors._id: 6687622ce61e065f5d9ee644
paper_authors.name: Max Zuo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-05T06:55:08.463Z
paper_authors._id: 64fbb457c7f04f7cee8624e0
paper_authors.avatarUrl: /avatars/f0512561780625d9be43f00dfd5cd46d.svg
paper_authors.fullname: Max Zuo
paper_authors.type: user
paper_authors.user: zuom
paper_authors._id: 6687622ce61e065f5d9ee645
paper_authors.name: Francisco Piedrahita Velez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T08:50:51.314Z
paper_authors._id: 64fba1c7fa6446542275ca91
paper_authors.avatarUrl: /avatars/dbb27f2ec7c815aede6483decec63f41.svg
paper_authors.fullname: Francisco J. Piedrahita-Velez
paper_authors.type: user
paper_authors.user: fpiedrah
paper_authors._id: 6687622ce61e065f5d9ee646
paper_authors.name: Xiaochen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T08:50:49.762Z
paper_authors._id: 65eba11c7cca5fa91c44d902
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65eba11c7cca5fa91c44d902/0efmgs_QhixkyX21RxTYa.jpeg
paper_authors.fullname: Jacob Li
paper_authors.type: user
paper_authors.user: jacobli
paper_authors._id: 6687622ce61e065f5d9ee647
paper_authors.name: Michael L. Littman
paper_authors._id: 6687622ce61e065f5d9ee648
paper_authors.name: Stephen H. Bach
paper_title: Planetarium: A Rigorous Benchmark for Translating Text to Structured
    Planning Languages
paper_summary: Many recent works have explored using language models for planning problems.
  One line of research focuses on translating natural language descriptions of
  planning tasks into structured planning languages, such as the planning domain
  definition language (PDDL). While this approach is promising, accurately
  measuring the quality of generated PDDL code continues to pose significant
  challenges. First, generated PDDL code is typically evaluated using planning
  validators that check whether the problem can be solved with a planner. This
  method is insufficient because a language model might generate valid PDDL code
  that does not align with the natural language description of the task. Second,
  existing evaluation sets often have natural language descriptions of the
  planning task that closely resemble the ground truth PDDL, reducing the
  challenge of the task. To bridge this gap, we introduce \benchmarkName, a
  benchmark designed to evaluate language models' ability to generate PDDL code
  from natural language descriptions of planning tasks. We begin by creating a
  PDDL equivalence algorithm that rigorously evaluates the correctness of PDDL
  code generated by language models by flexibly comparing it against a ground
  truth PDDL. Then, we present a dataset of 132,037 text-to-PDDL pairs across
  13 different tasks, with varying levels of difficulty. Finally, we evaluate
  several API-access and open-weight language models that reveal this task's
  complexity. For example, 87.6% of the PDDL problem descriptions generated by
  GPT-4o are syntactically parseable, 82.2% are valid, solve-able problems,
  but only 35.1% are semantically correct, highlighting the need for a more
  rigorous benchmark for this problem.
paper_ai_summary: A benchmark evaluates language models' ability to generate semantically correct PDDL code from natural language descriptions, revealing significant gaps in current solutions.
paper_ai_keywords: planning domain definition language (PDDL) planning validators natural language descriptions PDDL equivalence algorithm text-to-PDDL pairs language models syntactically parseable valid solve-able semantically correct