Planetarium: A Rigorous Benchmark for Translating Text to Structured
  Planning Languages
Many recent works have explored using language models for planning problems.
One line of research focuses on translating natural language descriptions of
planning tasks into structured planning languages, such as the planning domain
definition language (PDDL). While this approach is promising, accurately
measuring the quality of generated PDDL code continues to pose significant
challenges. First, generated PDDL code is typically evaluated using planning
validators that check whether the problem can be solved with a planner. This
method is insufficient because a language model might generate valid PDDL code
that does not align with the natural language description of the task. Second,
existing evaluation sets often have natural language descriptions of the
planning task that closely resemble the ground truth PDDL, reducing the
challenge of the task. To bridge this gap, we introduce \benchmarkName, a
benchmark designed to evaluate language models' ability to generate PDDL code
from natural language descriptions of planning tasks. We begin by creating a
PDDL equivalence algorithm that rigorously evaluates the correctness of PDDL
code generated by language models by flexibly comparing it against a ground
truth PDDL. Then, we present a dataset of 132,037 text-to-PDDL pairs across
13 different tasks, with varying levels of difficulty. Finally, we evaluate
several API-access and open-weight language models that reveal this task's
complexity. For example, 87.6% of the PDDL problem descriptions generated by
GPT-4o are syntactically parseable, 82.2% are valid, solve-able problems,
but only 35.1% are semantically correct, highlighting the need for a more
rigorous benchmark for this problem.
6687622ce61e065f5d9ee644
Max Zuo
claimed_verified
2024-07-05T06:55:08.463Z
64fbb457c7f04f7cee8624e0
/avatars/f0512561780625d9be43f00dfd5cd46d.svg
Max Zuo
user
zuom
6687622ce61e065f5d9ee645
Francisco Piedrahita Velez
claimed_verified
2024-07-08T08:50:51.314Z
64fba1c7fa6446542275ca91
/avatars/dbb27f2ec7c815aede6483decec63f41.svg
Francisco J. Piedrahita-Velez
user
fpiedrah
6687622ce61e065f5d9ee646
Xiaochen Li
claimed_verified
2024-07-08T08:50:49.762Z
65eba11c7cca5fa91c44d902
https://cdn-avatars.huggingface.co/v1/production/uploads/65eba11c7cca5fa91c44d902/0efmgs_QhixkyX21RxTYa.jpeg
Jacob Li
user
jacobli
6687622ce61e065f5d9ee647
Michael L. Littman
6687622ce61e065f5d9ee648
Stephen H. Bach
Planetarium: A Rigorous Benchmark for Translating Text to Structured
  Planning Languages
Many recent works have explored using language models for planning problems.
One line of research focuses on translating natural language descriptions of
planning tasks into structured planning languages, such as the planning domain
definition language (PDDL). While this approach is promising, accurately
measuring the quality of generated PDDL code continues to pose significant
challenges. First, generated PDDL code is typically evaluated using planning
validators that check whether the problem can be solved with a planner. This
method is insufficient because a language model might generate valid PDDL code
that does not align with the natural language description of the task. Second,
existing evaluation sets often have natural language descriptions of the
planning task that closely resemble the ground truth PDDL, reducing the
challenge of the task. To bridge this gap, we introduce \benchmarkName, a
benchmark designed to evaluate language models' ability to generate PDDL code
from natural language descriptions of planning tasks. We begin by creating a
PDDL equivalence algorithm that rigorously evaluates the correctness of PDDL
code generated by language models by flexibly comparing it against a ground
truth PDDL. Then, we present a dataset of 132,037 text-to-PDDL pairs across
13 different tasks, with varying levels of difficulty. Finally, we evaluate
several API-access and open-weight language models that reveal this task's
complexity. For example, 87.6% of the PDDL problem descriptions generated by
GPT-4o are syntactically parseable, 82.2% are valid, solve-able problems,
but only 35.1% are semantically correct, highlighting the need for a more
rigorous benchmark for this problem.
A benchmark evaluates language models' ability to generate semantically correct PDDL code from natural language descriptions, revealing significant gaps in current solutions.
planning domain definition language (PDDL) planning validators natural language descriptions PDDL equivalence algorithm text-to-PDDL pairs language models syntactically parseable valid solve-able semantically correct