Text2World: Benchmarking Large Language Models for Symbolic World Model
  Generation
Recently, there has been growing interest in leveraging large language models
(LLMs) to generate symbolic world models from textual descriptions. Although
LLMs have been extensively explored in the context of world modeling, prior
studies encountered several challenges, including evaluation randomness,
dependence on indirect metrics, and a limited domain scope. To address these
limitations, we introduce a novel benchmark, Text2World, based on planning
domain definition language (PDDL), featuring hundreds of diverse domains and
employing multi-criteria, execution-based metrics for a more robust evaluation.
We benchmark current LLMs using Text2World and find that reasoning models
trained with large-scale reinforcement learning outperform others. However,
even the best-performing model still demonstrates limited capabilities in world
modeling. Building on these insights, we examine several promising strategies
to enhance the world modeling capabilities of LLMs, including test-time
scaling, agent training, and more. We hope that Text2World can serve as a
crucial resource, laying the groundwork for future research in leveraging LLMs
as world models. The project page is available at
https://text-to-world.github.io/.
67b5473109afe1f3029835cb
Mengkang Hu
claimed_verified
2025-02-19T09:01:15.592Z
6237df4a5ab9df625fb70c1a
https://cdn-avatars.huggingface.co/v1/production/uploads/6237df4a5ab9df625fb70c1a/rmtSgp-_WvHZBXnzPrXL0.jpeg
Mengkang Hu
user
MengkangHu
67b5473109afe1f3029835cc
Tianxing Chen
67b5473109afe1f3029835cd
Yude Zou
claimed_verified
2025-02-20T09:37:22.709Z
67b4123c9a692ad23daa8a73
https://cdn-avatars.huggingface.co/v1/production/uploads/67b4123c9a692ad23daa8a73/vhkxv0NkM2socQ2tySzqc.jpeg
Yude Zou
user
xdzouyd
67b5473109afe1f3029835ce
Yuheng Lei
67b5473109afe1f3029835cf
Qiguang Chen
claimed_verified
2025-02-19T09:01:12.095Z
636f526a6cd69d9a36ff2b53
/avatars/8f2271a193fcac609d9be270552b5afa.svg
Qiguang Chen
user
LightChen2333
67b5473109afe1f3029835d0
Ming Li
67b5473109afe1f3029835d1
Hongyuan Zhang
67b5473109afe1f3029835d2
Wenqi Shao
67b5473109afe1f3029835d3
Ping Luo
Text2World: Benchmarking Large Language Models for Symbolic World Model
  Generation
Recently, there has been growing interest in leveraging large language models
(LLMs) to generate symbolic world models from textual descriptions. Although
LLMs have been extensively explored in the context of world modeling, prior
studies encountered several challenges, including evaluation randomness,
dependence on indirect metrics, and a limited domain scope. To address these
limitations, we introduce a novel benchmark, Text2World, based on planning
domain definition language (PDDL), featuring hundreds of diverse domains and
employing multi-criteria, execution-based metrics for a more robust evaluation.
We benchmark current LLMs using Text2World and find that reasoning models
trained with large-scale reinforcement learning outperform others. However,
even the best-performing model still demonstrates limited capabilities in world
modeling. Building on these insights, we examine several promising strategies
to enhance the world modeling capabilities of LLMs, including test-time
scaling, agent training, and more. We hope that Text2World can serve as a
crucial resource, laying the groundwork for future research in leveraging LLMs
as world models. The project page is available at
https://text-to-world.github.io/.
Text2World, a novel PDDL-based benchmark, evaluates world modeling capabilities of LLMs using diverse domains and multi-criteria metrics, highlighting the need for enhanced strategies.
large language models symbolic world models Text2World planning domain definition language reinforcement learning test-time scaling agent training