WALL-E: World Alignment by Rule Learning Improves World Model-based LLM
  Agents
Can large language models (LLMs) directly serve as powerful world models for
model-based agents? While the gaps between the prior knowledge of LLMs and the
specified environment's dynamics do exist, our study reveals that the gaps can
be bridged by aligning an LLM with its deployed environment and such "world
alignment" can be efficiently achieved by rule learning on LLMs. Given the rich
prior knowledge of LLMs, only a few additional rules suffice to align LLM
predictions with the specified environment dynamics. To this end, we propose a
neurosymbolic approach to learn these rules gradient-free through LLMs, by
inducing, updating, and pruning rules based on comparisons of agent-explored
trajectories and world model predictions. The resulting world model is composed
of the LLM and the learned rules. Our embodied LLM agent "WALL-E" is built upon
model-predictive control (MPC). By optimizing look-ahead actions based on the
precise world model, MPC significantly improves exploration and learning
efficiency. Compared to existing LLM agents, WALL-E's reasoning only requires a
few principal rules rather than verbose buffered trajectories being included in
the LLM input. On open-world challenges in Minecraft and ALFWorld, WALL-E
achieves higher success rates than existing methods, with lower costs on
replanning time and the number of tokens used for reasoning. In Minecraft,
WALL-E exceeds baselines by 15-30% in success rate while costing 8-20 fewer
replanning rounds and only 60-80% of tokens. In ALFWorld, its success rate
surges to a new record high of 95% only after 6 iterations.
670906442f30a8b4755bfd36
Siyu Zhou
claimed_verified
2024-10-11T11:08:51.027Z
6588f504d2ea3f329443e768
/avatars/ab8f3bc0844b02c5e26a450bc3ee7689.svg
Siyu Zhou
user
Sawyer99
670906442f30a8b4755bfd37
Tianyi Zhou
claimed_verified
2024-10-13T20:16:12.525Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
670906442f30a8b4755bfd38
Yijun Yang
claimed_verified
2024-10-13T20:16:09.783Z
645e553c3b6d85c65e8b0e54
/avatars/1fffc6499b9d65b21a895ca96f03b781.svg
Steven
user
yijunyang
670906442f30a8b4755bfd39
Guodong Long
670906442f30a8b4755bfd3a
Deheng Ye
670906442f30a8b4755bfd3b
Jing Jiang
670906442f30a8b4755bfd3c
Chengqi Zhang
WALL-E: World Alignment by Rule Learning Improves World Model-based LLM
  Agents
Can large language models (LLMs) directly serve as powerful world models for
model-based agents? While the gaps between the prior knowledge of LLMs and the
specified environment's dynamics do exist, our study reveals that the gaps can
be bridged by aligning an LLM with its deployed environment and such "world
alignment" can be efficiently achieved by rule learning on LLMs. Given the rich
prior knowledge of LLMs, only a few additional rules suffice to align LLM
predictions with the specified environment dynamics. To this end, we propose a
neurosymbolic approach to learn these rules gradient-free through LLMs, by
inducing, updating, and pruning rules based on comparisons of agent-explored
trajectories and world model predictions. The resulting world model is composed
of the LLM and the learned rules. Our embodied LLM agent "WALL-E" is built upon
model-predictive control (MPC). By optimizing look-ahead actions based on the
precise world model, MPC significantly improves exploration and learning
efficiency. Compared to existing LLM agents, WALL-E's reasoning only requires a
few principal rules rather than verbose buffered trajectories being included in
the LLM input. On open-world challenges in Minecraft and ALFWorld, WALL-E
achieves higher success rates than existing methods, with lower costs on
replanning time and the number of tokens used for reasoning. In Minecraft,
WALL-E exceeds baselines by 15-30% in success rate while costing 8-20 fewer
replanning rounds and only 60-80% of tokens. In ALFWorld, its success rate
surges to a new record high of 95% only after 6 iterations.
A neurosymbolic approach aligns large language models with their environment through rule learning, improving model-based agent performance in open-world tasks.
large language models (LLMs) world models model-based agents world alignment rule learning neurosymbolic approach model-predictive control (MPC) agent-explored trajectories world model predictions open-world challenges Minecraft ALFWorld replanning time tokens success rate