Visual Planning: Let's Think Only with Images
Recent advancements in Large Language Models (LLMs) and their multimodal
extensions (MLLMs) have substantially enhanced machine reasoning across diverse
tasks. However, these models predominantly rely on pure text as the medium for
both expressing and structuring reasoning, even when visual information is
present. In this work, we argue that language may not always be the most
natural or effective modality for reasoning, particularly in tasks involving
spatial and geometrical information. Motivated by this, we propose a new
paradigm, Visual Planning, which enables planning through purely visual
representations, independent of text. In this paradigm, planning is executed
via sequences of images that encode step-by-step inference in the visual
domain, akin to how humans sketch or visualize future actions. We introduce a
novel reinforcement learning framework, Visual Planning via Reinforcement
Learning (VPRL), empowered by GRPO for post-training large vision models,
leading to substantial improvements in planning in a selection of
representative visual navigation tasks, FrozenLake, Maze, and MiniBehavior. Our
visual planning paradigm outperforms all other planning variants that conduct
reasoning in the text-only space. Our results establish Visual Planning as a
viable and promising alternative to language-based reasoning, opening new
avenues for tasks that benefit from intuitive, image-based inference.
682abb7984695084c1a48eab
Yi Xu
claimed_verified
2025-05-20T07:23:00.135Z
644d55bd53ad80c659395473
https://cdn-avatars.huggingface.co/v1/production/uploads/644d55bd53ad80c659395473/Jdff2RyL6ZGhFSY8kab1L.jpeg
Yi Xu
user
yixu1
682abb7984695084c1a48eac
Chengzu Li
claimed_verified
2025-05-19T13:09:14.188Z
650e9b0288cdfe73a8575923
/avatars/0fc7fcd0776f63ea5f50a310e7def2f5.svg
Chengzu Li
user
chengzu
682abb7984695084c1a48ead
Han Zhou
claimed_verified
2025-05-19T06:42:16.276Z
62b279e92375526ae51a537b
https://cdn-avatars.huggingface.co/v1/production/uploads/62b279e92375526ae51a537b/U2DxDscDjQ6kWh-jMn0IG.jpeg
Han Zhou
user
hzhouml
682abb7984695084c1a48eae
Xingchen Wan
extracted_confirmed
2025-06-19T22:50:53.356Z
65bf213f8467e2a3d6374d4b
/avatars/0194cdba95d7a4c01fbbdd505e384a3d.svg
X Wan
user
masonxw
682abb7984695084c1a48eaf
Caiqi Zhang
admin_assigned
2025-05-19T07:23:48.005Z
63920dfac47e36ddeb8f1864
/avatars/c36cbf7b084d62368312e5c9292e4260.svg
Caiqi Zhang
user
caiqizh
682abb7984695084c1a48eb0
Anna Korhonen
admin_assigned
2025-05-19T07:23:42.059Z
617a6284941993035fbaf299
https://cdn-avatars.huggingface.co/v1/production/uploads/1635410461794-noauth.jpeg
Anna Korhonen
user
akorhonen
682abb7984695084c1a48eb1
Ivan Vulić
admin_assigned
2025-05-19T07:23:36.111Z
6273e70dc8d55dd434bd8e52
/avatars/3483eeda218e95b1eb00c3dc63c7d000.svg
Ivan Vulić
user
ivulic
Visual Planning: Let's Think Only with Images
Recent advancements in Large Language Models (LLMs) and their multimodal
extensions (MLLMs) have substantially enhanced machine reasoning across diverse
tasks. However, these models predominantly rely on pure text as the medium for
both expressing and structuring reasoning, even when visual information is
present. In this work, we argue that language may not always be the most
natural or effective modality for reasoning, particularly in tasks involving
spatial and geometrical information. Motivated by this, we propose a new
paradigm, Visual Planning, which enables planning through purely visual
representations, independent of text. In this paradigm, planning is executed
via sequences of images that encode step-by-step inference in the visual
domain, akin to how humans sketch or visualize future actions. We introduce a
novel reinforcement learning framework, Visual Planning via Reinforcement
Learning (VPRL), empowered by GRPO for post-training large vision models,
leading to substantial improvements in planning in a selection of
representative visual navigation tasks, FrozenLake, Maze, and MiniBehavior. Our
visual planning paradigm outperforms all other planning variants that conduct
reasoning in the text-only space. Our results establish Visual Planning as a
viable and promising alternative to language-based reasoning, opening new
avenues for tasks that benefit from intuitive, image-based inference.
Visual Planning, using sequences of images for planning, outperforms text-only reasoning methods in visual navigation tasks.
Large Language Models multimodal extensions Visual Planning reinforcement learning VPRL GRPO visual navigation tasks FrozenLake Maze MiniBehavior