ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and
  Vision-Language Models
Integrating Large Language Models with symbolic planners is a promising
direction for obtaining verifiable and grounded plans compared to planning in
natural language, with recent works extending this idea to visual domains using
Vision-Language Models (VLMs). However, rigorous comparison between
VLM-grounded symbolic approaches and methods that plan directly with a VLM has
been hindered by a lack of common environments, evaluation protocols and model
coverage. We introduce ViPlan, the first open-source benchmark for Visual
Planning with symbolic predicates and VLMs. ViPlan features a series of
increasingly challenging tasks in two domains: a visual variant of the classic
Blocksworld planning problem and a simulated household robotics environment. We
benchmark nine open-source VLM families across multiple sizes, along with
selected closed models, evaluating both VLM-grounded symbolic planning and
using the models directly to propose actions. We find symbolic planning to
outperform direct VLM planning in Blocksworld, where accurate image grounding
is crucial, whereas the opposite is true in the household robotics tasks, where
commonsense knowledge and the ability to recover from errors are beneficial.
Finally, we show that across most models and methods, there is no significant
benefit to using Chain-of-Thought prompting, suggesting that current VLMs still
struggle with visual reasoning.
682c389bc19ea9cd7d822b5c
Matteo Merler
admin_assigned
2025-05-20T08:39:09.385Z
644555c72d91b15b4c7ebd1c
https://cdn-avatars.huggingface.co/v1/production/uploads/644555c72d91b15b4c7ebd1c/28zmmIkLHUUiQXQ3RQlPM.jpeg
Matteo Merler
user
merlerm
682c389bc19ea9cd7d822b5d
Nicola Dainese
admin_assigned
2025-05-20T08:39:30.830Z
6382346663e3fab40c8c66f9
/avatars/bcdba23952ff465b8488bd68a61005e5.svg
Nicola Dainese
user
dainesn1
682c389bc19ea9cd7d822b5e
Minttu Alakuijala
claimed_verified
2025-05-20T08:23:26.845Z
64c268c4b57937d56d65e163
/avatars/bf290d81983703e457e709fec1a2300e.svg
Minttu Alakuijala
user
minttusofia
682c389bc19ea9cd7d822b5f
Giovanni Bonetta
admin_assigned
2025-05-20T08:39:37.830Z
60d9e5b71fa5d458da777550
https://cdn-avatars.huggingface.co/v1/production/uploads/1676481484908-60d9e5b71fa5d458da777550.png
Giovanni Bonetta
user
giobin
682c389bc19ea9cd7d822b60
Pietro Ferrazzi
admin_assigned
2025-05-20T08:40:03.086Z
6512b0e48c0f10eedb296c65
/avatars/46cf7ddf5f94468b7cf39a787741ca2d.svg
Pietro Ferrazzi
user
Pietroferr
682c389bc19ea9cd7d822b61
Yu Tian
682c389bc19ea9cd7d822b62
Bernardo Magnini
admin_assigned
2025-05-20T08:40:11.231Z
666d3b2bb955b0e655473ffe
/avatars/de83261afe1655b857a34f3c9f1d0bcc.svg
Bernardo Magnini
user
magnini
682c389bc19ea9cd7d822b63
Pekka Marttinen
ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and
  Vision-Language Models
Integrating Large Language Models with symbolic planners is a promising
direction for obtaining verifiable and grounded plans compared to planning in
natural language, with recent works extending this idea to visual domains using
Vision-Language Models (VLMs). However, rigorous comparison between
VLM-grounded symbolic approaches and methods that plan directly with a VLM has
been hindered by a lack of common environments, evaluation protocols and model
coverage. We introduce ViPlan, the first open-source benchmark for Visual
Planning with symbolic predicates and VLMs. ViPlan features a series of
increasingly challenging tasks in two domains: a visual variant of the classic
Blocksworld planning problem and a simulated household robotics environment. We
benchmark nine open-source VLM families across multiple sizes, along with
selected closed models, evaluating both VLM-grounded symbolic planning and
using the models directly to propose actions. We find symbolic planning to
outperform direct VLM planning in Blocksworld, where accurate image grounding
is crucial, whereas the opposite is true in the household robotics tasks, where
commonsense knowledge and the ability to recover from errors are beneficial.
Finally, we show that across most models and methods, there is no significant
benefit to using Chain-of-Thought prompting, suggesting that current VLMs still
struggle with visual reasoning.
ViPlan is an open-source benchmark comparing symbolic planning with Vision-Language Models and direct action proposals in visual domains, finding that symbolic planning excels in tasks requiring accurate image grounding.
Vision-Language Models VLMs symbolic planning Chain-of-Thought prompting Blocksworld visual planning household robotics image grounding commonsense knowledge error recovery visual reasoning