Compositional Foundation Models for Hierarchical Planning
To make effective decisions in novel environments with long-horizon goals, it
is crucial to engage in hierarchical reasoning across spatial and temporal
scales. This entails planning abstract subgoal sequences, visually reasoning
about the underlying plans, and executing actions in accordance with the
devised plan through visual-motor control. We propose Compositional Foundation
Models for Hierarchical Planning (HiP), a foundation model which leverages
multiple expert foundation model trained on language, vision and action data
individually jointly together to solve long-horizon tasks. We use a large
language model to construct symbolic plans that are grounded in the environment
through a large video diffusion model. Generated video plans are then grounded
to visual-motor control, through an inverse dynamics model that infers actions
from generated videos. To enable effective reasoning within this hierarchy, we
enforce consistency between the models via iterative refinement. We illustrate
the efficacy and adaptability of our approach in three different long-horizon
table-top manipulation tasks.
6507aba9a226ecc608f0d222
Anurag Ajay
extracted_pending
2023-09-18T01:45:18.213Z
64a7345b23622f7f188cea55
/avatars/069162d6d85bb0d9e810c24a867aff7b.svg
Anurag Ajay
user
anuragajay
6507aba9a226ecc608f0d223
Seungwook Han
6507aba9a226ecc608f0d224
Yilun Du
extracted_pending
2023-09-18T01:45:18.213Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
6507aba9a226ecc608f0d225
Shaung Li
extracted_confirmed
2025-03-04T13:56:14.905Z
6520d6db2a16045c092b3b36
/avatars/dab34f141a1aef39d00c789ff85e729f.svg
Seungwook Han
user
hanseungwook
6507aba9a226ecc608f0d226
Abhi Gupta
6507aba9a226ecc608f0d227
Tommi Jaakkola
6507aba9a226ecc608f0d228
Josh Tenenbaum
6507aba9a226ecc608f0d229
Leslie Kaelbling
6507aba9a226ecc608f0d22a
Akash Srivastava
6507aba9a226ecc608f0d22b
Pulkit Agrawal
Compositional Foundation Models for Hierarchical Planning
To make effective decisions in novel environments with long-horizon goals, it
is crucial to engage in hierarchical reasoning across spatial and temporal
scales. This entails planning abstract subgoal sequences, visually reasoning
about the underlying plans, and executing actions in accordance with the
devised plan through visual-motor control. We propose Compositional Foundation
Models for Hierarchical Planning (HiP), a foundation model which leverages
multiple expert foundation model trained on language, vision and action data
individually jointly together to solve long-horizon tasks. We use a large
language model to construct symbolic plans that are grounded in the environment
through a large video diffusion model. Generated video plans are then grounded
to visual-motor control, through an inverse dynamics model that infers actions
from generated videos. To enable effective reasoning within this hierarchy, we
enforce consistency between the models via iterative refinement. We illustrate
the efficacy and adaptability of our approach in three different long-horizon
table-top manipulation tasks.
HiP, a compositional foundation model, integrates language, vision, and action models to plan and execute complex, long-horizon manipulation tasks through hierarchical reasoning and iterative refinement.
hierarchical reasoning subgoal sequences symbolic plans video diffusion model inverse dynamics model compositional foundation models Hierarchical Planning (HiP) long-horizon tasks iterative refinement table-top manipulation tasks