title: Compositional Foundation Models for Hierarchical Planning
summary: To make effective decisions in novel environments with long-horizon goals, it
  is crucial to engage in hierarchical reasoning across spatial and temporal
  scales. This entails planning abstract subgoal sequences, visually reasoning
  about the underlying plans, and executing actions in accordance with the
  devised plan through visual-motor control. We propose Compositional Foundation
  Models for Hierarchical Planning (HiP), a foundation model which leverages
  multiple expert foundation model trained on language, vision and action data
  individually jointly together to solve long-horizon tasks. We use a large
  language model to construct symbolic plans that are grounded in the environment
  through a large video diffusion model. Generated video plans are then grounded
  to visual-motor control, through an inverse dynamics model that infers actions
  from generated videos. To enable effective reasoning within this hierarchy, we
  enforce consistency between the models via iterative refinement. We illustrate
  the efficacy and adaptability of our approach in three different long-horizon
  table-top manipulation tasks.
paper_authors._id: 6507aba9a226ecc608f0d222
paper_authors.name: Anurag Ajay
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-09-18T01:45:18.213Z
paper_authors._id: 64a7345b23622f7f188cea55
paper_authors.avatarUrl: /avatars/069162d6d85bb0d9e810c24a867aff7b.svg
paper_authors.fullname: Anurag Ajay
paper_authors.type: user
paper_authors.user: anuragajay
paper_authors._id: 6507aba9a226ecc608f0d223
paper_authors.name: Seungwook Han
paper_authors._id: 6507aba9a226ecc608f0d224
paper_authors.name: Yilun Du
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-09-18T01:45:18.213Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 6507aba9a226ecc608f0d225
paper_authors.name: Shaung Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-04T13:56:14.905Z
paper_authors._id: 6520d6db2a16045c092b3b36
paper_authors.avatarUrl: /avatars/dab34f141a1aef39d00c789ff85e729f.svg
paper_authors.fullname: Seungwook Han
paper_authors.type: user
paper_authors.user: hanseungwook
paper_authors._id: 6507aba9a226ecc608f0d226
paper_authors.name: Abhi Gupta
paper_authors._id: 6507aba9a226ecc608f0d227
paper_authors.name: Tommi Jaakkola
paper_authors._id: 6507aba9a226ecc608f0d228
paper_authors.name: Josh Tenenbaum
paper_authors._id: 6507aba9a226ecc608f0d229
paper_authors.name: Leslie Kaelbling
paper_authors._id: 6507aba9a226ecc608f0d22a
paper_authors.name: Akash Srivastava
paper_authors._id: 6507aba9a226ecc608f0d22b
paper_authors.name: Pulkit Agrawal
paper_title: Compositional Foundation Models for Hierarchical Planning
paper_summary: To make effective decisions in novel environments with long-horizon goals, it
  is crucial to engage in hierarchical reasoning across spatial and temporal
  scales. This entails planning abstract subgoal sequences, visually reasoning
  about the underlying plans, and executing actions in accordance with the
  devised plan through visual-motor control. We propose Compositional Foundation
  Models for Hierarchical Planning (HiP), a foundation model which leverages
  multiple expert foundation model trained on language, vision and action data
  individually jointly together to solve long-horizon tasks. We use a large
  language model to construct symbolic plans that are grounded in the environment
  through a large video diffusion model. Generated video plans are then grounded
  to visual-motor control, through an inverse dynamics model that infers actions
  from generated videos. To enable effective reasoning within this hierarchy, we
  enforce consistency between the models via iterative refinement. We illustrate
  the efficacy and adaptability of our approach in three different long-horizon
  table-top manipulation tasks.
paper_ai_keywords: hierarchical reasoning subgoal sequences visual reasoning visual-motor control foundation model expert foundation model language data vision data action data symbolic plans video diffusion model inverse dynamics model iterative refinement long-horizon tasks table-top manipulation tasks