title: Video Language Planning
summary: We are interested in enabling visual planning for complex long-horizon tasks
  in the space of generated videos and language, leveraging recent advances in
  large generative models pretrained on Internet-scale data. To this end, we
  present video language planning (VLP), an algorithm that consists of a tree
  search procedure, where we train (i) vision-language models to serve as both
  policies and value functions, and (ii) text-to-video models as dynamics models.
  VLP takes as input a long-horizon task instruction and current image
  observation, and outputs a long video plan that provides detailed multimodal
  (video and language) specifications that describe how to complete the final
  task. VLP scales with increasing computation budget where more computation time
  results in improved video plans, and is able to synthesize long-horizon video
  plans across different robotics domains: from multi-object rearrangement, to
  multi-camera bi-arm dexterous manipulation. Generated video plans can be
  translated into real robot actions via goal-conditioned policies, conditioned
  on each intermediate frame of the generated video. Experiments show that VLP
  substantially improves long-horizon task success rates compared to prior
  methods on both simulated and real robots (across 3 hardware platforms).
paper_authors._id: 652df61f54c1441c7edb303d
paper_authors.name: Yilun Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:23:28.094Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 652df61f54c1441c7edb303e
paper_authors.name: Mengjiao Yang
paper_authors._id: 652df61f54c1441c7edb303f
paper_authors.name: Pete Florence
paper_authors._id: 652df61f54c1441c7edb3040
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-15T08:04:20.071Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 652df61f54c1441c7edb3041
paper_authors.name: Ayzaan Wahid
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:23:51.752Z
paper_authors._id: 63922fc073bf5173e3631bb7
paper_authors.avatarUrl: /avatars/d672ca1cf5a541ec51f458b88eb9acec.svg
paper_authors.fullname: Ayzaan Wahid
paper_authors.type: user
paper_authors.user: ayzaan
paper_authors._id: 652df61f54c1441c7edb3042
paper_authors.name: Brian Ichter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:23:59.211Z
paper_authors._id: 633a26ab474cfeb1a864dc56
paper_authors.avatarUrl: /avatars/cd40accd894fc78810fc0d5108f413e9.svg
paper_authors.fullname: Brian I
paper_authors.type: user
paper_authors.user: brianichter
paper_authors._id: 652df61f54c1441c7edb3043
paper_authors.name: Pierre Sermanet
paper_authors._id: 652df61f54c1441c7edb3044
paper_authors.name: Tianhe Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:24:20.154Z
paper_authors._id: 64c95527d33e6a7f82b8b911
paper_authors.avatarUrl: /avatars/5efcf5d5a84766bbd3a5ec2354233f1c.svg
paper_authors.fullname: Tianhe Yu
paper_authors.type: user
paper_authors.user: keviny66
paper_authors._id: 652df61f54c1441c7edb3045
paper_authors.name: Pieter Abbeel
paper_authors._id: 652df61f54c1441c7edb3046
paper_authors.name: Joshua B. Tenenbaum
paper_authors._id: 652df61f54c1441c7edb3047
paper_authors.name: Leslie Kaelbling
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:24:36.617Z
paper_authors._id: 649780c6af212acd36eff177
paper_authors.avatarUrl: /avatars/473acf7502774d9ef9b9d1091d211d59.svg
paper_authors.fullname: Leslie Kaelbling
paper_authors.type: user
paper_authors.user: lpklpk
paper_authors._id: 652df61f54c1441c7edb3048
paper_authors.name: Andy Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:25:00.446Z
paper_authors._id: 62d5f9d705461e9459d20b91
paper_authors.avatarUrl: /avatars/120917da2d8587674505163fb960e27d.svg
paper_authors.fullname: Andy Zeng
paper_authors.type: user
paper_authors.user: andyzeng
paper_authors._id: 652df61f54c1441c7edb3049
paper_authors.name: Jonathan Tompson
paper_title: Video Language Planning
paper_summary: We are interested in enabling visual planning for complex long-horizon tasks
  in the space of generated videos and language, leveraging recent advances in
  large generative models pretrained on Internet-scale data. To this end, we
  present video language planning (VLP), an algorithm that consists of a tree
  search procedure, where we train (i) vision-language models to serve as both
  policies and value functions, and (ii) text-to-video models as dynamics models.
  VLP takes as input a long-horizon task instruction and current image
  observation, and outputs a long video plan that provides detailed multimodal
  (video and language) specifications that describe how to complete the final
  task. VLP scales with increasing computation budget where more computation time
  results in improved video plans, and is able to synthesize long-horizon video
  plans across different robotics domains: from multi-object rearrangement, to
  multi-camera bi-arm dexterous manipulation. Generated video plans can be
  translated into real robot actions via goal-conditioned policies, conditioned
  on each intermediate frame of the generated video. Experiments show that VLP
  substantially improves long-horizon task success rates compared to prior
  methods on both simulated and real robots (across 3 hardware platforms).
paper_ai_keywords: video language planning (VLP) tree search procedure vision-language models policies value functions text-to-video models Dynamics models long-horizon task instruction detailed multimodal video and language specifications multi-object rearrangement multi-camera bi-arm dexterous manipulation place-conditioned policies