title: This&That: Language-Gesture Controlled Video Generation for Robot
    Planning
summary: We propose a robot learning method for communicating, planning, and executing
  a wide range of tasks, dubbed This&That. We achieve robot planning for general
  tasks by leveraging the power of video generative models trained on
  internet-scale data containing rich physical and semantic context. In this
  work, we tackle three fundamental challenges in video-based planning: 1)
  unambiguous task communication with simple human instructions, 2) controllable
  video generation that respects user intents, and 3) translating visual planning
  into robot actions. We propose language-gesture conditioning to generate
  videos, which is both simpler and clearer than existing language-only methods,
  especially in complex and uncertain environments. We then suggest a behavioral
  cloning design that seamlessly incorporates the video plans. This&That
  demonstrates state-of-the-art effectiveness in addressing the above three
  challenges, and justifies the use of video generation as an intermediate
  representation for generalizable task planning and execution. Project website:
  https://cfeng16.github.io/this-and-that/.
paper_authors._id: 668d4fd76bacbfd13c62055a
paper_authors.name: Boyang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T08:27:15.636Z
paper_authors._id: 64ed876a74d9b58eabc769a4
paper_authors.avatarUrl: /avatars/9cf8af8e6f428b75827458b63d376ee3.svg
paper_authors.fullname: Boyang Wang
paper_authors.type: user
paper_authors.user: HikariDawn
paper_authors._id: 668d4fd76bacbfd13c62055b
paper_authors.name: Nikhil Sridhar
paper_authors._id: 668d4fd76bacbfd13c62055c
paper_authors.name: Chao Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-13T19:28:25.215Z
paper_authors._id: 645ab0b7c266796265baefa4
paper_authors.avatarUrl: /avatars/bdac661996b63c4b2a56881707afa01f.svg
paper_authors.fullname: Chao Feng
paper_authors.type: user
paper_authors.user: chfeng
paper_authors._id: 668d4fd76bacbfd13c62055d
paper_authors.name: Mark Van der Merwe
paper_authors._id: 668d4fd76bacbfd13c62055e
paper_authors.name: Adam Fishman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T17:34:36.934Z
paper_authors._id: 634b84b92cdd783d63177cd4
paper_authors.avatarUrl: /avatars/96ab7f0360db943ad053e913fbc7357e.svg
paper_authors.fullname: Adam Fishman
paper_authors.type: user
paper_authors.user: fishbotics
paper_authors._id: 668d4fd76bacbfd13c62055f
paper_authors.name: Nima Fazeli
paper_authors._id: 668d4fd76bacbfd13c620560
paper_authors.name: Jeong Joon Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T17:34:35.169Z
paper_authors._id: 6517d7f84fadbeb643c68269
paper_authors.avatarUrl: /avatars/ad21288123abcd4488ae1bdcb10b54e7.svg
paper_authors.fullname: JJ Park
paper_authors.type: user
paper_authors.user: jjparkcv
paper_title: This&That: Language-Gesture Controlled Video Generation for Robot
    Planning
paper_summary: We propose a robot learning method for communicating, planning, and executing
  a wide range of tasks, dubbed This&That. We achieve robot planning for general
  tasks by leveraging the power of video generative models trained on
  internet-scale data containing rich physical and semantic context. In this
  work, we tackle three fundamental challenges in video-based planning: 1)
  unambiguous task communication with simple human instructions, 2) controllable
  video generation that respects user intents, and 3) translating visual planning
  into robot actions. We propose language-gesture conditioning to generate
  videos, which is both simpler and clearer than existing language-only methods,
  especially in complex and uncertain environments. We then suggest a behavioral
  cloning design that seamlessly incorporates the video plans. This&That
  demonstrates state-of-the-art effectiveness in addressing the above three
  challenges, and justifies the use of video generation as an intermediate
  representation for generalizable task planning and execution. Project website:
  https://cfeng16.github.io/this-and-that/.
paper_ai_summary: A robot learning method uses video generative models and language-gesture conditioning to plan and execute tasks based on human instructions and visual context.
paper_ai_keywords: video generative models language-gesture conditioning behavioral cloning video-based planning task communication controllable video generation visual planning generalizable task planning execution