title: Vid2Robot: End-to-end Video-conditioned Policy Learning with
    Cross-Attention Transformers
summary: While large-scale robotic systems typically rely on textual instructions for
  tasks, this work explores a different approach: can robots infer the task
  directly from observing humans? This shift necessitates the robot's ability to
  decode human intent and translate it into executable actions within its
  physical constraints and environment. We introduce Vid2Robot, a novel
  end-to-end video-based learning framework for robots. Given a video
  demonstration of a manipulation task and current visual observations, Vid2Robot
  directly produces robot actions. This is achieved through a unified
  representation model trained on a large dataset of human video and robot
  trajectory. The model leverages cross-attention mechanisms to fuse prompt video
  features to the robot's current state and generate appropriate actions that
  mimic the observed task. To further improve policy performance, we propose
  auxiliary contrastive losses that enhance the alignment between human and robot
  video representations. We evaluate Vid2Robot on real-world robots,
  demonstrating a 20% improvement in performance compared to other
  video-conditioned policies when using human demonstration videos. Additionally,
  our model exhibits emergent capabilities, such as successfully transferring
  observed motions from one object to another, and long-horizon composition, thus
  showcasing its potential for real-world applications. Project website:
  vid2robot.github.io
paper_authors._id: 65fa61200174fa082183ec31
paper_authors.name: Vidhi Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T11:04:39.681Z
paper_authors._id: 6335dd2717cccb7421bb0def
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6335dd2717cccb7421bb0def/yFjtuIjyyh6MY42pKLLjw.png
paper_authors.fullname: Vidhi Jain
paper_authors.type: user
paper_authors.user: vidhij
paper_authors._id: 65fa61200174fa082183ec32
paper_authors.name: Maria Attarian
paper_authors._id: 65fa61200174fa082183ec33
paper_authors.name: Nikhil J Joshi
paper_authors._id: 65fa61200174fa082183ec34
paper_authors.name: Ayzaan Wahid
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:48:51.769Z
paper_authors._id: 63922fc073bf5173e3631bb7
paper_authors.avatarUrl: /avatars/d672ca1cf5a541ec51f458b88eb9acec.svg
paper_authors.fullname: Ayzaan Wahid
paper_authors.type: user
paper_authors.user: ayzaan
paper_authors._id: 65fa61200174fa082183ec35
paper_authors.name: Danny Driess
paper_authors._id: 65fa61200174fa082183ec36
paper_authors.name: Quan Vuong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:49:01.834Z
paper_authors._id: 63e1d6687fbb6ae4d4f4d980
paper_authors.avatarUrl: /avatars/4959319e69d69e03a31ff4be326e602a.svg
paper_authors.fullname: Quan Vuong
paper_authors.type: user
paper_authors.user: quintonx
paper_authors._id: 65fa61200174fa082183ec37
paper_authors.name: Pannag R Sanketi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:49:01.288Z
paper_authors._id: 66b56f73fc66a466e6104bfb
paper_authors.avatarUrl: /avatars/74855aaf580144fafead4b93d6a24a46.svg
paper_authors.fullname: Pannag Sanketi
paper_authors.type: user
paper_authors.user: prsanky
paper_authors._id: 65fa61200174fa082183ec38
paper_authors.name: Pierre Sermanet
paper_authors._id: 65fa61200174fa082183ec39
paper_authors.name: Stefan Welker
paper_authors._id: 65fa61200174fa082183ec3a
paper_authors.name: Christine Chan
paper_authors._id: 65fa61200174fa082183ec3b
paper_authors.name: Igor Gilitschenski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:49:24.394Z
paper_authors._id: 64893f0b9ab28735f1a1ec0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64893f0b9ab28735f1a1ec0b/2lPBjV16l2qLEWOKU2zwR.jpeg
paper_authors.fullname: Igor Gilitschenski
paper_authors.type: user
paper_authors.user: igilitschenski
paper_authors._id: 65fa61200174fa082183ec3c
paper_authors.name: Yonatan Bisk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:49:31.842Z
paper_authors._id: 63180f3c879073cf83f135b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63180f3c879073cf83f135b0/0ycXVKOywlU1hcvQ4MDws.jpeg
paper_authors.fullname: Yonatan Bisk
paper_authors.type: user
paper_authors.user: ybisk
paper_authors._id: 65fa61200174fa082183ec3d
paper_authors.name: Debidatta Dwibedi
paper_title: Vid2Robot: End-to-end Video-conditioned Policy Learning with
    Cross-Attention Transformers
paper_summary: While large-scale robotic systems typically rely on textual instructions for
  tasks, this work explores a different approach: can robots infer the task
  directly from observing humans? This shift necessitates the robot's ability to
  decode human intent and translate it into executable actions within its
  physical constraints and environment. We introduce Vid2Robot, a novel
  end-to-end video-based learning framework for robots. Given a video
  demonstration of a manipulation task and current visual observations, Vid2Robot
  directly produces robot actions. This is achieved through a unified
  representation model trained on a large dataset of human video and robot
  trajectory. The model leverages cross-attention mechanisms to fuse prompt video
  features to the robot's current state and generate appropriate actions that
  mimic the observed task. To further improve policy performance, we propose
  auxiliary contrastive losses that enhance the alignment between human and robot
  video representations. We evaluate Vid2Robot on real-world robots,
  demonstrating a 20% improvement in performance compared to other
  video-conditioned policies when using human demonstration videos. Additionally,
  our model exhibits emergent capabilities, such as successfully transferring
  observed motions from one object to another, and long-horizon composition, thus
  showcasing its potential for real-world applications. Project website:
  vid2robot.github.io
paper_ai_summary: Vid2Robot is a framework that uses video input to infer and execute robotic manipulation tasks by leveraging cross-attention and contrastive losses for improved policy performance and emergent capabilities.
paper_ai_keywords: Vid2Robot cross-attention mechanisms contrastive losses end-to-end video-based learning