Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable
  Robot Manipulation
How can robot manipulation policies generalize to novel tasks involving
unseen object types and new motions? In this paper, we provide a solution in
terms of predicting motion information from web data through human video
generation and conditioning a robot policy on the generated video. Instead of
attempting to scale robot data collection which is expensive, we show how we
can leverage video generation models trained on easily available web data, for
enabling generalization. Our approach Gen2Act casts language-conditioned
manipulation as zero-shot human video generation followed by execution with a
single policy conditioned on the generated video. To train the policy, we use
an order of magnitude less robot interaction data compared to what the video
prediction model was trained on. Gen2Act doesn't require fine-tuning the video
model at all and we directly use a pre-trained model for generating human
videos. Our results on diverse real-world scenarios show how Gen2Act enables
manipulating unseen object types and performing novel motions for tasks not
present in the robot data. Videos are at https://homangab.github.io/gen2act/
66f374b30b03bd54fcf2f3a5
Homanga Bharadhwaj
66f374b30b03bd54fcf2f3a6
Debidatta Dwibedi
66f374b30b03bd54fcf2f3a7
Abhinav Gupta
66f374b30b03bd54fcf2f3a8
Shubham Tulsiani
66f374b30b03bd54fcf2f3a9
Carl Doersch
66f374b30b03bd54fcf2f3aa
Ted Xiao
66f374b30b03bd54fcf2f3ab
Dhruv Shah
66f374b30b03bd54fcf2f3ac
Fei Xia
66f374b30b03bd54fcf2f3ad
Dorsa Sadigh
66f374b30b03bd54fcf2f3ae
Sean Kirmani
Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable
  Robot Manipulation
How can robot manipulation policies generalize to novel tasks involving
unseen object types and new motions? In this paper, we provide a solution in
terms of predicting motion information from web data through human video
generation and conditioning a robot policy on the generated video. Instead of
attempting to scale robot data collection which is expensive, we show how we
can leverage video generation models trained on easily available web data, for
enabling generalization. Our approach Gen2Act casts language-conditioned
manipulation as zero-shot human video generation followed by execution with a
single policy conditioned on the generated video. To train the policy, we use
an order of magnitude less robot interaction data compared to what the video
prediction model was trained on. Gen2Act doesn't require fine-tuning the video
model at all and we directly use a pre-trained model for generating human
videos. Our results on diverse real-world scenarios show how Gen2Act enables
manipulating unseen object types and performing novel motions for tasks not
present in the robot data. Videos are at https://homangab.github.io/gen2act/
Gen2Act enables robotic manipulation in novel tasks using human video generation conditioned on language inputs, reducing the need for extensive robot data collection.
language-conditioned zero-shot human video generation robot policy video prediction model robot interaction data