title: Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable
    Robot Manipulation
summary: How can robot manipulation policies generalize to novel tasks involving
  unseen object types and new motions? In this paper, we provide a solution in
  terms of predicting motion information from web data through human video
  generation and conditioning a robot policy on the generated video. Instead of
  attempting to scale robot data collection which is expensive, we show how we
  can leverage video generation models trained on easily available web data, for
  enabling generalization. Our approach Gen2Act casts language-conditioned
  manipulation as zero-shot human video generation followed by execution with a
  single policy conditioned on the generated video. To train the policy, we use
  an order of magnitude less robot interaction data compared to what the video
  prediction model was trained on. Gen2Act doesn't require fine-tuning the video
  model at all and we directly use a pre-trained model for generating human
  videos. Our results on diverse real-world scenarios show how Gen2Act enables
  manipulating unseen object types and performing novel motions for tasks not
  present in the robot data. Videos are at https://homangab.github.io/gen2act/
paper_authors._id: 66f374b30b03bd54fcf2f3a5
paper_authors.name: Homanga Bharadhwaj
paper_authors._id: 66f374b30b03bd54fcf2f3a6
paper_authors.name: Debidatta Dwibedi
paper_authors._id: 66f374b30b03bd54fcf2f3a7
paper_authors.name: Abhinav Gupta
paper_authors._id: 66f374b30b03bd54fcf2f3a8
paper_authors.name: Shubham Tulsiani
paper_authors._id: 66f374b30b03bd54fcf2f3a9
paper_authors.name: Carl Doersch
paper_authors._id: 66f374b30b03bd54fcf2f3aa
paper_authors.name: Ted Xiao
paper_authors._id: 66f374b30b03bd54fcf2f3ab
paper_authors.name: Dhruv Shah
paper_authors._id: 66f374b30b03bd54fcf2f3ac
paper_authors.name: Fei Xia
paper_authors._id: 66f374b30b03bd54fcf2f3ad
paper_authors.name: Dorsa Sadigh
paper_authors._id: 66f374b30b03bd54fcf2f3ae
paper_authors.name: Sean Kirmani
paper_title: Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable
    Robot Manipulation
paper_summary: How can robot manipulation policies generalize to novel tasks involving
  unseen object types and new motions? In this paper, we provide a solution in
  terms of predicting motion information from web data through human video
  generation and conditioning a robot policy on the generated video. Instead of
  attempting to scale robot data collection which is expensive, we show how we
  can leverage video generation models trained on easily available web data, for
  enabling generalization. Our approach Gen2Act casts language-conditioned
  manipulation as zero-shot human video generation followed by execution with a
  single policy conditioned on the generated video. To train the policy, we use
  an order of magnitude less robot interaction data compared to what the video
  prediction model was trained on. Gen2Act doesn't require fine-tuning the video
  model at all and we directly use a pre-trained model for generating human
  videos. Our results on diverse real-world scenarios show how Gen2Act enables
  manipulating unseen object types and performing novel motions for tasks not
  present in the robot data. Videos are at https://homangab.github.io/gen2act/
paper_ai_keywords: language-conditioned manipulation zero-shot human video generation video generation models robot manipulation policies robot interaction data pre-trained model