title: IRASim: Learning Interactive Real-Robot Action Simulators
summary: Scalable robot learning in the real world is limited by the cost and safety
  issues of real robots. In addition, rolling out robot trajectories in the real
  world can be time-consuming and labor-intensive. In this paper, we propose to
  learn an interactive real-robot action simulator as an alternative. We
  introduce a novel method, IRASim, which leverages the power of generative
  models to generate extremely realistic videos of a robot arm that executes a
  given action trajectory, starting from an initial given frame. To validate the
  effectiveness of our method, we create a new benchmark, IRASim Benchmark, based
  on three real-robot datasets and perform extensive experiments on the
  benchmark. Results show that IRASim outperforms all the baseline methods and is
  more preferable in human evaluations. We hope that IRASim can serve as an
  effective and scalable approach to enhance robot learning in the real world. To
  promote research for generative real-robot action simulators, we open-source
  code, benchmark, and checkpoints at https: //gen-irasim.github.io.
paper_authors._id: 6679a8667dc34fe17cb367f1
paper_authors.name: Fangqi Zhu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-25T06:22:06.567Z
paper_authors._id: 64d0b110d1fda042b57a3c0b
paper_authors.avatarUrl: /avatars/0cf0050b272d1c8e46e1299de2aea22a.svg
paper_authors.fullname: zhu
paper_authors.type: user
paper_authors.user: fangqi
paper_authors._id: 6679a8667dc34fe17cb367f2
paper_authors.name: Hongtao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:52:17.258Z
paper_authors._id: 667094befc615a62578c7a87
paper_authors.avatarUrl: /avatars/4e645f8cff38b0b6e0d1016cd5a23d89.svg
paper_authors.fullname: Hongtao Wu
paper_authors.type: user
paper_authors.user: danchaoren
paper_authors._id: 6679a8667dc34fe17cb367f3
paper_authors.name: Song Guo
paper_authors._id: 6679a8667dc34fe17cb367f4
paper_authors.name: Yuxiao Liu
paper_authors._id: 6679a8667dc34fe17cb367f5
paper_authors.name: Chilam Cheang
paper_authors._id: 6679a8667dc34fe17cb367f6
paper_authors.name: Tao Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:52:48.429Z
paper_authors._id: 6618de1782c27355a197b47f
paper_authors.avatarUrl: /avatars/ba746df3ae4d264af6184523930fba1c.svg
paper_authors.fullname: Kong
paper_authors.type: user
paper_authors.user: TaoKong
paper_title: IRASim: Learning Interactive Real-Robot Action Simulators
paper_summary: Scalable robot learning in the real world is limited by the cost and safety
  issues of real robots. In addition, rolling out robot trajectories in the real
  world can be time-consuming and labor-intensive. In this paper, we propose to
  learn an interactive real-robot action simulator as an alternative. We
  introduce a novel method, IRASim, which leverages the power of generative
  models to generate extremely realistic videos of a robot arm that executes a
  given action trajectory, starting from an initial given frame. To validate the
  effectiveness of our method, we create a new benchmark, IRASim Benchmark, based
  on three real-robot datasets and perform extensive experiments on the
  benchmark. Results show that IRASim outperforms all the baseline methods and is
  more preferable in human evaluations. We hope that IRASim can serve as an
  effective and scalable approach to enhance robot learning in the real world. To
  promote research for generative real-robot action simulators, we open-source
  code, benchmark, and checkpoints at https: //gen-irasim.github.io.
paper_ai_summary: IRASim utilizes generative models to create realistic robot action simulations, enhancing scalable robot learning by reducing the need for real-world testing and increasing efficiency.
paper_ai_keywords: generative models robot action simulator real-world robot learning benchmark human evaluations