title: RVT: Robotic View Transformer for 3D Object Manipulation
summary: For 3D object manipulation, methods that build an explicit 3D representation
  perform better than those relying only on camera images. But using explicit 3D
  representations like voxels comes at large computing cost, adversely affecting
  scalability. In this work, we propose RVT, a multi-view transformer for 3D
  manipulation that is both scalable and accurate. Some key features of RVT are
  an attention mechanism to aggregate information across views and re-rendering
  of the camera input from virtual views around the robot workspace. In
  simulations, we find that a single RVT model works well across 18 RLBench tasks
  with 249 task variations, achieving 26% higher relative success than the
  existing state-of-the-art method (PerAct). It also trains 36X faster than
  PerAct for achieving the same performance and achieves 2.3X the inference speed
  of PerAct. Further, RVT can perform a variety of manipulation tasks in the real
  world with just a few (sim10) demonstrations per task. Visual results, code,
  and trained model are provided at https://robotic-view-transformer.github.io/.
paper_authors._id: 649a85db685215f7248a3ffd
paper_authors.name: Ankit Goyal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T07:46:59.191Z
paper_authors._id: 666c6f67bc840e6748554a91
paper_authors.avatarUrl: /avatars/72121715e14f720e5c1d029b7f00d55d.svg
paper_authors.fullname: Ankit Goyal
paper_authors.type: user
paper_authors.user: ankgoyal
paper_authors._id: 649a85db685215f7248a3ffe
paper_authors.name: Jie Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-16T21:03:02.385Z
paper_authors._id: 66bf785fdccd1164803e0270
paper_authors.avatarUrl: /avatars/e762628f8b8f18cefd6a8ed3a7810ec9.svg
paper_authors.fullname: Jie Xu
paper_authors.type: user
paper_authors.user: xujie7979
paper_authors._id: 649a85db685215f7248a3fff
paper_authors.name: Yijie Guo
paper_authors._id: 649a85db685215f7248a4000
paper_authors.name: Valts Blukis
paper_authors._id: 649a85db685215f7248a4001
paper_authors.name: Yu-Wei Chao
paper_authors._id: 649a85db685215f7248a4002
paper_authors.name: Dieter Fox
paper_title: RVT: Robotic View Transformer for 3D Object Manipulation
paper_summary: For 3D object manipulation, methods that build an explicit 3D representation
  perform better than those relying only on camera images. But using explicit 3D
  representations like voxels comes at large computing cost, adversely affecting
  scalability. In this work, we propose RVT, a multi-view transformer for 3D
  manipulation that is both scalable and accurate. Some key features of RVT are
  an attention mechanism to aggregate information across views and re-rendering
  of the camera input from virtual views around the robot workspace. In
  simulations, we find that a single RVT model works well across 18 RLBench tasks
  with 249 task variations, achieving 26% higher relative success than the
  existing state-of-the-art method (PerAct). It also trains 36X faster than
  PerAct for achieving the same performance and achieves 2.3X the inference speed
  of PerAct. Further, RVT can perform a variety of manipulation tasks in the real
  world with just a few (sim10) demonstrations per task. Visual results, code,
  and trained model are provided at https://robotic-view-transformer.github.io/.
paper_ai_summary: RVT, a multi-view transformer, improves scalability and accuracy in 3D manipulation tasks by aggregating information across views and re-rendering from virtual perspectives, outperforming existing methods with faster training and inference.
paper_ai_keywords: multi-view transformer attention mechanism RVT PerAct RLBench tasks virtual views re-rendering 3D manipulation demonstrations