RVT: Robotic View Transformer for 3D Object Manipulation
For 3D object manipulation, methods that build an explicit 3D representation
perform better than those relying only on camera images. But using explicit 3D
representations like voxels comes at large computing cost, adversely affecting
scalability. In this work, we propose RVT, a multi-view transformer for 3D
manipulation that is both scalable and accurate. Some key features of RVT are
an attention mechanism to aggregate information across views and re-rendering
of the camera input from virtual views around the robot workspace. In
simulations, we find that a single RVT model works well across 18 RLBench tasks
with 249 task variations, achieving 26% higher relative success than the
existing state-of-the-art method (PerAct). It also trains 36X faster than
PerAct for achieving the same performance and achieves 2.3X the inference speed
of PerAct. Further, RVT can perform a variety of manipulation tasks in the real
world with just a few (sim10) demonstrations per task. Visual results, code,
and trained model are provided at https://robotic-view-transformer.github.io/.
649a85db685215f7248a3ffd
Ankit Goyal
claimed_verified
2024-06-15T07:46:59.191Z
666c6f67bc840e6748554a91
/avatars/72121715e14f720e5c1d029b7f00d55d.svg
Ankit Goyal
user
ankgoyal
649a85db685215f7248a3ffe
Jie Xu
claimed_verified
2024-08-16T21:03:02.385Z
66bf785fdccd1164803e0270
/avatars/e762628f8b8f18cefd6a8ed3a7810ec9.svg
Jie Xu
user
xujie7979
649a85db685215f7248a3fff
Yijie Guo
649a85db685215f7248a4000
Valts Blukis
649a85db685215f7248a4001
Yu-Wei Chao
649a85db685215f7248a4002
Dieter Fox
RVT: Robotic View Transformer for 3D Object Manipulation
For 3D object manipulation, methods that build an explicit 3D representation
perform better than those relying only on camera images. But using explicit 3D
representations like voxels comes at large computing cost, adversely affecting
scalability. In this work, we propose RVT, a multi-view transformer for 3D
manipulation that is both scalable and accurate. Some key features of RVT are
an attention mechanism to aggregate information across views and re-rendering
of the camera input from virtual views around the robot workspace. In
simulations, we find that a single RVT model works well across 18 RLBench tasks
with 249 task variations, achieving 26% higher relative success than the
existing state-of-the-art method (PerAct). It also trains 36X faster than
PerAct for achieving the same performance and achieves 2.3X the inference speed
of PerAct. Further, RVT can perform a variety of manipulation tasks in the real
world with just a few (sim10) demonstrations per task. Visual results, code,
and trained model are provided at https://robotic-view-transformer.github.io/.
RVT, a multi-view transformer, improves scalability and accuracy in 3D manipulation tasks by aggregating information across views and re-rendering from virtual perspectives, outperforming existing methods with faster training and inference.
multi-view transformer attention mechanism RVT PerAct RLBench tasks virtual views re-rendering 3D manipulation demonstrations