title: GNFactor: Multi-Task Real Robot Learning with Generalizable Neural
    Feature Fields
summary: It is a long-standing problem in robotics to develop agents capable of
  executing diverse manipulation tasks from visual observations in unstructured
  real-world environments. To achieve this goal, the robot needs to have a
  comprehensive understanding of the 3D structure and semantics of the scene. In
  this work, we present GNFactor, a visual behavior cloning agent for
  multi-task robotic manipulation with Generalizable Neural
  feature Fields. GNFactor jointly optimizes a generalizable neural
  field (GNF) as a reconstruction module and a Perceiver Transformer as a
  decision-making module, leveraging a shared deep 3D voxel representation. To
  incorporate semantics in 3D, the reconstruction module utilizes a
  vision-language foundation model (e.g., Stable Diffusion) to distill
  rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3
  real robot tasks and perform detailed ablations on 10 RLBench tasks with a
  limited number of demonstrations. We observe a substantial improvement of
  GNFactor over current state-of-the-art methods in seen and unseen tasks,
  demonstrating the strong generalization ability of GNFactor. Our project
  website is https://yanjieze.com/GNFactor/ .
paper_authors._id: 64f1a063a5ca432503ccb3df
paper_authors.name: Yanjie Ze
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-01T09:49:10.373Z
paper_authors._id: 63509bc859bfa9a85d4220aa
paper_authors.avatarUrl: /avatars/ca2cc9b87f5ca5cd51606b2f9edf89d0.svg
paper_authors.fullname: Yanjie Ze
paper_authors.type: user
paper_authors.user: yjze
paper_authors._id: 64f1a063a5ca432503ccb3e0
paper_authors.name: Ge Yan
paper_authors._id: 64f1a063a5ca432503ccb3e1
paper_authors.name: Yueh-Hua Wu
paper_authors._id: 64f1a063a5ca432503ccb3e2
paper_authors.name: Annabella Macaluso
paper_authors._id: 64f1a063a5ca432503ccb3e3
paper_authors.name: Yuying Ge
paper_authors._id: 64f1a063a5ca432503ccb3e4
paper_authors.name: Jianglong Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T11:53:57.188Z
paper_authors._id: 634e2696d00bb5d92c3f4e86
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674216878324-634e2696d00bb5d92c3f4e86.jpeg
paper_authors.fullname: Jianglong Ye
paper_authors.type: user
paper_authors.user: jianglongye
paper_authors._id: 64f1a063a5ca432503ccb3e5
paper_authors.name: Nicklas Hansen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T11:54:37.011Z
paper_authors._id: 62d8ab94bf50eca8389bbc01
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d8ab94bf50eca8389bbc01/ftQ86SJH3ISYwGTJWQCcq.png
paper_authors.fullname: Nicklas Hansen
paper_authors.type: user
paper_authors.user: nicklashansen
paper_authors._id: 64f1a063a5ca432503ccb3e6
paper_authors.name: Li Erran Li
paper_authors._id: 64f1a063a5ca432503ccb3e7
paper_authors.name: Xiaolong Wang
paper_title: GNFactor: Multi-Task Real Robot Learning with Generalizable Neural
    Feature Fields
paper_summary: It is a long-standing problem in robotics to develop agents capable of
  executing diverse manipulation tasks from visual observations in unstructured
  real-world environments. To achieve this goal, the robot needs to have a
  comprehensive understanding of the 3D structure and semantics of the scene. In
  this work, we present GNFactor, a visual behavior cloning agent for
  multi-task robotic manipulation with Generalizable Neural
  feature Fields. GNFactor jointly optimizes a generalizable neural
  field (GNF) as a reconstruction module and a Perceiver Transformer as a
  decision-making module, leveraging a shared deep 3D voxel representation. To
  incorporate semantics in 3D, the reconstruction module utilizes a
  vision-language foundation model (e.g., Stable Diffusion) to distill
  rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3
  real robot tasks and perform detailed ablations on 10 RLBench tasks with a
  limited number of demonstrations. We observe a substantial improvement of
  GNFactor over current state-of-the-art methods in seen and unseen tasks,
  demonstrating the strong generalization ability of GNFactor. Our project
  website is https://yanjieze.com/GNFactor/ .
paper_ai_keywords: GNFactor generalizable neural field (GNF) Perceiver Transformer deep 3D voxel representation vision-language foundation model Stable Diffusion RLBench tasks generalization ability