title: NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation
    Learning for Neural Radiance Fields
summary: Neural fields excel in computer vision and robotics due to their ability to
  understand the 3D visual world such as inferring semantics, geometry, and
  dynamics. Given the capabilities of neural fields in densely representing a 3D
  scene from 2D images, we ask the question: Can we scale their self-supervised
  pretraining, specifically using masked autoencoders, to generate effective 3D
  representations from posed RGB images. Owing to the astounding success of
  extending transformers to novel data modalities, we employ standard 3D Vision
  Transformers to suit the unique formulation of NeRFs. We leverage NeRF's
  volumetric grid as a dense input to the transformer, contrasting it with other
  3D representations such as pointclouds where the information density can be
  uneven, and the representation is irregular. Due to the difficulty of applying
  masked autoencoders to an implicit representation, such as NeRF, we opt for
  extracting an explicit representation that canonicalizes scenes across domains
  by employing the camera trajectory for sampling. Our goal is made possible by
  masking random patches from NeRF's radiance and density grid and employing a
  standard 3D Swin Transformer to reconstruct the masked patches. In doing so,
  the model can learn the semantic and spatial structure of complete scenes. We
  pretrain this representation at scale on our proposed curated posed-RGB data,
  totaling over 1.8 million images. Once pretrained, the encoder is used for
  effective 3D transfer learning. Our novel self-supervised pretraining for
  NeRFs, NeRF-MAE, scales remarkably well and improves performance on various
  challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining,
  NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF
  scene understanding baselines on Front3D and ScanNet datasets with an absolute
  performance improvement of over 20% AP50 and 8% AP25 for 3D object detection.
paper_authors._id: 66abcff29a9903e19a6c71f4
paper_authors.name: Muhammad Zubair Irshad
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-03T08:25:56.137Z
paper_authors._id: 62e458d33051028b542be2a0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659131510466-noauth.jpeg
paper_authors.fullname: Zubair Irshad
paper_authors.type: user
paper_authors.user: mirshad7
paper_authors._id: 66abcff29a9903e19a6c71f5
paper_authors.name: Sergey Zakharov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T16:02:27.368Z
paper_authors._id: 630fdeaba119d49bc1e77c13
paper_authors.avatarUrl: /avatars/5f5f3403278da36a3deb9df67eeb8c10.svg
paper_authors.fullname: Sergey Zakharov
paper_authors.type: user
paper_authors.user: zakharos
paper_authors._id: 66abcff29a9903e19a6c71f6
paper_authors.name: Vitor Guizilini
paper_authors._id: 66abcff29a9903e19a6c71f7
paper_authors.name: Adrien Gaidon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T16:02:01.674Z
paper_authors._id: 64af547e46a29208dafca2f3
paper_authors.avatarUrl: /avatars/daff86330d644319c10347d230ab2452.svg
paper_authors.fullname: Adrien Gaidon
paper_authors.type: user
paper_authors.user: adriengaidon
paper_authors._id: 66abcff29a9903e19a6c71f8
paper_authors.name: Zsolt Kira
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T16:01:55.140Z
paper_authors._id: 64eeaa32f3ef4a52e2b3f5f6
paper_authors.avatarUrl: /avatars/00ce423bee728a5123cf4bad422b1384.svg
paper_authors.fullname: Zsolt Kira
paper_authors.type: user
paper_authors.user: zsoltkira
paper_authors._id: 66abcff29a9903e19a6c71f9
paper_authors.name: Rares Ambrus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T16:01:28.544Z
paper_authors._id: 633b349b9f5846dc43045bf9
paper_authors.avatarUrl: /avatars/2eef6d250b56ae918144ac4a684bc4bd.svg
paper_authors.fullname: Rares Ambrus
paper_authors.type: user
paper_authors.user: raramb
paper_title: NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation
    Learning for Neural Radiance Fields
paper_summary: Neural fields excel in computer vision and robotics due to their ability to
  understand the 3D visual world such as inferring semantics, geometry, and
  dynamics. Given the capabilities of neural fields in densely representing a 3D
  scene from 2D images, we ask the question: Can we scale their self-supervised
  pretraining, specifically using masked autoencoders, to generate effective 3D
  representations from posed RGB images. Owing to the astounding success of
  extending transformers to novel data modalities, we employ standard 3D Vision
  Transformers to suit the unique formulation of NeRFs. We leverage NeRF's
  volumetric grid as a dense input to the transformer, contrasting it with other
  3D representations such as pointclouds where the information density can be
  uneven, and the representation is irregular. Due to the difficulty of applying
  masked autoencoders to an implicit representation, such as NeRF, we opt for
  extracting an explicit representation that canonicalizes scenes across domains
  by employing the camera trajectory for sampling. Our goal is made possible by
  masking random patches from NeRF's radiance and density grid and employing a
  standard 3D Swin Transformer to reconstruct the masked patches. In doing so,
  the model can learn the semantic and spatial structure of complete scenes. We
  pretrain this representation at scale on our proposed curated posed-RGB data,
  totaling over 1.8 million images. Once pretrained, the encoder is used for
  effective 3D transfer learning. Our novel self-supervised pretraining for
  NeRFs, NeRF-MAE, scales remarkably well and improves performance on various
  challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining,
  NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF
  scene understanding baselines on Front3D and ScanNet datasets with an absolute
  performance improvement of over 20% AP50 and 8% AP25 for 3D object detection.
paper_ai_summary: NeRF-MAE, a self-supervised pretraining approach using masked autoencoders and 3D Swin Transformers, enhances NeRF for effective 3D representation and improves scene understanding across datasets.
paper_ai_keywords: neural fields NeRF masked autoencoders 3D Vision Transformers volumetric grid pointclouds implicit representation explicit representation radiance grid density grid 3D Swin Transformer NeRF-MAE 3D transfer learning Front3D ScanNet 3D object detection