DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features
<|reference_start|>DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features: We propose DistillNeRF, a self-supervised learning framework addressing the challenge of understanding 3D environments from limited 2D observations in autonomous driving. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs, and is trained self-supervised with differentiable rendering to reconstruct RGB, depth, or feature images. Our first insight is to exploit per-scene optimized Neural Radiance Fields (NeRFs) by generating dense depth and virtual camera targets for training, thereby helping our model to learn 3D geometry from sparse non-overlapping image inputs. Second, to learn a semantically rich 3D representation, we propose distilling features from pre-trained 2D foundation models, such as CLIP or DINOv2, thereby enabling various downstream tasks without the need for costly 3D human annotations. To leverage these two insights, we introduce a novel model architecture with a two-stage lift-splat-shoot encoder and a parameterized sparse hierarchical voxel representation. Experimental results on the NuScenes dataset demonstrate that DistillNeRF significantly outperforms existing comparable self-supervised methods for scene reconstruction, novel view synthesis, and depth estimation; and it allows for competitive zero-shot 3D semantic occupancy prediction, as well as open-world scene understanding through distilled foundation model features. Demos and code will be available at https://distillnerf.github.io/.<|reference_end|>
@article{wang2024distillnerf:,
    title={DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by
  Distilling Neural Fields and Foundation Model Features},
    author={Letian Wang, Seung Wook Kim, Jiawei Yang, Cunjun Yu, Boris Ivanovic,
  Steven L. Waslander, Yue Wang, Sanja Fidler, Marco Pavone, Peter Karkus},
    journal={arXiv preprint arXiv:2406.12095},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.12095},
    primaryClass={cs.CV cs.AI cs.RO}
}