title: EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via
    Self-Supervision
summary: We present EmerNeRF, a simple yet powerful approach for learning
  spatial-temporal representations of dynamic driving scenes. Grounded in neural
  fields, EmerNeRF simultaneously captures scene geometry, appearance, motion,
  and semantics via self-bootstrapping. EmerNeRF hinges upon two core components:
  First, it stratifies scenes into static and dynamic fields. This decomposition
  emerges purely from self-supervision, enabling our model to learn from general,
  in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field
  from the dynamic field and uses this flow field to further aggregate
  multi-frame features, amplifying the rendering precision of dynamic objects.
  Coupling these three fields (static, dynamic, and flow) enables EmerNeRF to
  represent highly-dynamic scenes self-sufficiently, without relying on ground
  truth object annotations or pre-trained models for dynamic object segmentation
  or optical flow estimation. Our method achieves state-of-the-art performance in
  sensor simulation, significantly outperforming previous methods when
  reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In
  addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual
  foundation model features into 4D space-time and address a general positional
  bias in modern Transformers, significantly boosting 3D perception performance
  (e.g., 37.50% relative improvement in occupancy prediction accuracy on
  average). Finally, we construct a diverse and challenging 120-sequence dataset
  to benchmark neural fields under extreme and highly-dynamic settings.
paper_authors._id: 65487e36af5d7944326e34ca
paper_authors.name: Jiawei Yang
paper_authors._id: 65487e36af5d7944326e34cb
paper_authors.name: Boris Ivanovic
paper_authors._id: 65487e36af5d7944326e34cc
paper_authors.name: Or Litany
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-06T13:53:21.563Z
paper_authors._id: 630509558d7f39c0ecf29dc3
paper_authors.avatarUrl: /avatars/6b225b95f625152dc69fedb5f0b348d6.svg
paper_authors.fullname: Or Litany
paper_authors.type: user
paper_authors.user: orlitany
paper_authors._id: 65487e36af5d7944326e34cd
paper_authors.name: Xinshuo Weng
paper_authors._id: 65487e36af5d7944326e34ce
paper_authors.name: Seung Wook Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-06T13:53:36.472Z
paper_authors._id: 64fb5827d82fc6977d70582d
paper_authors.avatarUrl: /avatars/54180874d2f162572c414970b8750759.svg
paper_authors.fullname: Seung Wook Kim
paper_authors.type: user
paper_authors.user: swkimbd
paper_authors._id: 65487e36af5d7944326e34cf
paper_authors.name: Boyi Li
paper_authors._id: 65487e36af5d7944326e34d0
paper_authors.name: Tong Che
paper_authors._id: 65487e36af5d7944326e34d1
paper_authors.name: Danfei Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-06T13:56:25.046Z
paper_authors._id: 636eb10afd9751c3d08279e9
paper_authors.avatarUrl: /avatars/8e9833b2766c76e92be4dba2ceadd61f.svg
paper_authors.fullname: Xu
paper_authors.type: user
paper_authors.user: Danfei
paper_authors._id: 65487e36af5d7944326e34d2
paper_authors.name: Sanja Fidler
paper_authors._id: 65487e36af5d7944326e34d3
paper_authors.name: Marco Pavone
paper_authors._id: 65487e36af5d7944326e34d4
paper_authors.name: Yue Wang
paper_title: EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via
    Self-Supervision
paper_summary: We present EmerNeRF, a simple yet powerful approach for learning
  spatial-temporal representations of dynamic driving scenes. Grounded in neural
  fields, EmerNeRF simultaneously captures scene geometry, appearance, motion,
  and semantics via self-bootstrapping. EmerNeRF hinges upon two core components:
  First, it stratifies scenes into static and dynamic fields. This decomposition
  emerges purely from self-supervision, enabling our model to learn from general,
  in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field
  from the dynamic field and uses this flow field to further aggregate
  multi-frame features, amplifying the rendering precision of dynamic objects.
  Coupling these three fields (static, dynamic, and flow) enables EmerNeRF to
  represent highly-dynamic scenes self-sufficiently, without relying on ground
  truth object annotations or pre-trained models for dynamic object segmentation
  or optical flow estimation. Our method achieves state-of-the-art performance in
  sensor simulation, significantly outperforming previous methods when
  reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In
  addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual
  foundation model features into 4D space-time and address a general positional
  bias in modern Transformers, significantly boosting 3D perception performance
  (e.g., 37.50% relative improvement in occupancy prediction accuracy on
  average). Finally, we construct a diverse and challenging 120-sequence dataset
  to benchmark neural fields under extreme and highly-dynamic settings.
paper_ai_summary: EmerNeRF uses neural fields with self-bootstrapping to learn and represent static and dynamic components of driving scenes efficiently, achieving state-of-the-art performance in sensor simulation and semantic generalization.
paper_ai_keywords: neural fields EmerNeRF self-bootstrapping static fields dynamic fields flow field multi-frame features rendering precision 3D perception occupancy prediction 4D space-time semantic generalization