title: LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs
summary: A critical obstacle preventing NeRF models from being deployed broadly in the
  wild is their reliance on accurate camera poses. Consequently, there is growing
  interest in extending NeRF models to jointly optimize camera poses and scene
  representation, which offers an alternative to off-the-shelf SfM pipelines
  which have well-understood failure modes. Existing approaches for unposed NeRF
  operate under limited assumptions, such as a prior pose distribution or coarse
  pose initialization, making them less effective in a general setting. In this
  work, we propose a novel approach, LU-NeRF, that jointly estimates camera poses
  and neural radiance fields with relaxed assumptions on pose configuration. Our
  approach operates in a local-to-global manner, where we first optimize over
  local subsets of the data, dubbed mini-scenes. LU-NeRF estimates local pose and
  geometry for this challenging few-shot task. The mini-scene poses are brought
  into a global reference frame through a robust pose synchronization step, where
  a final global optimization of pose and scene can be performed. We show our
  LU-NeRF pipeline outperforms prior attempts at unposed NeRF without making
  restrictive assumptions on the pose prior. This allows us to operate in the
  general SE(3) pose setting, unlike the baselines. Our results also indicate our
  model can be complementary to feature-based SfM pipelines as it compares
  favorably to COLMAP on low-texture and low-resolution images.
paper_authors._id: 6482c04c2bc908e2d9f25aea
paper_authors.name: Zezhou Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:11:47.623Z
paper_authors._id: 64834da6c8b6f4a798f9f09d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64834da6c8b6f4a798f9f09d/EOxUisexBKD-L5yCD7Ecs.png
paper_authors.fullname: Cheng
paper_authors.type: user
paper_authors.user: Zezhou
paper_authors._id: 6482c04c2bc908e2d9f25aeb
paper_authors.name: Carlos Esteves
paper_authors._id: 6482c04c2bc908e2d9f25aec
paper_authors.name: Varun Jampani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:12:33.370Z
paper_authors._id: 630e7cfe3fc17ffc50f6e835
paper_authors.avatarUrl: /avatars/0b742ff094a09f9374fafcd97ab9e002.svg
paper_authors.fullname: Varun Jampani
paper_authors.type: user
paper_authors.user: varunjampani
paper_authors._id: 6482c04c2bc908e2d9f25aed
paper_authors.name: Abhishek Kar
paper_authors._id: 6482c04c2bc908e2d9f25aee
paper_authors.name: Subhransu Maji
paper_authors._id: 6482c04c2bc908e2d9f25aef
paper_authors.name: Ameesh Makadia
paper_title: LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs
paper_summary: A critical obstacle preventing NeRF models from being deployed broadly in the
  wild is their reliance on accurate camera poses. Consequently, there is growing
  interest in extending NeRF models to jointly optimize camera poses and scene
  representation, which offers an alternative to off-the-shelf SfM pipelines
  which have well-understood failure modes. Existing approaches for unposed NeRF
  operate under limited assumptions, such as a prior pose distribution or coarse
  pose initialization, making them less effective in a general setting. In this
  work, we propose a novel approach, LU-NeRF, that jointly estimates camera poses
  and neural radiance fields with relaxed assumptions on pose configuration. Our
  approach operates in a local-to-global manner, where we first optimize over
  local subsets of the data, dubbed mini-scenes. LU-NeRF estimates local pose and
  geometry for this challenging few-shot task. The mini-scene poses are brought
  into a global reference frame through a robust pose synchronization step, where
  a final global optimization of pose and scene can be performed. We show our
  LU-NeRF pipeline outperforms prior attempts at unposed NeRF without making
  restrictive assumptions on the pose prior. This allows us to operate in the
  general SE(3) pose setting, unlike the baselines. Our results also indicate our
  model can be complementary to feature-based SfM pipelines as it compares
  favorably to COLMAP on low-texture and low-resolution images.
paper_ai_summary: LU-NeRF jointly estimates camera poses and neural radiance fields without restrictive assumptions, improving unposed NeRF performance and complementing feature-based SfM pipelines.
paper_ai_keywords: NeRF camera poses neural radiance fields SfM mini-scenes pose synchronization SE(3)