LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs
A critical obstacle preventing NeRF models from being deployed broadly in the
wild is their reliance on accurate camera poses. Consequently, there is growing
interest in extending NeRF models to jointly optimize camera poses and scene
representation, which offers an alternative to off-the-shelf SfM pipelines
which have well-understood failure modes. Existing approaches for unposed NeRF
operate under limited assumptions, such as a prior pose distribution or coarse
pose initialization, making them less effective in a general setting. In this
work, we propose a novel approach, LU-NeRF, that jointly estimates camera poses
and neural radiance fields with relaxed assumptions on pose configuration. Our
approach operates in a local-to-global manner, where we first optimize over
local subsets of the data, dubbed mini-scenes. LU-NeRF estimates local pose and
geometry for this challenging few-shot task. The mini-scene poses are brought
into a global reference frame through a robust pose synchronization step, where
a final global optimization of pose and scene can be performed. We show our
LU-NeRF pipeline outperforms prior attempts at unposed NeRF without making
restrictive assumptions on the pose prior. This allows us to operate in the
general SE(3) pose setting, unlike the baselines. Our results also indicate our
model can be complementary to feature-based SfM pipelines as it compares
favorably to COLMAP on low-texture and low-resolution images.
6482c04c2bc908e2d9f25aea
Zezhou Cheng
admin_assigned
2023-06-12T13:11:47.623Z
64834da6c8b6f4a798f9f09d
https://cdn-avatars.huggingface.co/v1/production/uploads/64834da6c8b6f4a798f9f09d/EOxUisexBKD-L5yCD7Ecs.png
Cheng
user
Zezhou
6482c04c2bc908e2d9f25aeb
Carlos Esteves
6482c04c2bc908e2d9f25aec
Varun Jampani
admin_assigned
2023-06-12T13:12:33.370Z
630e7cfe3fc17ffc50f6e835
/avatars/0b742ff094a09f9374fafcd97ab9e002.svg
Varun Jampani
user
varunjampani
6482c04c2bc908e2d9f25aed
Abhishek Kar
6482c04c2bc908e2d9f25aee
Subhransu Maji
6482c04c2bc908e2d9f25aef
Ameesh Makadia
LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs
A critical obstacle preventing NeRF models from being deployed broadly in the
wild is their reliance on accurate camera poses. Consequently, there is growing
interest in extending NeRF models to jointly optimize camera poses and scene
representation, which offers an alternative to off-the-shelf SfM pipelines
which have well-understood failure modes. Existing approaches for unposed NeRF
operate under limited assumptions, such as a prior pose distribution or coarse
pose initialization, making them less effective in a general setting. In this
work, we propose a novel approach, LU-NeRF, that jointly estimates camera poses
and neural radiance fields with relaxed assumptions on pose configuration. Our
approach operates in a local-to-global manner, where we first optimize over
local subsets of the data, dubbed mini-scenes. LU-NeRF estimates local pose and
geometry for this challenging few-shot task. The mini-scene poses are brought
into a global reference frame through a robust pose synchronization step, where
a final global optimization of pose and scene can be performed. We show our
LU-NeRF pipeline outperforms prior attempts at unposed NeRF without making
restrictive assumptions on the pose prior. This allows us to operate in the
general SE(3) pose setting, unlike the baselines. Our results also indicate our
model can be complementary to feature-based SfM pipelines as it compares
favorably to COLMAP on low-texture and low-resolution images.
LU-NeRF jointly estimates camera poses and neural radiance fields without restrictive assumptions, improving unposed NeRF performance and complementing feature-based SfM pipelines.
NeRF camera poses neural radiance fields SfM mini-scenes pose synchronization SE(3)