EgoLifter: Open-world 3D Segmentation for Egocentric Perception
In this paper we present EgoLifter, a novel system that can automatically
segment scenes captured from egocentric sensors into a complete decomposition
of individual 3D objects. The system is specifically designed for egocentric
data where scenes contain hundreds of objects captured from natural
(non-scanning) motion. EgoLifter adopts 3D Gaussians as the underlying
representation of 3D scenes and objects and uses segmentation masks from the
Segment Anything Model (SAM) as weak supervision to learn flexible and
promptable definitions of object instances free of any specific object
taxonomy. To handle the challenge of dynamic objects in ego-centric videos, we
design a transient prediction module that learns to filter out dynamic objects
in the 3D reconstruction. The result is a fully automatic pipeline that is able
to reconstruct 3D object instances as collections of 3D Gaussians that
collectively compose the entire scene. We created a new benchmark on the Aria
Digital Twin dataset that quantitatively demonstrates its state-of-the-art
performance in open-world 3D segmentation from natural egocentric input. We run
EgoLifter on various egocentric activity datasets which shows the promise of
the method for 3D egocentric perception at scale.
6604e6ed25701f49be01d980
Qiao Gu
claimed_verified
2024-09-22T12:49:06.963Z
63d1df92f7f31a66a2d7292c
/avatars/eb3339c1f2c82742b518b8a7f142e99a.svg
Qiao Gu
user
guqiao
6604e6ed25701f49be01d981
Zhaoyang Lv
6604e6ed25701f49be01d982
Duncan Frost
6604e6ed25701f49be01d983
Simon Green
6604e6ed25701f49be01d984
Julian Straub
6604e6ed25701f49be01d985
Chris Sweeney
EgoLifter: Open-world 3D Segmentation for Egocentric Perception
In this paper we present EgoLifter, a novel system that can automatically
segment scenes captured from egocentric sensors into a complete decomposition
of individual 3D objects. The system is specifically designed for egocentric
data where scenes contain hundreds of objects captured from natural
(non-scanning) motion. EgoLifter adopts 3D Gaussians as the underlying
representation of 3D scenes and objects and uses segmentation masks from the
Segment Anything Model (SAM) as weak supervision to learn flexible and
promptable definitions of object instances free of any specific object
taxonomy. To handle the challenge of dynamic objects in ego-centric videos, we
design a transient prediction module that learns to filter out dynamic objects
in the 3D reconstruction. The result is a fully automatic pipeline that is able
to reconstruct 3D object instances as collections of 3D Gaussians that
collectively compose the entire scene. We created a new benchmark on the Aria
Digital Twin dataset that quantitatively demonstrates its state-of-the-art
performance in open-world 3D segmentation from natural egocentric input. We run
EgoLifter on various egocentric activity datasets which shows the promise of
the method for 3D egocentric perception at scale.
EgoLifter segments 3D scenes captured from egocentric sensors into individual objects using 3D Gaussians, SAM for weak supervision, and a transient prediction module for dynamic object filtering.
3D Gaussians Segment Anything Model (SAM) transient prediction module 3D reconstruction egocentric data open-world 3D segmentation Aria Digital Twin dataset 3D egocentric perception