title: EgoLifter: Open-world 3D Segmentation for Egocentric Perception
summary: In this paper we present EgoLifter, a novel system that can automatically
  segment scenes captured from egocentric sensors into a complete decomposition
  of individual 3D objects. The system is specifically designed for egocentric
  data where scenes contain hundreds of objects captured from natural
  (non-scanning) motion. EgoLifter adopts 3D Gaussians as the underlying
  representation of 3D scenes and objects and uses segmentation masks from the
  Segment Anything Model (SAM) as weak supervision to learn flexible and
  promptable definitions of object instances free of any specific object
  taxonomy. To handle the challenge of dynamic objects in ego-centric videos, we
  design a transient prediction module that learns to filter out dynamic objects
  in the 3D reconstruction. The result is a fully automatic pipeline that is able
  to reconstruct 3D object instances as collections of 3D Gaussians that
  collectively compose the entire scene. We created a new benchmark on the Aria
  Digital Twin dataset that quantitatively demonstrates its state-of-the-art
  performance in open-world 3D segmentation from natural egocentric input. We run
  EgoLifter on various egocentric activity datasets which shows the promise of
  the method for 3D egocentric perception at scale.
paper_authors._id: 6604e6ed25701f49be01d980
paper_authors.name: Qiao Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-22T12:49:06.963Z
paper_authors._id: 63d1df92f7f31a66a2d7292c
paper_authors.avatarUrl: /avatars/eb3339c1f2c82742b518b8a7f142e99a.svg
paper_authors.fullname: Qiao Gu
paper_authors.type: user
paper_authors.user: guqiao
paper_authors._id: 6604e6ed25701f49be01d981
paper_authors.name: Zhaoyang Lv
paper_authors._id: 6604e6ed25701f49be01d982
paper_authors.name: Duncan Frost
paper_authors._id: 6604e6ed25701f49be01d983
paper_authors.name: Simon Green
paper_authors._id: 6604e6ed25701f49be01d984
paper_authors.name: Julian Straub
paper_authors._id: 6604e6ed25701f49be01d985
paper_authors.name: Chris Sweeney
paper_title: EgoLifter: Open-world 3D Segmentation for Egocentric Perception
paper_summary: In this paper we present EgoLifter, a novel system that can automatically
  segment scenes captured from egocentric sensors into a complete decomposition
  of individual 3D objects. The system is specifically designed for egocentric
  data where scenes contain hundreds of objects captured from natural
  (non-scanning) motion. EgoLifter adopts 3D Gaussians as the underlying
  representation of 3D scenes and objects and uses segmentation masks from the
  Segment Anything Model (SAM) as weak supervision to learn flexible and
  promptable definitions of object instances free of any specific object
  taxonomy. To handle the challenge of dynamic objects in ego-centric videos, we
  design a transient prediction module that learns to filter out dynamic objects
  in the 3D reconstruction. The result is a fully automatic pipeline that is able
  to reconstruct 3D object instances as collections of 3D Gaussians that
  collectively compose the entire scene. We created a new benchmark on the Aria
  Digital Twin dataset that quantitatively demonstrates its state-of-the-art
  performance in open-world 3D segmentation from natural egocentric input. We run
  EgoLifter on various egocentric activity datasets which shows the promise of
  the method for 3D egocentric perception at scale.
paper_ai_keywords: 3D Gaussians Segment Anything Model (SAM) weak supervision transient prediction module 3D reconstruction 3D segmentation Aria Digital Twin dataset 3D egocentric perception