title: GARField: Group Anything with Radiance Fields
summary: Grouping is inherently ambiguous due to the multiple levels of granularity in
  which one can decompose a scene -- should the wheels of an excavator be
  considered separate or part of the whole? We present Group Anything with
  Radiance Fields (GARField), an approach for decomposing 3D scenes into a
  hierarchy of semantically meaningful groups from posed image inputs. To do this
  we embrace group ambiguity through physical scale: by optimizing a
  scale-conditioned 3D affinity feature field, a point in the world can belong to
  different groups of different sizes. We optimize this field from a set of 2D
  masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine
  hierarchy, using scale to consistently fuse conflicting masks from different
  viewpoints. From this field we can derive a hierarchy of possible groupings via
  automatic tree construction or user interaction. We evaluate GARField on a
  variety of in-the-wild scenes and find it effectively extracts groups at many
  levels: clusters of objects, objects, and various subparts. GARField inherently
  represents multi-view consistent groupings and produces higher fidelity groups
  than the input SAM masks. GARField's hierarchical grouping could have exciting
  downstream applications such as 3D asset extraction or dynamic scene
  understanding. See the project website at https://www.garfield.studio/
paper_authors._id: 65a8c02c9fa2a0f9a2b18e41
paper_authors.name: Chung Min Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T10:14:17.239Z
paper_authors._id: 6348a53762c668c7b48bd796
paper_authors.avatarUrl: /avatars/a60214a0150b9c7047c6eaf482bc195b.svg
paper_authors.fullname: Chung Min Kim
paper_authors.type: user
paper_authors.user: chungmin99
paper_authors._id: 65a8c02c9fa2a0f9a2b18e42
paper_authors.name: Mingxuan Wu
paper_authors._id: 65a8c02c9fa2a0f9a2b18e43
paper_authors.name: Justin Kerr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T10:14:32.002Z
paper_authors._id: 63ec69436ef3ce22b88b636d
paper_authors.avatarUrl: /avatars/83436c44a4650c8c81da769e024409a7.svg
paper_authors.fullname: Justin Kerr
paper_authors.type: user
paper_authors.user: wisepigeon
paper_authors._id: 65a8c02c9fa2a0f9a2b18e44
paper_authors.name: Ken Goldberg
paper_authors._id: 65a8c02c9fa2a0f9a2b18e45
paper_authors.name: Matthew Tancik
paper_authors._id: 65a8c02c9fa2a0f9a2b18e46
paper_authors.name: Angjoo Kanazawa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T10:14:49.447Z
paper_authors._id: 6478cf7150ff7001631679c3
paper_authors.avatarUrl: /avatars/65ec385a9cc44c972e6caf952e759ff1.svg
paper_authors.fullname: Angjoo Kanazawa
paper_authors.type: user
paper_authors.user: akanazawa
paper_title: GARField: Group Anything with Radiance Fields
paper_summary: Grouping is inherently ambiguous due to the multiple levels of granularity in
  which one can decompose a scene -- should the wheels of an excavator be
  considered separate or part of the whole? We present Group Anything with
  Radiance Fields (GARField), an approach for decomposing 3D scenes into a
  hierarchy of semantically meaningful groups from posed image inputs. To do this
  we embrace group ambiguity through physical scale: by optimizing a
  scale-conditioned 3D affinity feature field, a point in the world can belong to
  different groups of different sizes. We optimize this field from a set of 2D
  masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine
  hierarchy, using scale to consistently fuse conflicting masks from different
  viewpoints. From this field we can derive a hierarchy of possible groupings via
  automatic tree construction or user interaction. We evaluate GARField on a
  variety of in-the-wild scenes and find it effectively extracts groups at many
  levels: clusters of objects, objects, and various subparts. GARField inherently
  represents multi-view consistent groupings and produces higher fidelity groups
  than the input SAM masks. GARField's hierarchical grouping could have exciting
  downstream applications such as 3D asset extraction or dynamic scene
  understanding. See the project website at https://www.garfield.studio/
paper_ai_summary: GARField extracts semantically meaningful hierarchical groupings from 3D scenes using scale-conditioned 3D affinity feature fields derived from 2D masks.
paper_ai_keywords: scale-conditioned 3D affinity feature field Group Anything with Radiance Fields (GARField) Segment Anything (SAM) coarse-to-fine hierarchy automatic tree construction multi-view consistency 3D asset extraction dynamic scene understanding