title: Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping
summary: Grasping objects by a specific part is often crucial for safety and for
  executing downstream tasks. Yet, learning-based grasp planners lack this
  behavior unless they are trained on specific object part data, making it a
  significant challenge to scale object diversity. Instead, we propose LERF-TOGO,
  Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which
  uses vision-language models zero-shot to output a grasp distribution over an
  object given a natural language query. To accomplish this, we first reconstruct
  a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D
  language field queryable with text. However, LERF has no sense of objectness,
  meaning its relevancy outputs often return incomplete activations over an
  object which are insufficient for subsequent part queries. LERF-TOGO mitigates
  this lack of spatial grouping by extracting a 3D object mask via DINO features
  and then conditionally querying LERF on this mask to obtain a semantic
  distribution over the object with which to rank grasps from an off-the-shelf
  grasp planner. We evaluate LERF-TOGO's ability to grasp task-oriented object
  parts on 31 different physical objects, and find it selects grasps on the
  correct part in 81% of all trials and grasps successfully in 69%. See the
  project website at: lerftogo.github.io
paper_authors._id: 6507a4929c5e6b2767d8bb33
paper_authors.name: Adam Rashid
paper_authors._id: 6507a4929c5e6b2767d8bb34
paper_authors.name: Satvik Sharma
paper_authors._id: 6507a4929c5e6b2767d8bb35
paper_authors.name: Chung Min Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:08:20.921Z
paper_authors._id: 6348a53762c668c7b48bd796
paper_authors.avatarUrl: /avatars/a60214a0150b9c7047c6eaf482bc195b.svg
paper_authors.fullname: Chung Min Kim
paper_authors.type: user
paper_authors.user: chungmin99
paper_authors._id: 6507a4929c5e6b2767d8bb36
paper_authors.name: Justin Kerr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:08:27.909Z
paper_authors._id: 63ec69436ef3ce22b88b636d
paper_authors.avatarUrl: /avatars/83436c44a4650c8c81da769e024409a7.svg
paper_authors.fullname: Justin Kerr
paper_authors.type: user
paper_authors.user: wisepigeon
paper_authors._id: 6507a4929c5e6b2767d8bb37
paper_authors.name: Lawrence Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:09:01.231Z
paper_authors._id: 6303ff0c7b50dd9d0a3777e8
paper_authors.avatarUrl: /avatars/1c65d5936fe0cafd1fadd1102fad28db.svg
paper_authors.fullname: Lawrence Chen
paper_authors.type: user
paper_authors.user: lawrencechen
paper_authors._id: 6507a4929c5e6b2767d8bb38
paper_authors.name: Angjoo Kanazawa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:09:08.408Z
paper_authors._id: 6478cf7150ff7001631679c3
paper_authors.avatarUrl: /avatars/65ec385a9cc44c972e6caf952e759ff1.svg
paper_authors.fullname: Angjoo Kanazawa
paper_authors.type: user
paper_authors.user: akanazawa
paper_authors._id: 6507a4929c5e6b2767d8bb39
paper_authors.name: Ken Goldberg
paper_title: Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping
paper_summary: Grasping objects by a specific part is often crucial for safety and for
  executing downstream tasks. Yet, learning-based grasp planners lack this
  behavior unless they are trained on specific object part data, making it a
  significant challenge to scale object diversity. Instead, we propose LERF-TOGO,
  Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which
  uses vision-language models zero-shot to output a grasp distribution over an
  object given a natural language query. To accomplish this, we first reconstruct
  a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D
  language field queryable with text. However, LERF has no sense of objectness,
  meaning its relevancy outputs often return incomplete activations over an
  object which are insufficient for subsequent part queries. LERF-TOGO mitigates
  this lack of spatial grouping by extracting a 3D object mask via DINO features
  and then conditionally querying LERF on this mask to obtain a semantic
  distribution over the object with which to rank grasps from an off-the-shelf
  grasp planner. We evaluate LERF-TOGO's ability to grasp task-oriented object
  parts on 31 different physical objects, and find it selects grasps on the
  correct part in 81% of all trials and grasps successfully in 69%. See the
  project website at: lerftogo.github.io
paper_ai_keywords: LERF (Language Embedded Radiance Fields) CLIP embeddings 3D language field DINO features 3D object mask semantic distribution task-oriented grasping off-the-shelf grasp planner