Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping
Grasping objects by a specific part is often crucial for safety and for
executing downstream tasks. Yet, learning-based grasp planners lack this
behavior unless they are trained on specific object part data, making it a
significant challenge to scale object diversity. Instead, we propose LERF-TOGO,
Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which
uses vision-language models zero-shot to output a grasp distribution over an
object given a natural language query. To accomplish this, we first reconstruct
a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D
language field queryable with text. However, LERF has no sense of objectness,
meaning its relevancy outputs often return incomplete activations over an
object which are insufficient for subsequent part queries. LERF-TOGO mitigates
this lack of spatial grouping by extracting a 3D object mask via DINO features
and then conditionally querying LERF on this mask to obtain a semantic
distribution over the object with which to rank grasps from an off-the-shelf
grasp planner. We evaluate LERF-TOGO's ability to grasp task-oriented object
parts on 31 different physical objects, and find it selects grasps on the
correct part in 81% of all trials and grasps successfully in 69%. See the
project website at: lerftogo.github.io
6507a4929c5e6b2767d8bb33
Adam Rashid
6507a4929c5e6b2767d8bb34
Satvik Sharma
6507a4929c5e6b2767d8bb35
Chung Min Kim
admin_assigned
2023-09-18T10:08:20.921Z
6348a53762c668c7b48bd796
/avatars/a60214a0150b9c7047c6eaf482bc195b.svg
Chung Min Kim
user
chungmin99
6507a4929c5e6b2767d8bb36
Justin Kerr
admin_assigned
2023-09-18T10:08:27.909Z
63ec69436ef3ce22b88b636d
/avatars/83436c44a4650c8c81da769e024409a7.svg
Justin Kerr
user
wisepigeon
6507a4929c5e6b2767d8bb37
Lawrence Chen
admin_assigned
2023-09-18T10:09:01.231Z
6303ff0c7b50dd9d0a3777e8
/avatars/1c65d5936fe0cafd1fadd1102fad28db.svg
Lawrence Chen
user
lawrencechen
6507a4929c5e6b2767d8bb38
Angjoo Kanazawa
admin_assigned
2023-09-18T10:09:08.408Z
6478cf7150ff7001631679c3
/avatars/65ec385a9cc44c972e6caf952e759ff1.svg
Angjoo Kanazawa
user
akanazawa
6507a4929c5e6b2767d8bb39
Ken Goldberg
Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping
Grasping objects by a specific part is often crucial for safety and for
executing downstream tasks. Yet, learning-based grasp planners lack this
behavior unless they are trained on specific object part data, making it a
significant challenge to scale object diversity. Instead, we propose LERF-TOGO,
Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which
uses vision-language models zero-shot to output a grasp distribution over an
object given a natural language query. To accomplish this, we first reconstruct
a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D
language field queryable with text. However, LERF has no sense of objectness,
meaning its relevancy outputs often return incomplete activations over an
object which are insufficient for subsequent part queries. LERF-TOGO mitigates
this lack of spatial grouping by extracting a 3D object mask via DINO features
and then conditionally querying LERF on this mask to obtain a semantic
distribution over the object with which to rank grasps from an off-the-shelf
grasp planner. We evaluate LERF-TOGO's ability to grasp task-oriented object
parts on 31 different physical objects, and find it selects grasps on the
correct part in 81% of all trials and grasps successfully in 69%. See the
project website at: lerftogo.github.io
LERF-TOGO uses vision-language models and DINO features for zero-shot task-oriented grasping by extracting object masks and ranking grasps over specific parts.
LERF Language Embedded Radiance Fields CLIP embeddings 3D language field DINO features grasp distribution object mask semantic distribution grasp planner task-oriented grasping