Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
Self-supervised and language-supervised image models contain rich knowledge
of the world that is important for generalization. Many robotic tasks, however,
require a detailed understanding of 3D geometry, which is often lacking in 2D
image features. This work bridges this 2D-to-3D gap for robotic manipulation by
leveraging distilled feature fields to combine accurate 3D geometry with rich
semantics from 2D foundation models. We present a few-shot learning method for
6-DOF grasping and placing that harnesses these strong spatial and semantic
priors to achieve in-the-wild generalization to unseen objects. Using features
distilled from a vision-language model, CLIP, we present a way to designate
novel objects for manipulation via free-text natural language, and demonstrate
its ability to generalize to unseen expressions and novel categories of
objects.
64ddaa7de9073130dc157996
William Shen
64ddaa7de9073130dc157997
Ge Yang
admin_assigned
2023-08-25T13:15:49.645Z
5f03ee98ff69163f6f59e9e2
https://cdn-avatars.huggingface.co/v1/production/uploads/1594093169905-noauth.jpeg
Ge Yang
user
geyang
64ddaa7de9073130dc157998
Alan Yu
64ddaa7de9073130dc157999
Jansen Wong
64ddaa7de9073130dc15799a
Leslie Pack Kaelbling
64ddaa7de9073130dc15799b
Phillip Isola
admin_assigned
2023-08-25T13:16:20.618Z
623cb1f3ca87989bbf797eee
/avatars/13fbd5b123b8676561751f820317600c.svg
Phillip Isola
user
phillipi
Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
Self-supervised and language-supervised image models contain rich knowledge
of the world that is important for generalization. Many robotic tasks, however,
require a detailed understanding of 3D geometry, which is often lacking in 2D
image features. This work bridges this 2D-to-3D gap for robotic manipulation by
leveraging distilled feature fields to combine accurate 3D geometry with rich
semantics from 2D foundation models. We present a few-shot learning method for
6-DOF grasping and placing that harnesses these strong spatial and semantic
priors to achieve in-the-wild generalization to unseen objects. Using features
distilled from a vision-language model, CLIP, we present a way to designate
novel objects for manipulation via free-text natural language, and demonstrate
its ability to generalize to unseen expressions and novel categories of
objects.
A method using distilled feature fields combines 3D geometry with 2D semantic features from CLIP to enable few-shot learning for robotic manipulation with generalization to new objects and expressions.
self-supervised models language-supervised models 3D geometry 2D image features distilled feature fields 6-DOF grasping CLIP vision-language model few-shot learning in-the-wild generalization