title: Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
summary: Self-supervised and language-supervised image models contain rich knowledge
  of the world that is important for generalization. Many robotic tasks, however,
  require a detailed understanding of 3D geometry, which is often lacking in 2D
  image features. This work bridges this 2D-to-3D gap for robotic manipulation by
  leveraging distilled feature fields to combine accurate 3D geometry with rich
  semantics from 2D foundation models. We present a few-shot learning method for
  6-DOF grasping and placing that harnesses these strong spatial and semantic
  priors to achieve in-the-wild generalization to unseen objects. Using features
  distilled from a vision-language model, CLIP, we present a way to designate
  novel objects for manipulation via free-text natural language, and demonstrate
  its ability to generalize to unseen expressions and novel categories of
  objects.
paper_authors._id: 64ddaa7de9073130dc157996
paper_authors.name: William Shen
paper_authors._id: 64ddaa7de9073130dc157997
paper_authors.name: Ge Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-25T13:15:49.645Z
paper_authors._id: 5f03ee98ff69163f6f59e9e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594093169905-noauth.jpeg
paper_authors.fullname: Ge Yang
paper_authors.type: user
paper_authors.user: geyang
paper_authors._id: 64ddaa7de9073130dc157998
paper_authors.name: Alan Yu
paper_authors._id: 64ddaa7de9073130dc157999
paper_authors.name: Jansen Wong
paper_authors._id: 64ddaa7de9073130dc15799a
paper_authors.name: Leslie Pack Kaelbling
paper_authors._id: 64ddaa7de9073130dc15799b
paper_authors.name: Phillip Isola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-25T13:16:20.618Z
paper_authors._id: 623cb1f3ca87989bbf797eee
paper_authors.avatarUrl: /avatars/13fbd5b123b8676561751f820317600c.svg
paper_authors.fullname: Phillip Isola
paper_authors.type: user
paper_authors.user: phillipi
paper_title: Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
paper_summary: Self-supervised and language-supervised image models contain rich knowledge
  of the world that is important for generalization. Many robotic tasks, however,
  require a detailed understanding of 3D geometry, which is often lacking in 2D
  image features. This work bridges this 2D-to-3D gap for robotic manipulation by
  leveraging distilled feature fields to combine accurate 3D geometry with rich
  semantics from 2D foundation models. We present a few-shot learning method for
  6-DOF grasping and placing that harnesses these strong spatial and semantic
  priors to achieve in-the-wild generalization to unseen objects. Using features
  distilled from a vision-language model, CLIP, we present a way to designate
  novel objects for manipulation via free-text natural language, and demonstrate
  its ability to generalize to unseen expressions and novel categories of
  objects.
paper_ai_summary: A method using distilled feature fields combines 3D geometry with 2D semantic features from CLIP to enable few-shot learning for robotic manipulation with generalization to new objects and expressions.
paper_ai_keywords: self-supervised models language-supervised models 3D geometry 2D image features distilled feature fields 6-DOF grasping CLIP vision-language model few-shot learning in-the-wild generalization