title: Distilling and Retrieving Generalizable Knowledge for Robot Manipulation
    via Language Corrections
summary: Today's robot policies exhibit subpar performance when faced with the
  challenge of generalizing to novel environments. Human corrective feedback is a
  crucial form of guidance to enable such generalization. However, adapting to
  and learning from online human corrections is a non-trivial endeavor: not only
  do robots need to remember human feedback over time to retrieve the right
  information in new settings and reduce the intervention rate, but also they
  would need to be able to respond to feedback that can be arbitrary corrections
  about high-level human preferences to low-level adjustments to skill
  parameters. In this work, we present Distillation and Retrieval of Online
  Corrections (DROC), a large language model (LLM)-based system that can respond
  to arbitrary forms of language feedback, distill generalizable knowledge from
  corrections, and retrieve relevant past experiences based on textual and visual
  similarity for improving performance in novel settings. DROC is able to respond
  to a sequence of online language corrections that address failures in both
  high-level task plans and low-level skill primitives. We demonstrate that DROC
  effectively distills the relevant information from the sequence of online
  corrections in a knowledge base and retrieves that knowledge in settings with
  new task or object instances. DROC outperforms other techniques that directly
  generate robot code via LLMs by using only half of the total number of
  corrections needed in the first round and requires little to no corrections
  after two iterations. We show further results, videos, prompts and code on
  https://sites.google.com/stanford.edu/droc .
paper_authors._id: 655ac617180749bf320cbbc2
paper_authors.name: Lihan Zha
paper_authors._id: 655ac617180749bf320cbbc3
paper_authors.name: Yuchen Cui
paper_authors._id: 655ac617180749bf320cbbc4
paper_authors.name: Li-Heng Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T13:44:19.745Z
paper_authors._id: 64518f70110343638e39e3d5
paper_authors.avatarUrl: /avatars/10e70f69174acbf7687618b70bdb845b.svg
paper_authors.fullname: Li-Heng Lin
paper_authors.type: user
paper_authors.user: lihenglin
paper_authors._id: 655ac617180749bf320cbbc5
paper_authors.name: Minae Kwon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T13:44:09.974Z
paper_authors._id: 645dd1f6a03f3ebb0be3d4de
paper_authors.avatarUrl: /avatars/9a3ec029036403c3c57428c98fa1501c.svg
paper_authors.fullname: Minae Kwon
paper_authors.type: user
paper_authors.user: minaek
paper_authors._id: 655ac617180749bf320cbbc6
paper_authors.name: Montserrat Gonzalez Arenas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T13:43:41.474Z
paper_authors._id: 63f47a88121f9894707465ed
paper_authors.avatarUrl: /avatars/d85d409d19068aea02a2532b587dd1ef.svg
paper_authors.fullname: Montserrat Gonzalez Arenas
paper_authors.type: user
paper_authors.user: montse90
paper_authors._id: 655ac617180749bf320cbbc7
paper_authors.name: Andy Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-21T10:24:37.248Z
paper_authors._id: 62d5f9d705461e9459d20b91
paper_authors.avatarUrl: /avatars/120917da2d8587674505163fb960e27d.svg
paper_authors.fullname: Andy Zeng
paper_authors.type: user
paper_authors.user: andyzeng
paper_authors._id: 655ac617180749bf320cbbc8
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T08:09:47.614Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 655ac617180749bf320cbbc9
paper_authors.name: Dorsa Sadigh
paper_title: Distilling and Retrieving Generalizable Knowledge for Robot Manipulation
    via Language Corrections
paper_summary: Today's robot policies exhibit subpar performance when faced with the
  challenge of generalizing to novel environments. Human corrective feedback is a
  crucial form of guidance to enable such generalization. However, adapting to
  and learning from online human corrections is a non-trivial endeavor: not only
  do robots need to remember human feedback over time to retrieve the right
  information in new settings and reduce the intervention rate, but also they
  would need to be able to respond to feedback that can be arbitrary corrections
  about high-level human preferences to low-level adjustments to skill
  parameters. In this work, we present Distillation and Retrieval of Online
  Corrections (DROC), a large language model (LLM)-based system that can respond
  to arbitrary forms of language feedback, distill generalizable knowledge from
  corrections, and retrieve relevant past experiences based on textual and visual
  similarity for improving performance in novel settings. DROC is able to respond
  to a sequence of online language corrections that address failures in both
  high-level task plans and low-level skill primitives. We demonstrate that DROC
  effectively distills the relevant information from the sequence of online
  corrections in a knowledge base and retrieves that knowledge in settings with
  new task or object instances. DROC outperforms other techniques that directly
  generate robot code via LLMs by using only half of the total number of
  corrections needed in the first round and requires little to no corrections
  after two iterations. We show further results, videos, prompts and code on
  https://sites.google.com/stanford.edu/droc .
paper_ai_keywords: large language model (LLM) knowledge base textual and visual similarity high-level task plans low-level skill primitives distillation retrieval sequence of online language corrections