Distilling and Retrieving Generalizable Knowledge for Robot Manipulation
  via Language Corrections
Today's robot policies exhibit subpar performance when faced with the
challenge of generalizing to novel environments. Human corrective feedback is a
crucial form of guidance to enable such generalization. However, adapting to
and learning from online human corrections is a non-trivial endeavor: not only
do robots need to remember human feedback over time to retrieve the right
information in new settings and reduce the intervention rate, but also they
would need to be able to respond to feedback that can be arbitrary corrections
about high-level human preferences to low-level adjustments to skill
parameters. In this work, we present Distillation and Retrieval of Online
Corrections (DROC), a large language model (LLM)-based system that can respond
to arbitrary forms of language feedback, distill generalizable knowledge from
corrections, and retrieve relevant past experiences based on textual and visual
similarity for improving performance in novel settings. DROC is able to respond
to a sequence of online language corrections that address failures in both
high-level task plans and low-level skill primitives. We demonstrate that DROC
effectively distills the relevant information from the sequence of online
corrections in a knowledge base and retrieves that knowledge in settings with
new task or object instances. DROC outperforms other techniques that directly
generate robot code via LLMs by using only half of the total number of
corrections needed in the first round and requires little to no corrections
after two iterations. We show further results, videos, prompts and code on
https://sites.google.com/stanford.edu/droc .
655ac617180749bf320cbbc2
Lihan Zha
claimed_verified
2025-06-09T10:15:22.788Z
655c52793d1cee8fcbc11a46
/avatars/b8a3af9b187bf5220df0b3d528671e56.svg
Lihan Zha
user
lihzha
655ac617180749bf320cbbc3
Yuchen Cui
655ac617180749bf320cbbc4
Li-Heng Lin
admin_assigned
2023-11-20T13:44:19.745Z
64518f70110343638e39e3d5
/avatars/10e70f69174acbf7687618b70bdb845b.svg
Li-Heng Lin
user
lihenglin
655ac617180749bf320cbbc5
Minae Kwon
admin_assigned
2023-11-20T13:44:09.974Z
645dd1f6a03f3ebb0be3d4de
/avatars/9a3ec029036403c3c57428c98fa1501c.svg
Minae Kwon
user
minaek
655ac617180749bf320cbbc6
Montserrat Gonzalez Arenas
admin_assigned
2023-11-20T13:43:41.474Z
63f47a88121f9894707465ed
/avatars/d85d409d19068aea02a2532b587dd1ef.svg
Montserrat Gonzalez Arenas
user
montse90
655ac617180749bf320cbbc7
Andy Zeng
claimed_verified
2023-11-21T10:24:37.248Z
62d5f9d705461e9459d20b91
/avatars/120917da2d8587674505163fb960e27d.svg
Andy Zeng
user
andyzeng
655ac617180749bf320cbbc8
Fei Xia
claimed_verified
2024-01-23T08:09:47.614Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
655ac617180749bf320cbbc9
Dorsa Sadigh
Distilling and Retrieving Generalizable Knowledge for Robot Manipulation
  via Language Corrections
Today's robot policies exhibit subpar performance when faced with the
challenge of generalizing to novel environments. Human corrective feedback is a
crucial form of guidance to enable such generalization. However, adapting to
and learning from online human corrections is a non-trivial endeavor: not only
do robots need to remember human feedback over time to retrieve the right
information in new settings and reduce the intervention rate, but also they
would need to be able to respond to feedback that can be arbitrary corrections
about high-level human preferences to low-level adjustments to skill
parameters. In this work, we present Distillation and Retrieval of Online
Corrections (DROC), a large language model (LLM)-based system that can respond
to arbitrary forms of language feedback, distill generalizable knowledge from
corrections, and retrieve relevant past experiences based on textual and visual
similarity for improving performance in novel settings. DROC is able to respond
to a sequence of online language corrections that address failures in both
high-level task plans and low-level skill primitives. We demonstrate that DROC
effectively distills the relevant information from the sequence of online
corrections in a knowledge base and retrieves that knowledge in settings with
new task or object instances. DROC outperforms other techniques that directly
generate robot code via LLMs by using only half of the total number of
corrections needed in the first round and requires little to no corrections
after two iterations. We show further results, videos, prompts and code on
https://sites.google.com/stanford.edu/droc .
A large language model-based system, DROC, effectively processes and applies human corrections to improve robot performance in new environments by distilling and retrieving relevant knowledge.
large language model distillation retrieval online corrections high-level task plans low-level skill primitives knowledge base textual similarity visual similarity robot code generation