Learning to Learn Faster from Human Feedback with Language Model
  Predictive Control
Large language models (LLMs) have been shown to exhibit a wide range of
capabilities, such as writing robot code from language commands -- enabling
non-experts to direct robot behaviors, modify them based on feedback, or
compose them to perform new tasks. However, these capabilities (driven by
in-context learning) are limited to short-term interactions, where users'
feedback remains relevant for only as long as it fits within the context size
of the LLM, and can be forgotten over longer interactions. In this work, we
investigate fine-tuning the robot code-writing LLMs, to remember their
in-context interactions and improve their teachability i.e., how efficiently
they adapt to human inputs (measured by average number of corrections before
the user considers the task successful). Our key observation is that when
human-robot interactions are formulated as a partially observable Markov
decision process (in which human language inputs are observations, and robot
code outputs are actions), then training an LLM to complete previous
interactions can be viewed as training a transition dynamics model -- that can
be combined with classic robotics techniques such as model predictive control
(MPC) to discover shorter paths to success. This gives rise to Language Model
Predictive Control (LMPC), a framework that fine-tunes PaLM 2 to improve its
teachability on 78 tasks across 5 robot embodiments -- improving non-expert
teaching success rates of unseen tasks by 26.9% while reducing the average
number of human corrections from 2.4 to 1.9. Experiments show that LMPC also
produces strong meta-learners, improving the success rate of in-context
learning new tasks on unseen robot embodiments and APIs by 31.5%. See videos,
code, and demos at: https://robot-teaching.github.io/.
65d43f62011fd6cb37886bb3
Jacky Liang
admin_assigned
2024-02-20T10:22:47.468Z
63017a70792a47211c3714c1
https://cdn-avatars.huggingface.co/v1/production/uploads/1667856283106-63017a70792a47211c3714c1.jpeg
Jacky Liang
user
jackyliang42
65d43f62011fd6cb37886bb4
Fei Xia
claimed_verified
2024-02-20T08:37:53.157Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
65d43f62011fd6cb37886bb5
Wenhao Yu
65d43f62011fd6cb37886bb6
Andy Zeng
admin_assigned
2024-02-20T10:40:27.249Z
62d5f9d705461e9459d20b91
/avatars/120917da2d8587674505163fb960e27d.svg
Andy Zeng
user
andyzeng
65d43f62011fd6cb37886bb7
Montserrat Gonzalez Arenas
admin_assigned
2024-02-20T10:40:33.846Z
63f47a88121f9894707465ed
/avatars/d85d409d19068aea02a2532b587dd1ef.svg
Montserrat Gonzalez Arenas
user
montse90
65d43f62011fd6cb37886bb8
Maria Attarian
65d43f62011fd6cb37886bb9
Maria Bauza
65d43f62011fd6cb37886bba
Matthew Bennice
65d43f62011fd6cb37886bbb
Alex Bewley
admin_assigned
2024-02-20T10:41:08.853Z
6492dfe783ff9ba7a9bbdeea
/avatars/e1d8be8bb68200230cdf05497479e6f4.svg
Alex Bewley
user
bewley
65d43f62011fd6cb37886bbc
Adil Dostmohamed
admin_assigned
2024-02-20T10:41:15.758Z
63b7d36469b7bd7324f9f438
/avatars/67b1864c378102b1cf2de571cce7bf9a.svg
Adil Dostmohamed
user
adild
65d43f62011fd6cb37886bbd
Chuyuan Kelly Fu
65d43f62011fd6cb37886bbe
Nimrod Gileadi
65d43f62011fd6cb37886bbf
Marissa Giustina
65d43f62011fd6cb37886bc0
Keerthana Gopalakrishnan
admin_assigned
2024-02-20T10:41:44.056Z
632f7dee34c562cb1d380fdc
/avatars/d689e260535b8aa775654160028d4ec8.svg
Keerthana Gopalakrishnan
user
keerthanpg
65d43f62011fd6cb37886bc1
Leonard Hasenclever
65d43f62011fd6cb37886bc2
Jan Humplik
65d43f62011fd6cb37886bc3
Jasmine Hsu
65d43f62011fd6cb37886bc4
Nikhil Joshi
admin_assigned
2024-02-20T10:42:34.771Z
63c15485894342c8964c481d
/avatars/cd938a12eb73a8395872f4058d85b0c5.svg
nikhil joshi
user
valeska
65d43f62011fd6cb37886bc5
Ben Jyenis
65d43f62011fd6cb37886bc6
Chase Kew
65d43f62011fd6cb37886bc7
Sean Kirmani
admin_assigned
2024-02-20T10:42:53.356Z
648c85f33ffc11989bddcce1
/avatars/bdde73b95a36e32cb2975656cca46022.svg
Sean Kirmani
user
skirmani
65d43f62011fd6cb37886bc8
Tsang-Wei Edward Lee
65d43f62011fd6cb37886bc9
Kuang-Huei Lee
65d43f62011fd6cb37886bca
Assaf Hurwitz Michaely
65d43f62011fd6cb37886bcb
Joss Moore
65d43f62011fd6cb37886bcc
Ken Oslund
65d43f62011fd6cb37886bcd
Dushyant Rao
65d43f62011fd6cb37886bce
Allen Ren
admin_assigned
2024-02-20T10:46:49.066Z
6392628841491be61b52a4dc
/avatars/c1e59fbf2252e332a3c86c2bd18afd09.svg
Allen Z. Ren
user
allenzren
65d43f62011fd6cb37886bcf
Baruch Tabanpour
65d43f62011fd6cb37886bd0
Quan Vuong
admin_assigned
2024-02-20T10:46:24.802Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
65d43f62011fd6cb37886bd1
Ayzaan Wahid
admin_assigned
2024-02-20T10:46:11.933Z
63922fc073bf5173e3631bb7
/avatars/d672ca1cf5a541ec51f458b88eb9acec.svg
Ayzaan Wahid
user
ayzaan
65d43f62011fd6cb37886bd2
Ted Xiao
65d43f62011fd6cb37886bd3
Ying Xu
65d43f62011fd6cb37886bd4
Vincent Zhuang
65d43f62011fd6cb37886bd5
Peng Xu
65d43f62011fd6cb37886bd6
Erik Frey
65d43f62011fd6cb37886bd7
Ken Caluwaerts
65d43f62011fd6cb37886bd8
Tingnan Zhang
65d43f62011fd6cb37886bd9
Brian Ichter
admin_assigned
2024-02-20T10:44:44.875Z
633a26ab474cfeb1a864dc56
/avatars/cd40accd894fc78810fc0d5108f413e9.svg
Brian I
user
brianichter
65d43f62011fd6cb37886bda
Jonathan Tompson
65d43f62011fd6cb37886bdb
Leila Takayama
65d43f62011fd6cb37886bdc
Vincent Vanhoucke
65d43f62011fd6cb37886bdd
Izhak Shafran
65d43f62011fd6cb37886bde
Maja Mataric
65d43f62011fd6cb37886bdf
Dorsa Sadigh
65d43f62011fd6cb37886be0
Nicolas Heess
65d43f62011fd6cb37886be1
Kanishka Rao
65d43f62011fd6cb37886be2
Nik Stewart
65d43f62011fd6cb37886be3
Jie Tan
65d43f62011fd6cb37886be4
Carolina Parada
admin_assigned
2024-02-20T10:43:39.863Z
65b7c3e87817e067a5b66cf1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KysLXVoFmkPM2pQMgDlsb.jpeg
Carolina Parada
user
CarolinaParada
Learning to Learn Faster from Human Feedback with Language Model
  Predictive Control
Large language models (LLMs) have been shown to exhibit a wide range of
capabilities, such as writing robot code from language commands -- enabling
non-experts to direct robot behaviors, modify them based on feedback, or
compose them to perform new tasks. However, these capabilities (driven by
in-context learning) are limited to short-term interactions, where users'
feedback remains relevant for only as long as it fits within the context size
of the LLM, and can be forgotten over longer interactions. In this work, we
investigate fine-tuning the robot code-writing LLMs, to remember their
in-context interactions and improve their teachability i.e., how efficiently
they adapt to human inputs (measured by average number of corrections before
the user considers the task successful). Our key observation is that when
human-robot interactions are formulated as a partially observable Markov
decision process (in which human language inputs are observations, and robot
code outputs are actions), then training an LLM to complete previous
interactions can be viewed as training a transition dynamics model -- that can
be combined with classic robotics techniques such as model predictive control
(MPC) to discover shorter paths to success. This gives rise to Language Model
Predictive Control (LMPC), a framework that fine-tunes PaLM 2 to improve its
teachability on 78 tasks across 5 robot embodiments -- improving non-expert
teaching success rates of unseen tasks by 26.9% while reducing the average
number of human corrections from 2.4 to 1.9. Experiments show that LMPC also
produces strong meta-learners, improving the success rate of in-context
learning new tasks on unseen robot embodiments and APIs by 31.5%. See videos,
code, and demos at: https://robot-teaching.github.io/.
Language Model Predictive Control (LMPC) improves the teachability of robot code-writing LLMs through fine-tuning, enhancing task success rates and reducing human corrections.
LLMs in-context learning robot code-writing partially observable Markov decision process transition dynamics model model predictive control LMPC PaLM 2 teachability meta-learners