title: Learning to Learn Faster from Human Feedback with Language Model
    Predictive Control
summary: Large language models (LLMs) have been shown to exhibit a wide range of
  capabilities, such as writing robot code from language commands -- enabling
  non-experts to direct robot behaviors, modify them based on feedback, or
  compose them to perform new tasks. However, these capabilities (driven by
  in-context learning) are limited to short-term interactions, where users'
  feedback remains relevant for only as long as it fits within the context size
  of the LLM, and can be forgotten over longer interactions. In this work, we
  investigate fine-tuning the robot code-writing LLMs, to remember their
  in-context interactions and improve their teachability i.e., how efficiently
  they adapt to human inputs (measured by average number of corrections before
  the user considers the task successful). Our key observation is that when
  human-robot interactions are formulated as a partially observable Markov
  decision process (in which human language inputs are observations, and robot
  code outputs are actions), then training an LLM to complete previous
  interactions can be viewed as training a transition dynamics model -- that can
  be combined with classic robotics techniques such as model predictive control
  (MPC) to discover shorter paths to success. This gives rise to Language Model
  Predictive Control (LMPC), a framework that fine-tunes PaLM 2 to improve its
  teachability on 78 tasks across 5 robot embodiments -- improving non-expert
  teaching success rates of unseen tasks by 26.9% while reducing the average
  number of human corrections from 2.4 to 1.9. Experiments show that LMPC also
  produces strong meta-learners, improving the success rate of in-context
  learning new tasks on unseen robot embodiments and APIs by 31.5%. See videos,
  code, and demos at: https://robot-teaching.github.io/.
paper_authors._id: 65d43f62011fd6cb37886bb3
paper_authors.name: Jacky Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:22:47.468Z
paper_authors._id: 63017a70792a47211c3714c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667856283106-63017a70792a47211c3714c1.jpeg
paper_authors.fullname: Jacky Liang
paper_authors.type: user
paper_authors.user: jackyliang42
paper_authors._id: 65d43f62011fd6cb37886bb4
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T08:37:53.157Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 65d43f62011fd6cb37886bb5
paper_authors.name: Wenhao Yu
paper_authors._id: 65d43f62011fd6cb37886bb6
paper_authors.name: Andy Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:40:27.249Z
paper_authors._id: 62d5f9d705461e9459d20b91
paper_authors.avatarUrl: /avatars/120917da2d8587674505163fb960e27d.svg
paper_authors.fullname: Andy Zeng
paper_authors.type: user
paper_authors.user: andyzeng
paper_authors._id: 65d43f62011fd6cb37886bb7
paper_authors.name: Montserrat Gonzalez Arenas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:40:33.846Z
paper_authors._id: 63f47a88121f9894707465ed
paper_authors.avatarUrl: /avatars/d85d409d19068aea02a2532b587dd1ef.svg
paper_authors.fullname: Montserrat Gonzalez Arenas
paper_authors.type: user
paper_authors.user: montse90
paper_authors._id: 65d43f62011fd6cb37886bb8
paper_authors.name: Maria Attarian
paper_authors._id: 65d43f62011fd6cb37886bb9
paper_authors.name: Maria Bauza
paper_authors._id: 65d43f62011fd6cb37886bba
paper_authors.name: Matthew Bennice
paper_authors._id: 65d43f62011fd6cb37886bbb
paper_authors.name: Alex Bewley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:41:08.853Z
paper_authors._id: 6492dfe783ff9ba7a9bbdeea
paper_authors.avatarUrl: /avatars/e1d8be8bb68200230cdf05497479e6f4.svg
paper_authors.fullname: Alex Bewley
paper_authors.type: user
paper_authors.user: bewley
paper_authors._id: 65d43f62011fd6cb37886bbc
paper_authors.name: Adil Dostmohamed
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:41:15.758Z
paper_authors._id: 63b7d36469b7bd7324f9f438
paper_authors.avatarUrl: /avatars/67b1864c378102b1cf2de571cce7bf9a.svg
paper_authors.fullname: Adil Dostmohamed
paper_authors.type: user
paper_authors.user: adild
paper_authors._id: 65d43f62011fd6cb37886bbd
paper_authors.name: Chuyuan Kelly Fu
paper_authors._id: 65d43f62011fd6cb37886bbe
paper_authors.name: Nimrod Gileadi
paper_authors._id: 65d43f62011fd6cb37886bbf
paper_authors.name: Marissa Giustina
paper_authors._id: 65d43f62011fd6cb37886bc0
paper_authors.name: Keerthana Gopalakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:41:44.056Z
paper_authors._id: 632f7dee34c562cb1d380fdc
paper_authors.avatarUrl: /avatars/d689e260535b8aa775654160028d4ec8.svg
paper_authors.fullname: Keerthana Gopalakrishnan
paper_authors.type: user
paper_authors.user: keerthanpg
paper_authors._id: 65d43f62011fd6cb37886bc1
paper_authors.name: Leonard Hasenclever
paper_authors._id: 65d43f62011fd6cb37886bc2
paper_authors.name: Jan Humplik
paper_authors._id: 65d43f62011fd6cb37886bc3
paper_authors.name: Jasmine Hsu
paper_authors._id: 65d43f62011fd6cb37886bc4
paper_authors.name: Nikhil Joshi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:42:34.771Z
paper_authors._id: 63c15485894342c8964c481d
paper_authors.avatarUrl: /avatars/cd938a12eb73a8395872f4058d85b0c5.svg
paper_authors.fullname: nikhil joshi
paper_authors.type: user
paper_authors.user: valeska
paper_authors._id: 65d43f62011fd6cb37886bc5
paper_authors.name: Ben Jyenis
paper_authors._id: 65d43f62011fd6cb37886bc6
paper_authors.name: Chase Kew
paper_authors._id: 65d43f62011fd6cb37886bc7
paper_authors.name: Sean Kirmani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:42:53.356Z
paper_authors._id: 648c85f33ffc11989bddcce1
paper_authors.avatarUrl: /avatars/bdde73b95a36e32cb2975656cca46022.svg
paper_authors.fullname: Sean Kirmani
paper_authors.type: user
paper_authors.user: skirmani
paper_authors._id: 65d43f62011fd6cb37886bc8
paper_authors.name: Tsang-Wei Edward Lee
paper_authors._id: 65d43f62011fd6cb37886bc9
paper_authors.name: Kuang-Huei Lee
paper_authors._id: 65d43f62011fd6cb37886bca
paper_authors.name: Assaf Hurwitz Michaely
paper_authors._id: 65d43f62011fd6cb37886bcb
paper_authors.name: Joss Moore
paper_authors._id: 65d43f62011fd6cb37886bcc
paper_authors.name: Ken Oslund
paper_authors._id: 65d43f62011fd6cb37886bcd
paper_authors.name: Dushyant Rao
paper_authors._id: 65d43f62011fd6cb37886bce
paper_authors.name: Allen Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:46:49.066Z
paper_authors._id: 6392628841491be61b52a4dc
paper_authors.avatarUrl: /avatars/c1e59fbf2252e332a3c86c2bd18afd09.svg
paper_authors.fullname: Allen Z. Ren
paper_authors.type: user
paper_authors.user: allenzren
paper_authors._id: 65d43f62011fd6cb37886bcf
paper_authors.name: Baruch Tabanpour
paper_authors._id: 65d43f62011fd6cb37886bd0
paper_authors.name: Quan Vuong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:46:24.802Z
paper_authors._id: 63e1d6687fbb6ae4d4f4d980
paper_authors.avatarUrl: /avatars/4959319e69d69e03a31ff4be326e602a.svg
paper_authors.fullname: Quan Vuong
paper_authors.type: user
paper_authors.user: quintonx
paper_authors._id: 65d43f62011fd6cb37886bd1
paper_authors.name: Ayzaan Wahid
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:46:11.933Z
paper_authors._id: 63922fc073bf5173e3631bb7
paper_authors.avatarUrl: /avatars/d672ca1cf5a541ec51f458b88eb9acec.svg
paper_authors.fullname: Ayzaan Wahid
paper_authors.type: user
paper_authors.user: ayzaan
paper_authors._id: 65d43f62011fd6cb37886bd2
paper_authors.name: Ted Xiao
paper_authors._id: 65d43f62011fd6cb37886bd3
paper_authors.name: Ying Xu
paper_authors._id: 65d43f62011fd6cb37886bd4
paper_authors.name: Vincent Zhuang
paper_authors._id: 65d43f62011fd6cb37886bd5
paper_authors.name: Peng Xu
paper_authors._id: 65d43f62011fd6cb37886bd6
paper_authors.name: Erik Frey
paper_authors._id: 65d43f62011fd6cb37886bd7
paper_authors.name: Ken Caluwaerts
paper_authors._id: 65d43f62011fd6cb37886bd8
paper_authors.name: Tingnan Zhang
paper_authors._id: 65d43f62011fd6cb37886bd9
paper_authors.name: Brian Ichter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:44:44.875Z
paper_authors._id: 633a26ab474cfeb1a864dc56
paper_authors.avatarUrl: /avatars/cd40accd894fc78810fc0d5108f413e9.svg
paper_authors.fullname: Brian I
paper_authors.type: user
paper_authors.user: brianichter
paper_authors._id: 65d43f62011fd6cb37886bda
paper_authors.name: Jonathan Tompson
paper_authors._id: 65d43f62011fd6cb37886bdb
paper_authors.name: Leila Takayama
paper_authors._id: 65d43f62011fd6cb37886bdc
paper_authors.name: Vincent Vanhoucke
paper_authors._id: 65d43f62011fd6cb37886bdd
paper_authors.name: Izhak Shafran
paper_authors._id: 65d43f62011fd6cb37886bde
paper_authors.name: Maja Mataric
paper_authors._id: 65d43f62011fd6cb37886bdf
paper_authors.name: Dorsa Sadigh
paper_authors._id: 65d43f62011fd6cb37886be0
paper_authors.name: Nicolas Heess
paper_authors._id: 65d43f62011fd6cb37886be1
paper_authors.name: Kanishka Rao
paper_authors._id: 65d43f62011fd6cb37886be2
paper_authors.name: Nik Stewart
paper_authors._id: 65d43f62011fd6cb37886be3
paper_authors.name: Jie Tan
paper_authors._id: 65d43f62011fd6cb37886be4
paper_authors.name: Carolina Parada
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:43:39.863Z
paper_authors._id: 65b7c3e87817e067a5b66cf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KysLXVoFmkPM2pQMgDlsb.jpeg
paper_authors.fullname: Carolina Parada
paper_authors.type: user
paper_authors.user: CarolinaParada
paper_title: Learning to Learn Faster from Human Feedback with Language Model
    Predictive Control
paper_summary: Large language models (LLMs) have been shown to exhibit a wide range of
  capabilities, such as writing robot code from language commands -- enabling
  non-experts to direct robot behaviors, modify them based on feedback, or
  compose them to perform new tasks. However, these capabilities (driven by
  in-context learning) are limited to short-term interactions, where users'
  feedback remains relevant for only as long as it fits within the context size
  of the LLM, and can be forgotten over longer interactions. In this work, we
  investigate fine-tuning the robot code-writing LLMs, to remember their
  in-context interactions and improve their teachability i.e., how efficiently
  they adapt to human inputs (measured by average number of corrections before
  the user considers the task successful). Our key observation is that when
  human-robot interactions are formulated as a partially observable Markov
  decision process (in which human language inputs are observations, and robot
  code outputs are actions), then training an LLM to complete previous
  interactions can be viewed as training a transition dynamics model -- that can
  be combined with classic robotics techniques such as model predictive control
  (MPC) to discover shorter paths to success. This gives rise to Language Model
  Predictive Control (LMPC), a framework that fine-tunes PaLM 2 to improve its
  teachability on 78 tasks across 5 robot embodiments -- improving non-expert
  teaching success rates of unseen tasks by 26.9% while reducing the average
  number of human corrections from 2.4 to 1.9. Experiments show that LMPC also
  produces strong meta-learners, improving the success rate of in-context
  learning new tasks on unseen robot embodiments and APIs by 31.5%. See videos,
  code, and demos at: https://robot-teaching.github.io/.
paper_ai_summary: Language Model Predictive Control (LMPC) improves the teachability of robot code-writing LLMs through fine-tuning, enhancing task success rates and reducing human corrections.
paper_ai_keywords: LLMs in-context learning robot code-writing partially observable Markov decision process transition dynamics model model predictive control LMPC PaLM 2 teachability meta-learners