title: Language to Rewards for Robotic Skill Synthesis
summary: Large language models (LLMs) have demonstrated exciting progress in acquiring
  diverse new capabilities through in-context learning, ranging from logical
  reasoning to code-writing. Robotics researchers have also explored using LLMs
  to advance the capabilities of robotic control. However, since low-level robot
  actions are hardware-dependent and underrepresented in LLM training corpora,
  existing efforts in applying LLMs to robotics have largely treated LLMs as
  semantic planners or relied on human-engineered control primitives to interface
  with the robot. On the other hand, reward functions are shown to be flexible
  representations that can be optimized for control policies to achieve diverse
  tasks, while their semantic richness makes them suitable to be specified by
  LLMs. In this work, we introduce a new paradigm that harnesses this realization
  by utilizing LLMs to define reward parameters that can be optimized and
  accomplish variety of robotic tasks. Using reward as the intermediate interface
  generated by LLMs, we can effectively bridge the gap between high-level
  language instructions or corrections to low-level robot actions. Meanwhile,
  combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive
  behavior creation experience where users can immediately observe the results
  and provide feedback to the system. To systematically evaluate the performance
  of our proposed method, we designed a total of 17 tasks for a simulated
  quadruped robot and a dexterous manipulator robot. We demonstrate that our
  proposed method reliably tackles 90% of the designed tasks, while a baseline
  using primitive skills as the interface with Code-as-policies achieves 50% of
  the tasks. We further validated our method on a real robot arm where complex
  manipulation skills such as non-prehensile pushing emerge through our
  interactive system.
paper_authors._id: 648bcdb4a7bfa231b27faab6
paper_authors.name: Wenhao Yu
paper_authors._id: 648bcdb4a7bfa231b27faab7
paper_authors.name: Nimrod Gileadi
paper_authors._id: 648bcdb4a7bfa231b27faab8
paper_authors.name: Chuyuan Fu
paper_authors._id: 648bcdb4a7bfa231b27faab9
paper_authors.name: Sean Kirmani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-16T17:38:34.340Z
paper_authors._id: 648c85f33ffc11989bddcce1
paper_authors.avatarUrl: /avatars/bdde73b95a36e32cb2975656cca46022.svg
paper_authors.fullname: Sean Kirmani
paper_authors.type: user
paper_authors.user: skirmani
paper_authors._id: 648bcdb4a7bfa231b27faaba
paper_authors.name: Kuang-Huei Lee
paper_authors._id: 648bcdb4a7bfa231b27faabb
paper_authors.name: Montse Gonzalez Arenas
paper_authors._id: 648bcdb4a7bfa231b27faabc
paper_authors.name: Hao-Tien Lewis Chiang
paper_authors._id: 648bcdb4a7bfa231b27faabd
paper_authors.name: Tom Erez
paper_authors._id: 648bcdb4a7bfa231b27faabe
paper_authors.name: Leonard Hasenclever
paper_authors._id: 648bcdb4a7bfa231b27faabf
paper_authors.name: Jan Humplik
paper_authors._id: 648bcdb4a7bfa231b27faac0
paper_authors.name: Brian Ichter
paper_authors._id: 648bcdb4a7bfa231b27faac1
paper_authors.name: Ted Xiao
paper_authors._id: 648bcdb4a7bfa231b27faac2
paper_authors.name: Peng Xu
paper_authors._id: 648bcdb4a7bfa231b27faac3
paper_authors.name: Andy Zeng
paper_authors._id: 648bcdb4a7bfa231b27faac4
paper_authors.name: Tingnan Zhang
paper_authors._id: 648bcdb4a7bfa231b27faac5
paper_authors.name: Nicolas Heess
paper_authors._id: 648bcdb4a7bfa231b27faac6
paper_authors.name: Dorsa Sadigh
paper_authors._id: 648bcdb4a7bfa231b27faac7
paper_authors.name: Jie Tan
paper_authors._id: 648bcdb4a7bfa231b27faac8
paper_authors.name: Yuval Tassa
paper_authors._id: 648bcdb4a7bfa231b27faac9
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-16T13:57:17.803Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_title: Language to Rewards for Robotic Skill Synthesis
paper_summary: Large language models (LLMs) have demonstrated exciting progress in acquiring
  diverse new capabilities through in-context learning, ranging from logical
  reasoning to code-writing. Robotics researchers have also explored using LLMs
  to advance the capabilities of robotic control. However, since low-level robot
  actions are hardware-dependent and underrepresented in LLM training corpora,
  existing efforts in applying LLMs to robotics have largely treated LLMs as
  semantic planners or relied on human-engineered control primitives to interface
  with the robot. On the other hand, reward functions are shown to be flexible
  representations that can be optimized for control policies to achieve diverse
  tasks, while their semantic richness makes them suitable to be specified by
  LLMs. In this work, we introduce a new paradigm that harnesses this realization
  by utilizing LLMs to define reward parameters that can be optimized and
  accomplish variety of robotic tasks. Using reward as the intermediate interface
  generated by LLMs, we can effectively bridge the gap between high-level
  language instructions or corrections to low-level robot actions. Meanwhile,
  combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive
  behavior creation experience where users can immediately observe the results
  and provide feedback to the system. To systematically evaluate the performance
  of our proposed method, we designed a total of 17 tasks for a simulated
  quadruped robot and a dexterous manipulator robot. We demonstrate that our
  proposed method reliably tackles 90% of the designed tasks, while a baseline
  using primitive skills as the interface with Code-as-policies achieves 50% of
  the tasks. We further validated our method on a real robot arm where complex
  manipulation skills such as non-prehensile pushing emerge through our
  interactive system.
paper_ai_keywords: Large language models (LLMs) in-context learning logical reasoning code-writing robotic control low-level robot actions hardware-dependent LLM training corpora semantic planners control primitives reward functions control policies reward parameters MuJoCo MPC quadruped robot dexterous manipulator robot Code-as-policies non-prehensile pushing