title: Eureka: Human-Level Reward Design via Coding Large Language Models
summary: Large Language Models (LLMs) have excelled as high-level semantic planners
  for sequential decision-making tasks. However, harnessing them to learn complex
  low-level manipulation tasks, such as dexterous pen spinning, remains an open
  problem. We bridge this fundamental gap and present Eureka, a human-level
  reward design algorithm powered by LLMs. Eureka exploits the remarkable
  zero-shot generation, code-writing, and in-context improvement capabilities of
  state-of-the-art LLMs, such as GPT-4, to perform evolutionary optimization over
  reward code. The resulting rewards can then be used to acquire complex skills
  via reinforcement learning. Without any task-specific prompting or pre-defined
  reward templates, Eureka generates reward functions that outperform expert
  human-engineered rewards. In a diverse suite of 29 open-source RL environments
  that include 10 distinct robot morphologies, Eureka outperforms human experts
  on 83% of the tasks, leading to an average normalized improvement of 52%. The
  generality of Eureka also enables a new gradient-free in-context learning
  approach to reinforcement learning from human feedback (RLHF), readily
  incorporating human inputs to improve the quality and the safety of the
  generated rewards without model updating. Finally, using Eureka rewards in a
  curriculum learning setting, we demonstrate for the first time, a simulated
  Shadow Hand capable of performing pen spinning tricks, adeptly manipulating a
  pen in circles at rapid speed.
paper_authors._id: 6531ddd1a1162a3f87999d40
paper_authors.name: Yecheng Jason Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:32:56.146Z
paper_authors._id: 63d57836652b74a957da1e3f
paper_authors.avatarUrl: /avatars/8200cd11fe245400de7750795323ea24.svg
paper_authors.fullname: Yecheng Ma
paper_authors.type: user
paper_authors.user: Mark10667
paper_authors._id: 6531ddd1a1162a3f87999d41
paper_authors.name: William Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-21T09:22:39.778Z
paper_authors._id: 640fb56dc025ddf6189a891f
paper_authors.avatarUrl: /avatars/a20f74fb0108a8e9ac40ffe60fefa5c7.svg
paper_authors.fullname: Will Liang
paper_authors.type: user
paper_authors.user: willjhliang
paper_authors._id: 6531ddd1a1162a3f87999d42
paper_authors.name: Guanzhi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:33:53.375Z
paper_authors._id: 628353d76db5127097ce65cd
paper_authors.avatarUrl: /avatars/b8e519114d0926db4e2f9a0514e5cb32.svg
paper_authors.fullname: Guanzhi Wang
paper_authors.type: user
paper_authors.user: guanzhi
paper_authors._id: 6531ddd1a1162a3f87999d43
paper_authors.name: De-An Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:34:03.392Z
paper_authors._id: 641d1c5ec3983aa94915c162
paper_authors.avatarUrl: /avatars/127985b837ecf61e43c835deee578b5e.svg
paper_authors.fullname: De-An Huang
paper_authors.type: user
paper_authors.user: deahuang
paper_authors._id: 6531ddd1a1162a3f87999d44
paper_authors.name: Osbert Bastani
paper_authors._id: 6531ddd1a1162a3f87999d45
paper_authors.name: Dinesh Jayaraman
paper_authors._id: 6531ddd1a1162a3f87999d46
paper_authors.name: Yuke Zhu
paper_authors._id: 6531ddd1a1162a3f87999d47
paper_authors.name: Linxi Fan
paper_authors._id: 6531ddd1a1162a3f87999d48
paper_authors.name: Anima Anandkumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-21T09:22:44.711Z
paper_authors._id: 6532920b3e385cfc6002938d
paper_authors.avatarUrl: /avatars/cb9cc6d2733031582c83f56dc6cd1dd5.svg
paper_authors.fullname: Anima Anandkumar
paper_authors.type: user
paper_authors.user: animakumar
paper_title: Eureka: Human-Level Reward Design via Coding Large Language Models
paper_summary: Large Language Models (LLMs) have excelled as high-level semantic planners
  for sequential decision-making tasks. However, harnessing them to learn complex
  low-level manipulation tasks, such as dexterous pen spinning, remains an open
  problem. We bridge this fundamental gap and present Eureka, a human-level
  reward design algorithm powered by LLMs. Eureka exploits the remarkable
  zero-shot generation, code-writing, and in-context improvement capabilities of
  state-of-the-art LLMs, such as GPT-4, to perform evolutionary optimization over
  reward code. The resulting rewards can then be used to acquire complex skills
  via reinforcement learning. Without any task-specific prompting or pre-defined
  reward templates, Eureka generates reward functions that outperform expert
  human-engineered rewards. In a diverse suite of 29 open-source RL environments
  that include 10 distinct robot morphologies, Eureka outperforms human experts
  on 83% of the tasks, leading to an average normalized improvement of 52%. The
  generality of Eureka also enables a new gradient-free in-context learning
  approach to reinforcement learning from human feedback (RLHF), readily
  incorporating human inputs to improve the quality and the safety of the
  generated rewards without model updating. Finally, using Eureka rewards in a
  curriculum learning setting, we demonstrate for the first time, a simulated
  Shadow Hand capable of performing pen spinning tricks, adeptly manipulating a
  pen in circles at rapid speed.
paper_ai_summary: Eureka, an LLM-powered reward design algorithm, generates human-level reward functions enhancing reinforcement learning for complex manipulation tasks and RLHF, including pen spinning with a simulated Shadow Hand.
paper_ai_keywords: Large Language Models LLMs zero-shot generation code-writing in-context improvement evolutionary optimization reinforcement learning reward functions RLHF gradient-free in-context learning curriculum learning simulated Shadow Hand