title: Contrastive Example-Based Control
summary: While many real-world problems that might benefit from reinforcement
  learning, these problems rarely fit into the MDP mold: interacting with the
  environment is often expensive and specifying reward functions is challenging.
  Motivated by these challenges, prior work has developed data-driven approaches
  that learn entirely from samples from the transition dynamics and examples of
  high-return states. These methods typically learn a reward function from
  high-return states, use that reward function to label the transitions, and then
  apply an offline RL algorithm to these transitions. While these methods can
  achieve good results on many tasks, they can be complex, often requiring
  regularization and temporal difference updates. In this paper, we propose a
  method for offline, example-based control that learns an implicit model of
  multi-step transitions, rather than a reward function. We show that this
  implicit model can represent the Q-values for the example-based control
  problem. Across a range of state-based and image-based offline control tasks,
  our method outperforms baselines that use learned reward functions; additional
  experiments demonstrate improved robustness and scaling with dataset size.
paper_authors._id: 64c096905a8f5b0d13ba0786
paper_authors.name: Kyle Hatch
paper_authors._id: 64c096905a8f5b0d13ba0787
paper_authors.name: Benjamin Eysenbach
paper_authors._id: 64c096905a8f5b0d13ba0788
paper_authors.name: Rafael Rafailov
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-07-26T03:44:17.081Z
paper_authors._id: 64701c98d742e9ef6522598f
paper_authors.avatarUrl: /avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
paper_authors.fullname: Rafael Rafailov
paper_authors.type: user
paper_authors.user: rmrafailov
paper_authors._id: 64c096905a8f5b0d13ba0789
paper_authors.name: Tianhe Yu
paper_authors._id: 64c096905a8f5b0d13ba078a
paper_authors.name: Ruslan Salakhutdinov
paper_authors._id: 64c096905a8f5b0d13ba078b
paper_authors.name: Sergey Levine
paper_authors._id: 64c096905a8f5b0d13ba078c
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-26T09:10:34.830Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_title: Contrastive Example-Based Control
paper_summary: While many real-world problems that might benefit from reinforcement
  learning, these problems rarely fit into the MDP mold: interacting with the
  environment is often expensive and specifying reward functions is challenging.
  Motivated by these challenges, prior work has developed data-driven approaches
  that learn entirely from samples from the transition dynamics and examples of
  high-return states. These methods typically learn a reward function from
  high-return states, use that reward function to label the transitions, and then
  apply an offline RL algorithm to these transitions. While these methods can
  achieve good results on many tasks, they can be complex, often requiring
  regularization and temporal difference updates. In this paper, we propose a
  method for offline, example-based control that learns an implicit model of
  multi-step transitions, rather than a reward function. We show that this
  implicit model can represent the Q-values for the example-based control
  problem. Across a range of state-based and image-based offline control tasks,
  our method outperforms baselines that use learned reward functions; additional
  experiments demonstrate improved robustness and scaling with dataset size.
paper_ai_summary: A method for offline, example-based control learns an implicit model of multi-step transitions, outperforming reward function-based methods and demonstrating robustness and scaling with dataset size.
paper_ai_keywords: reinforcement learning MDP offline RL reward function transition dynamics high-return states Q-values state-based control image-based control implicit model