Contrastive Example-Based Control
While many real-world problems that might benefit from reinforcement
learning, these problems rarely fit into the MDP mold: interacting with the
environment is often expensive and specifying reward functions is challenging.
Motivated by these challenges, prior work has developed data-driven approaches
that learn entirely from samples from the transition dynamics and examples of
high-return states. These methods typically learn a reward function from
high-return states, use that reward function to label the transitions, and then
apply an offline RL algorithm to these transitions. While these methods can
achieve good results on many tasks, they can be complex, often requiring
regularization and temporal difference updates. In this paper, we propose a
method for offline, example-based control that learns an implicit model of
multi-step transitions, rather than a reward function. We show that this
implicit model can represent the Q-values for the example-based control
problem. Across a range of state-based and image-based offline control tasks,
our method outperforms baselines that use learned reward functions; additional
experiments demonstrate improved robustness and scaling with dataset size.
64c096905a8f5b0d13ba0786
Kyle Hatch
64c096905a8f5b0d13ba0787
Benjamin Eysenbach
64c096905a8f5b0d13ba0788
Rafael Rafailov
extracted_pending
2023-07-26T03:44:17.081Z
64701c98d742e9ef6522598f
/avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
Rafael Rafailov
user
rmrafailov
64c096905a8f5b0d13ba0789
Tianhe Yu
64c096905a8f5b0d13ba078a
Ruslan Salakhutdinov
64c096905a8f5b0d13ba078b
Sergey Levine
64c096905a8f5b0d13ba078c
Chelsea Finn
admin_assigned
2023-07-26T09:10:34.830Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
Contrastive Example-Based Control
While many real-world problems that might benefit from reinforcement
learning, these problems rarely fit into the MDP mold: interacting with the
environment is often expensive and specifying reward functions is challenging.
Motivated by these challenges, prior work has developed data-driven approaches
that learn entirely from samples from the transition dynamics and examples of
high-return states. These methods typically learn a reward function from
high-return states, use that reward function to label the transitions, and then
apply an offline RL algorithm to these transitions. While these methods can
achieve good results on many tasks, they can be complex, often requiring
regularization and temporal difference updates. In this paper, we propose a
method for offline, example-based control that learns an implicit model of
multi-step transitions, rather than a reward function. We show that this
implicit model can represent the Q-values for the example-based control
problem. Across a range of state-based and image-based offline control tasks,
our method outperforms baselines that use learned reward functions; additional
experiments demonstrate improved robustness and scaling with dataset size.
A method for offline, example-based control learns an implicit model of multi-step transitions, outperforming reward function-based methods and demonstrating robustness and scaling with dataset size.
reinforcement learning MDP offline RL reward function transition dynamics high-return states Q-values state-based control image-based control implicit model