Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
<|reference_start|>Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards: Preference-based reinforcement learning (PbRL) aligns a robot behavior with human preferences via a reward function learned from binary feedback over agent behaviors. We show that dynamics-aware reward functions improve the sample efficiency of PbRL by an order of magnitude. In our experiments we iterate between: (1) learning a dynamics-aware state-action representation (z^{sa}) via a self-supervised temporal consistency task, and (2) bootstrapping the preference-based reward function from (z^{sa}), which results in faster policy learning and better final policy performance. For example, on quadruped-walk, walker-walk, and cheetah-run, with 50 preference labels we achieve the same performance as existing approaches with 500 preference labels, and we recover 83\% and 66\% of ground truth reward policy performance versus only 38\% and 21\%. The performance gains demonstrate the benefits of explicitly learning a dynamics-aware reward model. Repo: \texttt{https://github.com/apple/ml-reed}.<|reference_end|>
@article{metcalf2024sample-efficient,
    title={Sample-Efficient Preference-based Reinforcement Learning with Dynamics
  Aware Rewards},
    author={Katherine Metcalf, Miguel Sarabia, Natalie Mackraz, Barry-John
  Theobald},
    journal={arXiv preprint arXiv:2402.17975},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.17975},
    primaryClass={cs.AI cs.LG}
}