Deep Reinforcement Learning from Human Preferences
For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than 1% of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any which have been previously learned from human feedback.
Overall I find this paper is generally interesting, clearly presented, and technically sound. My concerns are that the contributions of this paper seems rather incremental when compared to previous work. Also, some of the experiments would benefit from further analysis. Let me elaborate below:
Summary: This paper advocates a preference-based approach for teaching an RL agent to perform a task. A human observes two trajectories generated by different policies and indicates which one is better performing the desired task. Using these indications, the agent infers a reward signal that is hopefully consistent with the preferences of the human, and then uses RL to optimize a policy that maximizes the inferred rewards.
My main hesitation for accepting this paper is that the method presented is not sufficiently different from previous work on preference-based RL (such as Wirth '16). The authors acknowledge that the main contribution of this work is to scale up preference-based RL to more complex domains by using deep reinforcement learning. I am less enthusiastic about this paper if the main algorithmic contribution is just using a NN to approximate the policy and extending to Mujoco/Atari.
Finally, I am concerned with the fact that synthetic queries are outperforming RL using the true reward function. It seems to me that the synthetic query examples should do strictly worse than having the actual reward function as they are just eliciting preferences based on sub-trajectories of the real rewards. The fact that they are in some cases solidly outperforming standard RL is highly unexpected and, in my opinion, deserves a more rigorous analysis. I don't find the authors explanation of the learned reward function being smoother a satisfying one.