Reward learning from human preferences and demonstrations in Atari
To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. We train a deep neural network to model the reward function and use its predicted reward to train an DQN-based deep reinforcement learning agent on 9 Atari games. Our approach beats the imitation learning baseline in 7 games and achieves strictly superhuman performance on 2 games without using game rewards. Additionally, we investigate the goodness of fit of the reward model, present some reward hacking problems, and study the effects of noise in the human labels.
As the title implies, this paper examines imitation learning that combines human demonstrations and human preferences. The main algorithm builds on DQFD to learn Q-Values from human demonstrations and subsequently fine-tunes the policy using preference elicitation methods. More specifically, preferences are compiled into a surrogate reward function which is then used to further optimize the policy. The resulting algorithm is validated on nine Atari environments and results show that the technique of combining demonstrations with preferences is better than either using either source of feedback alone.  Overall, the paper is clearly written, tackles a well-scoped problem, and presents compelling results. It's great to consider learning from more than a single type of human feedback and the combination of human demonstrations with preferences is a promising one.  My largest concern is how reliant the algorithm is on preference feedback induced from the synthetic oracle. As the authors acknowledge, human preference feedback seems to nearly always perform worse than oracle preferences. It was unclear whether the problem stemmed from inexperienced human demonstrators or the simple fact that humans are bad at providing preference feedback. The latter makes me concerned about the ultimate usefulness of this approach.  While it makes sense that the oracle preference feedback was used to automate the experimentation and remove the need for humans, it really detracts from the argument of learning on environments with no reward. I'm afraid the oracle could simply be helping the agent induce a learned version of the environment rewards. If this is the case, the proper point of comparison should be DQFD with access to true environment rewards.  Another experiment that I would like to see would be a comparison between having human annotators spend time providing extra demonstrations versus preference labels. I wonder if the eventual algorithm performance would be better by directing humans to provide additional demonstrations rather than preferences.  After reading the author rebuttal, my evaluation still remains at weak accept.