Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment. This is a critical shortcoming for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -e.g. systems that learn from human interaction. Thus, we develop a novel class of off-policy batch RL algorithms which use KLcontrol to penalize divergence from a pre-trained prior model of probable actions. This KL-constraint reduces extrapolation error, enabling effective offline learning, without exploration, from a fixed batch of data. We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning. This Way Off-Policy (WOP) algorithm is tested on both traditional RL tasks from OpenAI Gym, and on the problem of open-domain dialog generation; a challenging reinforcement learning problem with a 20,000 dimensional action space. WOP allows for the extraction of multiple different reward functions post-hoc from collected human interaction data, and can learn effectively from all of these. We test real-world generalization by deploying dialog models live to converse with humans in an open-domain setting, and demonstrate that WOP achieves significant improvements over state-of-the-art prior methods in batch deep RL.
This paper presents the Way Off-Policy (WOP) algorithm which penalize divergence fro a strong pre-trained prior model as a way to avoid extrapolation error and instability in batch reinforcement learning (BRL). KL-divergence penalties have previously been explored extensively in the on-policy setting (mainly with policy gradient methods) and the authors claim they are the first to apply this idea to off-policy RL. I am not familiar enough with related work to evaluate this statement, but given it’s true, this is an interesting application of these ideas. I also like the application to dialog because pre-trained language models have been shown to offer very good priors for high-quality natural language generation.
Unfortunately, before any other comments I would like to point out that this paper is not properly anonymized. Both the submitted code and the online demo that is linked to from the paper contain the author names and their affiliation (the page linked to in Section 1 contains the author names but the other one in Section 5 does not—it mentions they have been redacted). Apparently there is also an arXiv version of the paper which I was not aware of and which would be fine if this version was not cited from the README file in the submitted code. Please be more careful with respect to anonymizing your submission!
Overall the paper proposes an interesting algorithm that combines existing ideas in a new way and arrives at an algorithm that achieves good performance, based on the experimental evaluation provided in the paper. The experiments performed on dialog are good and I really like that human evaluation was used. However, the traditional RL experiments are lacking. The authors only present results for the CartPole environment in the main paper, which is a very easy problem and so not that interesting. In the appendix they also show results for the Acrobot environment, which are a bit weaker for WOP. Both are quite simple problems and it’s quite easy to learn a good prior model. Why not present results for harder problems? It should be easy to run on more Gym environments and see if WOP still does well in harder problems. Other than that, I like the main ideas in the paper, but I think they could be presented better. Given this and the anonymization issue I lean towards rejecting this paper for now.
Other comments:
(1) I don’t necessarily agree with the statement “Because learning language online from humans on the internet can result in inappropriate behavior (see Horton (2016)), learning offline using BRL is imperative.” I would rather just say that learning offline is one way of dealing with this issue but it’s not the only one or necessarily the best one. In general, the paper defends the batch RL setting quite extensively, but I feel some of the statements may be overly strong. There is a lot of prior work that shows when and why the batch RL setting is useful and so I would add more references to such work.
(2) In Section A.1 you present the implicit metrics that were used and the weight of each metric. These weights seem *extremely* arbitrary to me so could you please provide some insight (and references) as to how they were chosen? I like the results presented in Table 3, in terms of the insight they provide. Are these what you used to derive the weights presented in Section A.1?
(3) The yellow shading in Figure 1 is not really visible. You could try using a different color for the shading (e.g., use gray and make the walls blue or something like that).
(4) The axis labels in the plots of Figure 2 are too tiny. These plots should be made bigger to be readable in printed form. Also, I would recommend you order the labels in the legend in maybe chronological order or something like, which would also put WOP at the top to clarify it’s the proposed method.
Disclaimer: I am not too familiar with this area of research and the related work and so my comments and evaluation should be taken into account in that context.