Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose a simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior -the advantage-weighted behavior model (ABM) -to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources. We find improvements on competitive baselines in a variety of RL tasks -including standard continuous control benchmarks and multitask learning for simulated and real-world robots. Videos are available at https: //sites.google.com/view/behavior-modelling-priors.
This paper proposes a method for “offline RL” (a.k.a “batch RL”), i.e. reinforcement learning from a given static dataset, with no option to perform on-policy data collection. Contrary to prior work (Fujimoto et al, Kumar et al, Agarwal et al) in this area which focuses on making Q-learning robust in the offline RL setting, the authors in this paper instead propose making the policy update step robust to the batch RL setting. They achieve this by constraining the policy that is being learned to stay close to a learned “prior policy” (using KL divergence as a distance metric). The authors provide two different ways of learning this prior policy  - the first one is a simple behavior model policy (which involves fitting a model using maximum likelihood on the entire dataset - effectively performing behavior cloning on the entire dataset), while the second is a more sophisticated “advantage behavior model”, which fits a model only on the “good” data (i.e. environment transitions with a positive advantage, where the advantage is estimated using the current policy). The authors extend the MPO and SVG algorithms to work with this constrained update step. 
The authors provide experiments on the standard DM control suite tasks and some robotic tasks (both simulated and real world). For all experiments, the authors collect data by first running MPO in the standard RL setting (i.e. online data collection), and use the replay buffer from these successful RL runs to relearn a policy. The proposed method performs comparable to other methods in some settings (like all the experiments in the 2K episodes setting in Figure 2), and performs slightly better in some other settings (like Hopper and Quadruped in the 10K episodes setting). The gains are more significant in the non-standard robotics domains (Stack, Stack/Leave). The real world robotics results are not compared to any baseline (perhaps due to evaluation being expensive / time-consuming). However, I think the experimental results, especially Figures 7 and 8, point to a potential flaw in the proposed algorithm. In Figure 7 in the appendix, the learned behavior model (ABM) gets about the same performance as the proposed method (ABM+MPO) in pretty much all the four tasks. In one of the tasks (hopper), ABM+MPO does slightly better than ABM alone, but in this case plane MPO is better than ABM + MPO, indicating that this might just be a task on which MPO does well (for some unexplained reason). Similarly, in the robotics results in Figure 8, ABM alone does comparable to ABM+MPO in five of the seven tasks. 
These experimental results imply to me that most of the performance actually comes from the learned prior (ABM), and whether performing MPO on top of it leads to any improvement or not is hard to predict^[1]. The ABM model itself is somewhat novel, and similar ideas under review at ICLR this year have shown that this alone is a useful algorithm for offline RL, see https://openreview.net/forum?id=H1gdF34FvS and https://openreview.net/forum?id=BJlnmgrFvS. However, the ABM model is not emphasized much in the paper. It is also unclear to my as to why the ABM/BM results were placed in the Appendix for Figure 3, while similar results for experiments in Figure 1 were provided using dotted lines.  
For the paper to be accepted for publication, I think it needs to make a stronger argument (experimentally, at least) about the proposed algorithm being superior to ABM. If this is not really the case, then I think it would require a substantial rewrite to emphasize that ABM is where most of the model performance comes from (as seen in concurrent work listed above). 
[1]: A slight caveat here: in the proposed Algorithm 1, the ABM model is learned online along with the policy, but I believe it could be learned without the policy - for example, by using MC returns as in https://openreview.net/forum?id=H1gdF34FvS. 
----------------------
Edit: The author response has convinced me to bump my rating to a weak accept.