Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy setting. However, policy gradients can suffer from large variance that may limit performance, and in practice require carefully tuned entropy regularization to prevent policy collapse. As an alternative to policy gradient algorithms, we introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs policy iteration based on a learned statevalue function. We show that V-MPO surpasses previously reported scores for both the Atari-57 and DMLab-30 benchmark suites in the multi-task setting, and does so reliably without importance weighting, entropy regularization, or population-based tuning of hyperparameters. On individual DMLab and Atari levels, the proposed algorithm can achieve scores that are substantially higher than has previously been reported. V-MPO is also applicable to problems with high-dimensional, continuous action spaces, which we demonstrate in the context of learning to control simulated humanoids with 22 degrees of freedom from full state observations and 56 degrees of freedom from pixel observations, as well as example OpenAI Gym tasks where V-MPO achieves substantially higher asymptotic scores than previously reported.
The paper proposes an online variant of MPO, V-MPO. Compared to MPO, the main difference seems to be in the E-step. Instead of optimizing the non-parametric distribution towards a parameterized Q-function, V-MPO learns the V-function and updates the non-parametric distribution towards the advantages, which can be estimated on the samples of the last roll-outs based on the empirical returns and the learned V-function. Of course, updating towards (exponentiated and normalized) Q- or A-function does not make a difference. There are also some minor changes (which might still be crucial) such as an entropy constraint in the M-step and using only the top-k advantages during the E-Step (an option that was discussed in Abdolmaleki, et al. 2018a).
V-MPO is evaluated on DLM, ALE and two humanoid tasks from the DeepMind Control Suite. In most of these tasks V-MPO achieves returns that are to the best of my knowledge higher than any previously reported ones. However, the experiments also use a very large number of system interactions (in the order of billions).
Contribution / Significance:
I think that there will be relatively high interest in the paper due to the reported performances.
The technical contribution seems a bit incremental compared to MPO. Also, by learning a value function V-MPO gets closer to REPS. The submission lists the use of top-k samples and the M-step KL bound as the main differences to REPS. However, the former is not evaluated in the submission and the latter, albeit crucial, seems to be a relatively small modification. I do think that there are more differences to REPS, most important probably in the way of learning the value function and the corresponding differences in the derivations. However, I think that the differences abd similarities to MPO and REPS need to be discussed more thoroughly.
Soundness:
The derivation of V-MPO is relatively sound. The optimization of the KL constraints seems very approximate, although it seems to work well in practice.
Clarity:
I do not like the way the algorithm is presented. The submission specifies the complete loss function already at the beginning of the "Method"-Section and derives/motivates the individual terms in hindsight. The spaghetti-code like structure unnecessarily forces the reader to jump between pages or keeps the reader in the dark. I also do not like the "stop-gradient" notation which in my opinion puts the focus on low-level implementation details at the cost of not properly explaining what the optimization actually does. I think that paper is well-written in general, but the structure needs to be improved.
Experiments:
The evaluation clearly focuses on achieving the best performance, and does a good job in that regard. 
However, a good evaluation should also help in understanding the mechanics of V-MPO. How does k (in top-k) affect the performance? How well are the constraints met during optimization? How does V-MPO compare to related on-policy methods (e.g. TRPO/PPO) on slightly more computational constrainted settings (eg rllab/mujoco with < 1e7 steps)?
Questions:
- Did you experiment with controlled entropy reduction akin to MORE, instead of using fixed 
epsilon eta?
- Can you give a rough estimate of the computational time required to perform these experiments on a standard desktop pc? It really is difficult for me to even roughly estimate it.
Assessment:
Currently, I am leaning to accept because I think that V-MPO is overall a nice work. However I do think that submission needs to be revised. I mainly think that the structure needs to be improved and that V-MPO needs to better related to closely related work. I listed some additional experiments that would significantly improve the submission in my opinion.