In this paper, a new population-guided parallel learning scheme is proposed to enhance the performance of off-policy reinforcement learning (RL). In the proposed scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collaboration with the guidance of the best policy information. The key point is that the information of the best policy is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search region by the multiple learners. The guidance by the previous best policy and the enlarged range enable faster and better policy search. Monotone improvement of the expected cumulative return by the proposed scheme is proved theoretically. Working algorithms are constructed by applying the proposed scheme to the twin delayed deep deterministic (TD3) policy gradient algorithm. Numerical results show that the constructed algorithm outperforms most of the current state-of-the-art RL algorithms, and the gain is significant in the case of sparse reward environment.
The authors propose another method of doing population-based training of RL policies. During the training process, there are N workers running in N copies of the environment, each with different parameter settings for the policies and value networks. Each worker pushes data to a shared replay buffer of experience. The paper claims that a natural approach is to have a chief job periodically poll for the best worker, then replace the weights of each worker with the best one. Whenever this occurs, this reduces the diversity within the population.
In its place, the authors propose a soft-update in the chief. At every merging cycle, the chief queries which worker performs best. If that worker is worker B, it emits pi_B's parameters to each of the other workers. Instead of replacing the parameters exactly, worker i's loss is then augmented by beta * D(pi_i, pi_B), where D is some distance measure that is measured over states sampled from the replay buffer. The "soft" update encourages individual workers to match pi_B without directly replacing their parameters, which maintains diversity in the population. In this work, pi is always represented by a deterministic policy and D is the mean-squared-error in action space (this is argued as equivalent to the KL divergence between the two policies if the policies were represented by Gaussian with the same, constant standard deviation). The beta parameter is updated online using heuristics based on how D(pi_i, pi_B) compares to D(pi_i, old_pi_i). Using TD3 as a base algorithm, the population-based version performs better, and there are ablations for various parts of the population algorithm.
I thought this paper was interesting, but thought it was strange that there were very few comparisons to other population / ensemble-based training methods. In particular they mention the copying problem as a downside of population-based training (PBT), but do not compare against PBT at all. Additionally, my understanding of PBT is that when they replace bad agents with the best agent, they only replace the worst performing agents (not all of them), and they additionally add some random perturbations to their hyperparameter settings. This goes counter to the claim that they collapse the population to a single point- by my reading the exploration step avoids this collapse.
An experiment I'd like to see is trying PBT, where different workers do in fact use different hyperparameters. My understanding is that in P3S-TD3 there is a single hyperparameter setting shared across all workers (plus some hyperparameters deciding the soft update).
I'd also like to see ablations for the Resetting variant (Re-TD3), where only the bottom half or 2/3rds of the workers are reset. This would give empirical evidence for the "population collapse" intuition - we should expect to see some improvements if we avoid totally collapsing the population, while still copying enough to partially exploit the current best setting.
Many inequalities in the paper are argued by compare the expectation of negative Q of one policy to the negative Q of another - I believe the derivations would be much easier to follow if the authors simply multiplied all sides by -1 and adjusted inequalities accordingly. It is much easier to think about Q-value-1 > Q-value-2 rather than -Q-value-1 < -Q-value-2 when trying to interpret what the equation is saying.
For related work, papers on evolutionary strategies and the various self-play-in-a-population papers seem relevant, since these often take the form of having each worker i do a different perturbation that is later merged by a chief.
In Figure 4 it feels weird that results are the regular Mujoco envs for 2 problems and the delayed envs for the other 2 problems. When looking at the appendix, it's rather clearly cherry picked to show the best results in favor of PS3-TD3. I would prefer the Delayed MuJoCo experiments be in a separate figure, or to include the TD3/SAC/ACKTR/PPO/etc. results for the delayed envs as well (these don't appear to be in the appendix)
On the theoretical results: the 1st assumption seems very strong. The first assumption argues that pi_B is always 1-step better than pi_old for every state. That assumption already takes you very far towards arguing "updating pi_old to pi_B is good". The 2nd assumption is more reasonable but I'm confused how rho and d play into the theoretical results. Do they play any role in how much the policy is expected to improve, or do the constants just need to exist?
The last comment on the theory side is that I still don't understand the intuition for why we want to learn beta such that 
KL(pi_new || pi_b) = max {rho * KL_max(pi_new || pi_old), d}
In the practical algorithm, beta is updated online to increase / decrease the importance of the "match pi_B" term if the ratio between the two strays too far from 1 (with the threshold set to [1/1.5, 1.5] in a manner similar to PPO's approach). But why should it be important for the two values to be close to one another? Let me write out the derivation continuing from Eqn (57) in the appendix.
With a substitution that doesn't use (c) to drop the beta * (KL - KL) term, we get
E_{pi_b}[-Q_new] >= E_{pi_new}[-Q_new] + beta * (KL - KL)
-->
E_{pi_new}[Q_new] >= E_{pi_b}[Q_new] + beta * (KL - KL)
Then, in Theorem 1, we recursively apply this inequality, accumulating a number of beta * (KL - KL) terms. In the end we get
Q_new >= (discounted sum rewards from pi_b) + (discounted sum of beta * (KL - KL) with expectation over states from pi_b)
= Q_pi_b + (sum of beta *(KL - KL) terms)
By my reading, shouldn't this mean we want KL(pi_new || pi_b) - max {rho * KL_max(pi_new || pi_old), d} to be as large as possible, rather than 0? The more positive this term is, the more improvement we get between Q_new and Q_pi_b.
--------------------------
Overall, this feels like a good paper, but I'm not too familiar with prior empirical results for population-based RL methods. The ablations suggested that pretty much any reasonable population-based method outperformed using a single worker, and because of this it seems especially important to have ablations to other population-based prior work, rather than just variants of its own method. 
I would be okay with this paper as-is despite some of its flaws, but think it could be better pending rebuttal.
Edit: I have read the author reply and other reviews. I do not plan to change my rating but do think the paper is improved by the added baselines and better explanation of what the beta adaptation rule is doing. I would ask the authors to make sure this description is as clear as possible, as the argued improvement gap seems central to the work.