Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach. However, in continuous state and actions spaces and a Gaussian policy -common in computer animation and robotics -PPO is prone to getting stuck in local optima. In this paper, we observe a tendency of PPO to prematurely shrink the exploration variance, which naturally leads to slow progress. Motivated by this, we borrow ideas from CMA-ES, a black-box optimization method designed for intelligent adaptive Gaussian exploration, to derive PPO-CMA, a novel proximal policy optimization approach that can expand the exploration variance on objective function slopes and shrink the variance when close to the optimum. This is implemented by using separate neural networks for policy mean and variance and training the mean and variance in separate passes. Our experiments demonstrate a clear improvement over vanilla PPO in many difficult OpenAI Gym MuJoCo tasks.
This paper proposes an improvement of the PPO algorithm inspired by some components of the CMA-ES black-box optimization method. The authors evaluate the proposed method on a few Mujoco domains and compare it with PPO method using simpler exploration strategies. The results show that PPO-CMA less likely to getting stuck in local optima especially in Humanoid and Swimmer environments. 
Major comments:
The reason that CMAES discards the worst batch of the solutions is that it cannot utilize the quality of the solutions, i.e., it treats every solution equally. But PPO/TRPO can surely be aware of the value of the advantage, and thus can learn to move away from the bad area. The motivation of remove the bad samples is thus not sound, as the model cannot be aware of the areas of the bad samples and can try to repetitively explore the bad area. 
Please be aware that CMAES can get stuck in local optima as well. There is no general convergence guarantee of CMAES.
Detailed comments:
- In page 4, it is claimed that "actions with a negative $A^\pi$ may cause instability, especially when one considers training for several epochs at each iteration using the same data" and demonstrate this with Figure 2. This is not rigorous. If you just reduce all the negative advantage value to zero and calculate its gradient, the method is similar to just use half of step-size in policy gradient. I speculate that if you halve the step-size in "Policy Gradient" setting, the results will be similar to the "Policy Gradient(only positive advantages)" setting. Furthermore, different from importance sampling technique, pruning all the negative advantage will lose much **useful** information to improve policy. So I think this is maybe not a perfect way to avoid instability although it works in experiments.
- There have been a variety of techniques proposed to improve exploration based on derivative-free optimization method. But in my opinion, the way you combine with CMA-ES to improve exploration ability is not so reasonable. Except for the advantage function is change when policy is updated (which has mentioned in "D LIMITATIONS"), I consider that you do not make good use of the exploration feature in CMA-ES. The main reason that CMA-ES can explore better come from the randomness of parameter generation (line 2 in Algorithm 2). So it can generate more diverse policy than derivative-based approach. However, in PPO-CMA, you just replace it with the sampling of policy actions, which is not significant benefit to exploration. It more suitable to say that you "design a novel way to optimize Gaussian policy with separate network for it mean and variance inspired by the CMA-ES method rather than "provides a new link between RL and ES approaches to policy optimization" (in page 10).
- In experiments, there are still something not so clear:
1. In Figure 5, I notice that the PPO algorithm you implemented improved and then drop down quickly in Humanoid-v2 and InvertedDoublePendulum-v2, which like due to too large step-size. Have you tried to reduce it? Or there are some other reasons leading to this phenomenon.
2. What's the purpose of larger budget? You choose a bigger iteration budget than origin PPO implementation.
3. What the experiments have observed may not due to the clipping of negative reward, but could due to the scaling down of the reward. Please try reward normalization.