We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms, Proximal Policy Optimization and Trust Region Policy Optimization. We investigate the consequences of "codelevel optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b)  fundamentally change how RL methods function. These insights show the difficulty, and importance, of attributing performance gains in deep reinforcement learning.
Summary
This paper calls to attention the importance of specifying all performance altering implementation details that are current inherent in the state-of-the-art deep policy gradient community. Specifically, this paper builds very closely on the work started by  Henderson et al. 2017, building a conversation around the importance of more rigorous and careful scientific study of published algorithms. This paper identifies many "code-level optimizations" that account for the differences between the popular TRPO and PPO deep policy gradient algorithms. The paper then subselects four of these optimizations and carefully investigates their impact on the final performance of each algorithm. The clear conclusion from the paper is that the touted algorithmic improvement of PPO over TRPO has negligible effect on performance, and any previously reported differences are due only to what were considered unimportant implementation details.
Review
This paper investigates the claims made by Schulman et al. 2017 carefully, by investigating the impact of PPO's clipping mechanism on maintaining a valid trust-region; the central claim made by PPO's originating paper. The empirical results suggest that PPO is not sufficient for maintaining a valid trust-region, however the "code-level optimizations" that differ between the TRPO implementation the PPO implementation are sufficient. The ablation study of the four optimizations studied by the paper shows dramatic and clear results suggesting that annealing stepsizes and normalize rewards make very strong differences in learning performance; much more effect than demonstrated by the differences between TRPO and PPO's core algorithmic contribution as demonstrated in Figure 2 and even more strongly in Figure 3. I find the work included in this paper to be novel and a valuable contribution to the field.
For the above reasons, I recommend to accept this paper for publication at ICLR. In the following paragraphs I will discuss why I only recommend a weak accept instead of a strong accept.
My primary concern with the empirical study is the use of only three random seeds. As demonstrated in Henderson et al. 2017 (which is heavily cited in this paper), using such a small number of random seeds can have very misleading results. Although the effects appear very strong in the empirical studies in this paper, the effects likewise appear strong in Henderson et al.'s Figure 6 where 10 random seeds were split into two groups for the same algorithm. For this paper to make such strong claims about the negligence of the careful scientific study on TRPO and PPO, it would be best if this paper included far more random seeds in its investigation.
My second concern is with the discussion and conclusions drawn from Tables 1 and 2. It appears that the inclusion of clipping plays a strong role in the variance of each algorithm on every domain except Hopper. Specifically, the algorithms that include clipping appear to be much lower variance than the algorithms including clipping. Admittedly using only 3 seeds means that investigating the variance appropriately is near impossible (see the above paragraph), however variance should be considered and discussed in a conversation about the effects of the core contribution of PPO. If clipping leads to more consistent results across runs, even if those results are a little worse, it is still a valid and important contribution.
The paper cites Henderson et al. 2017 in several places. I would point out (perhaps in the introduction) that this paper builds on work already done in Henderson et al. 2017. Specifically, Henderson et al. 2017 investigates the effects of using different codebases for TRPO and shows that these different codebases result in dramatically different performance. The similarity to the investigation in this paper to too close to be unreported. However, I find that the investigation in this paper is much more complete and insightful than that of Henderson et al. 2017 (this paper has a more narrow focus), thus contributes significantly and meaningfully to this ongoing conversation.
Additional Comments (do not affect score)
It might be worthwhile to move the related work section to the beginning of the paper, either merged with the introduction or immediately after. This section is of critical importance to understanding the scope of this paper and for understanding why you are studying what you study. In fact, there is already a bit of duplication between the related works and introduction sections, so the paper could likely gain some additional real-estate by combining these.
I disagree with the terminology "code-level optimizations" and I find that it is misleading. This caused a bit of confusion on my first pass reading the paper, as I originally was expected the code differences to be more akin to using Tensorflow vs PyTorch or switching hash table functions, etc. Instead the changes focused on in this paper are changes to the problem specification and algorithm implementation. These are not simply implementation details as "code-level optimizations" suggests, but are rather details that necessarily must be included in peer-reviewed works. I don't have a suggested name to switch to, but felt strongly enough to mention it.