Consistent and reproducible evaluation of Deep Reinforcement Learning (DRL) is not straightforward. In the Arcade Learning Environment (ALE), small changes in environment parameters such as stochasticity or the maximum allowed play time can lead to very different performance. In this work, we discuss the difficulties of comparing different agents trained on ALE. In order to take a step further towards reproducible and comparable DRL, we introduce SABER, a Standardized Atari BEnchmark for general Reinforcement learning algorithms. Our methodology extends previous recommendations and contains a complete set of environment parameters as well as train and test procedures. We then use SABER to evaluate the current state of the art, Rainbow. Furthermore, we introduce a human world records baseline, and argue that previous claims of expert or superhuman performance of DRL might not be accurate. Finally, we propose Rainbow-IQN by extending Rainbow with Implicit Quantile Networks (IQN) leading to new stateof-the-art performance. Source code is available for reproducibility.
This paper revisits the way RL algorithms are typically evaluated on the ALE benchmark, advocating for several key changes that contribute to more robust and reliable comparisons between algorithms. It also brings the following additional contributions: (1) a new measure of comparison to human performance based on actual human world records (which shows that RL algorithms are not as « super-human » as is generally believed), and (2) an evaluation (based on the proposed guidelines) of Rainbow as well as a Rainbow-IQN variant (replacing the C51 component of Rainbow with Implicit Quantile Networks), showing that the latter brings a significant improvement upon the original Rainbow algorithm.
Overall I am leaning towards acceptance as I believe that such papers encouraging better benchmarking practice on Atari are definitely needed. Even if the technical contribution is limited, this paper could have a positive impact on the field by providing a clearer picture of the current state of deep RL algorithms on Atari (assuming that other researchers start following these recommendations -- and if that is not the case at least it will highlight issues with the way evaluation is currently done).
I do have a few concerns / questions though:
1.	I am not convinced by the recommendation to use performance during training for evaluation purpose. In Machado et al. (2018) it is argued that « this better aligns the performance metric with the goal of continual learning », but most deep RL algorithms trained on Atari games have not been intended to be used in a continual learning setting. It definitely has the advantage of being simple, but it seems to me that it can cause some issues, like making it difficult to compare different exploration techniques for off-policy learning (the exploration may cause poor behavior during training even if it helps the agent learn a better greedy policy), and more generally not being representative of the common practical use case where the goal is to obtain the best agent possible to use in production (with no further learning). Finally, it could make results even harder to reproduce due to the potential high variance of an agent’s performance at a fixed # of timesteps (vs. considering the max performance it can reach over the whole period). As a result, I am currently reluctant to see the proposed performance measure become the standard evaluation metric on ALE, and I would appreciate some additional justification from the authors on this point.
2.	Why not suggest to remove reward clipping in the recommendations? As mentioned in Section 6, reward clipping can prevent RL algorithms from properly playing some games, and thus in my opinion should be removed if the goal is to reach the highest score possible on all games. It seems to me that the choice of clipping the reward should be part of the algorithm (if it is not able to handle the high variety of « raw » rewards) and not of the benchmark environment, thus enabling further advancements towards algorithms that are robust to a wide range of rewards.
3.	Why bother to keep the mean performance when, as mentioned, it is highly sensitive to outliers compared to the median?
Additional remarks:
•	I might have missed it but I do not see the link to the source code. Am I correct to assume it will be released, to help with reproducibility?
•	It is not clear, when reading the paper, that the distributed version of Rainbow is actually constrained to mimic a single agent sequential algorithm in the experiments. I would suggest to remove mentions of the distributed version in the main text to avoid confusion, and mention it only in the Appendix section where it is used.
•	The « infinite reward loop » point at the end of Section 6 does not seem relevant in the list of reasons why Deep RL algorithms are far from the best human performance, since with infinite playtime and an infinite reward loop, the algorithm should be guaranteed to outperform humans.
•	I would have appreciated an evaluation of Rainbow-IQN with the current most commonly used evaluation schemes (e.g. the one used in the original Rainbow paper), for comparison purpose (even if such an evaluation has flaws, it is often the only performance measure available for existing deep RL algorithms)
Review update: thank you for the response, I am currently keeping my "Weak accept" rating because I agree it is important to highlight and (try to) fix the problems with the way algorithms are currently evaluated on ALE, in spite of the limited technical contributions (and the fact that I remained unconvinced regarding #1)