This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is a collection of carefully-designed experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives. First, to collect clear, informative and scalable problems that capture key issues in the design of general and efficient learning algorithms. Second, to study agent behaviour through their performance on these shared benchmarks. To complement this effort, we open source github.com/anon/bsuite, which automates evaluation and analysis of any agent on bsuite. This library facilitates reproducible and accessible research on the core issues in RL, and ultimately the design of superior learning algorithms. Our code is Python, and easy to use within existing projects. We include examples with OpenAI Baselines, Dopamine as well as new reference implementations. Going forward, we hope to incorporate more excellent experiments from the research community, and commit to a periodic review of bsuite from a committee of prominent researchers.
This paper presents the « Behavior Suite for Reinforcement Learning » (bsuite), which is a set of RL tasks (called « experiments ») meant to evaluate an algorithm’s ability to solve various key challenges in RL. Importantly, these experiments are designed to run fast enough that one can benchmark a new algorithm within a reasonable amount of time (and money). They can thus be seen as a « test suite » for RL, limited to small toy problems but very useful to efficiently debug RL algorithms and get an overview of some of their key properties. The paper describes the motivation behind bsuite, shows detailed results from some classical RL algorithms on a couple of experiments, and gives a high-level overview of how the code is structured.
I really believe such a suite of RL tasks can indeed be extremely useful to RL researchers developing new algorithms, and as a result I would like to encourage this initiative and see it published at ICLR to help it gain additional traction within the RL community.
The paper is easy to read, motivates well the reasons behind bsuite, and shows some convincing examples. However, in my opinion there remain a few important issues with this submission:
1.	There is no « related work » section to position bsuite within the landscape of RL benchmarks (ex: DMLab, ALE / MinAtar, MuJoCo tasks, etc.). I believe it is important to add one.
2.	The current collection of experiments appears to be quite limited. The authors acknowledge the lack of hierarchical RL, but what about other aspects like continuous control, parameterized actions, multi-agent, state representation learning, continual learning, transfer learning, imitation learning / inverse RL, self-play, etc? It is unclear to me whether the goal is to grow bsuite in all these directions (and more) over time, or if there is some kind of « boundary » the authors have in mind regarding the scope of bsuite. Regardless, the fact is that in its current form, bsuite appears to be suited only to a limited subset of current RL research.
3.	I wish an anonymized version of the code had been provided, so that reviewers could test it. In particular I wonder (a) if it is easy to setup and run under Windows, and (b) if it is straighforward to plug a bsuite experiment within an algorithm based on the popular OpenAI gym API (I think the latter is true from what is said at the end of Section 4, but I would have appreciated being able to try it out myself).
Additional minor remarks:
•	I noticed two anoymity-related issues with the provided links: (1) the Google Colab notebook revealed to me the name of its author when clicking the « Open in Playground » link to be able to run it, and (2) the bsuite-tutorial link asks for permission, which might let the authors access reviewer info. I would not hold it against the authors though as I believe these are genuine mistakes and they did their best to preserve anonymity.
•	t > 2 in Section 2.1 should probably be t >= 2
•	In FIg. 2b the label for the y axis seems incorrect since good results are near 0
•	Please explain what is the dashed grey line in Fig. 4b
•	I was unable to understand the last 2 sentences of Section 4
•	Sections C.2, D.2 and E.2 all have the same plots
•	A few typos: incomplete sentence near bottom of p.3 (« the internal workings… »), « These assessment », « expeirments », « recurrant », « length ? 1 », « together with an analysis parses this data », « anonimize », « bsuite environments by implementing », « even if require »
Review update: the authors have addressed my concerns, and I look forward to using bsuite in my research => review score increased to "Accept"