Reinforcement learning algorithms have been shown to effectively learn tasks in a variety of static, deterministic, and simplistic environments, but their application to environments which are characteristic of dynamic lifelong settings encountered in the real world has been limited. Understanding the impact of specific environmental properties on the learning dynamics of reinforcement learning algorithms is important as we want to align the environments in which we develop our algorithms with the real world, and this is strongly coupled with the type of intelligence which can be learned. In this work, we study what we refer to as ecological reinforcement learning: the interaction between properties of the environment and the reinforcement learning agent. To this end, we introduce environments with characteristics that we argue better reflect natural environments: non-episodic learning, uninformative "fundamental drive" reward signals, and natural dynamics that cause the environment to change even when the agent fails to take intelligent actions. We show these factors can have a profound effect on the learning progress of reinforcement learning algorithms. Surprisingly, we find that these seemingly more challenging learning conditions can often make reinforcement learning agents learn more effectively. Through this study, we hope to shift the focus of the community towards learning in realistic, natural environments with dynamic elements.
Summary
This paper discusses the value of creating more challenging environments for training reinforcement learning agents. Specifically, the paper focuses on three characteristics of the environment that the paper claims are necessary for developing intelligent agents. The first of these properties is stochasticity in the environment transitions, specifically stochasticity that is independent of the action taken by the agent. The next is sparsity of rewards; discontinuing the use of reward shaping to define desired behavior. Finally the paper argues that environments should not be episodic, that natural environments are continuing tasks so research focus should be around solving continuous tasks.
Review
My primary concern about this paper is the largely insufficient literature review. Many of the claims made in the motivation of this paper are not novel to this paper and are, in fact, incredibly vibrant sub-communities of study within the field of Reinforcement Learning. A more careful literature review should have easily found these communities and more nuanced claims could have been made. I will give concrete examples of the most important cases of missing literature in the following paragraphs, but this list is not exhaustive.
The paper claims that most standard RL environments include detailed reward functions that unnecessarily shape learning and inject bias into the learning process. While I agree that this is problematic, I disagree that this paper provides any novel insights towards this problem. The problem of learning from sparse rewards is well-known in the RL community and is a hot-topic of study. Even in the standard environments cited by the paper we have Montezuma's Revenge and Pitfall, two environment notorious for their difficulty due to sparse reward; each of which with its own host of literature surrounding only the single environment (for instance, I encourage the authors to investigate the highly controversial Go-Explore paper by Uber and its references). Other environments not considered by this paper include the Malmo (Minecraft) learning environment, around which NeurIPS 2019 hosted an extremely sparse reward competition. Another (overlapping) community of RL research interested in the sparse reward setting is the intrinsic reward community, one such paper being Riedmiller and Hafner et al. 2018.
This paper claims that all standard environments are episodic. Of the environments listed as "standard" by this paper, this claim does not even hold. However, there is a large chunk of the RL community that is not represented here. The continual learning and life-long learning communities are focused exclusively on the problem of non-episodic learning. Some example environments used by the community include Malmo, MuJoCo, DeepMind's Lab environment, and many smaller toy domains designed to showcase individual problems including Cart-Pole, RiverSwim, Pendulum, and Acrobot; with the smaller environments from OpenAi Gym cited by this paper. Another smaller community to investigate would be the average reward formulation of the RL problem, which fairly exclusively focuses on the continual learning problem.
Finally, this paper seems vaguely reminiscent of a few particular environments that I have seen in the literature previously. For example, Berkeley's robot task discovery playpen (see for example Singh, Yang, Hartikainen, Finn, Levine 2019). Or an even more similar simulated environment being the Playroom environment by Singh, Barto, Chentanez 2005. Finally, Malmo has been used in a similar way as the tool-building examples mentioned in this paper.
There were a few key issues with the experiments discussed in this paper. The first of which being "Hypothesis 1" which states: "Non-episodic learning is more difficult than episodic learning because the agent must handle a non-stationary learning problem." This hypothesis alone does not appear to be uniformly true. In fact, imagine a simple 5-state random walk markov chain environment without termination. Each state is visited infinitely many times, so the chain is ergodic and there are no non-stationary points. The empirical section uses meta-parameters that were ill-motivated with no discussion about meta-parameter selection. It is critical to point out that the stepsize used for an episodic problem will likely not be the optimal stepsize for the corresponding non-episodic problem, as the magnitude and variance of the considered returns are necessarily different (in response to Figure 3). Further, evaluating over 3 random seeds simply is not sufficient to make any statistically significant claims when comparing any of these curves (for instance in Figure 5).
Additional Comments (do not influence rating)
For a paper exclusively introducing a new control environment, it is critical to include a discussion about the exploration problem. At the very least, I would appreciate seeing sensitivity curves for values of epsilon.
This paper makes many strong claims about the nature of intelligence that are neither supported in the work or are accepted in the community. While it is intuitive that the environment plays a critical role in developing intelligence, the lack of universal definition of intelligence makes this a non-falsifiable claim. Although I appreciate the point the authors are trying to make, which is that RL research frequently is done in the realm of toy simulated domains, I do not think that this paper includes the appropriate supporting evidence to validate such lofty claims.
It would be interesting to change the exploration method from epsilon-greedy to sampling according to the softmax action distribution. This can have dramatically improved performance on non-adversarial exploration problems, and reduces the need for scheduled epsilon decay. It additionally reduces the need for two extra meta-parameters, allowing the empirical claims to be made more strongly without performing some sweep over parameters.
--------------------
Edit after reading discussion, rebuttal, and edits to manuscript.
The paper's intended contribution was different than I had realized during the initial review phase. I appreciate the effort the author put into the response and the changes made to the literature review section of the paper. I think these help to demonstrate the scope and placement of the work considerably.
I still believe the paper over-states the novelty of the results and I still find it difficult to understand their utility. I believe that the entirety of the paper falls under the domain of exploration in RL, but this is not made clear through the introduction or related works section (though the updates to the related works section help considerably). 
Stochasticity in the environment helping the agent to learn is not a surprising finding at all if the exploration method is insufficient. To give a small example, imagine an agent wandering in a tabular gridworld. If the agent has no method of exploration, and the environment has sparse reward, then it is not surprising that the agent would get stuck in a corner. If the environment was modified so that each transition had a 5% chance of randomly going another direction (e.g. the "right" action has a 5% chance of becoming an "up" action), then we've effectively encoded epsilon-greedy exploration through the environment dynamics. All of this comes down to say, it is difficult to separate the dynamics of the environment from the agent's exploration method. I think it would require a careful study that considers these aspects more explicitly.
To summarize, I think the paper could easily be accepted to a future conference, but I think it is important to:
- Make connection between the insights and exploration clear, specifically designing the introduction, lit review, and experiments around this connection.
- Make sure the contributions are extremely clear in the writing. Demonstrate those contributions directly in the empirical section.
- Tone down the claims in the writing. Many of the claims about the state of the field in RL are demonstrably incorrect. I agree with the sentiment trying to be expressed, but the absolutism makes it difficult to separate that sentiment from a deep understanding of the current RL literature with only the most important papers cited, or a fundamentally insufficient lit review. To make claims about the state of a field the lit review should be rather extensive.