Machine learning has shown growing success in recent years. However, current machine learning systems are highly specialized, trained for particular problems or domains, and typically on a single narrow dataset. Human learning, on the other hand, is highly general and adaptable. Never-ending learning is a machine learning paradigm that aims to bridge this gap, with the goal of encouraging researchers to design machine learning systems that can learn to perform a wider variety of inter-related tasks in more complex environments. To date, there is no environment or testbed to facilitate the development and evaluation of never-ending learning systems. To this end, we propose the Jelly Bean World testbed. The Jelly Bean World allows experimentation over two-dimensional grid worlds which are filled with items and in which agents can navigate. This testbed provides environments that are sufficiently complex and where more generally intelligent algorithms ought to perform better than current state-of-the-art reinforcement learning approaches. It does so by producing non-stationary environments and facilitating experimentation with multi-task, multi-agent, multi-modal, and curriculum learning settings. We hope that the Jelly Bean World will prompt new interest in the development of never-ending learning, and more broadly general intelligence.
Summary
This paper introduces a new environment for testing lifelong or never-ending learning. The goal of the environment is to act as a new benchmark testbed for challenging existing agents and models across areas of research, encouraging and pushing new research towards solving challenges in curriculum learning, exploration, representation learning, and continual learning. The contributions in this paper extend upon previous work by building an easily controllable environment generator with key necessary features for lifelong learning including: non-stationarity, multiple task specification, and multiple sets of observable features.
Review
The paper highlights many key characteristics of an environment that are challenging to current RL models. This focus on building a benchmark upon which further research can measure performance is important. I find the proposed environment to be incredibly intriguing and would find it valuable to the field of lifelong learning (or continual learning or never-ending learning, etc.). I think the size and scope of the environment generator is impressive, showing a considerable amount of engineering effort has gone into its design.
The largest overarching issue that I would like to point out is the limited study of modelling choices. I am not an expert on applied Reinforcement Learning, so I can make very few claims about the validity of the chosen network architecture or use of the PPO policy-gradient algorithm for this environment. However, it is critical, in my view, that a paper introducing a new environment studies these effects itself; demonstrating how various degrees of learning capacity or wider ranges of learning algorithms behave in the given environment. If a slightly larger network architecture trivially solves each task in this environment, can this still be considered a benchmark task? A key result in the paper that I would like to see further investigated (even with only a different network architecture) would be Figure 6, the comparison between scent, vision, and vision+scent. It is unclear to me why the scent features would be so challenging to learn from and specifically why they would harm the representation so permanently. A deeper study using only the scent features would be valuable to me. In its current state, it appears that these feature provide no additional information and are thus not necessary to include in the environment; breaking one of the primary motivating features of JBW: the multi-modality. I recognize that the paper comments on the orthogonality of the scent playing a role, and notes that further results are included on a not-yet-available website (presumably to maintain anonymity). However, I would like to see these results included in the appendix of the paper so I could better assess the utility of the scent features. Perhaps an additional result showing the average reward versus the cosine distance (or other measure of orthogonality) between "jellybeans" and "onions" would additionally motivate the utility of the scent features.
The paper empirically investigates the use of curriculum learning to accelerate learning for a particular task. The paper then claims that curriculum learning improves learning speed, but ultimately does improve final performance. This demonstration is intended to showcase the use of the proposed environment (JBW) for curriculum learning. However, there are few key issues with this empirical study. First, the paper shows the reward rate of 3 different curricula but does mention the metric used to compare the agents during the time the curricula is active. It is implied that the metric is the reward rate of each individual agent; however, each agent has a unique reward function making comparisons between agents impossible. Curriculum #2 can only receive positive rewards while Curriculum #1 can only receive negative rewards. Naturally this means that Curriculum #2 must have strictly greater or equal reward rate over Curriculum #1. Even in the case that the final objective specifies the metric used, these are still highly non-comparable entities. A suggestion to improve this result would be to run each curricula for 100k as a "pretraining" phase, then to restart the agents to the same state in the environment and measure their performance from there.
The case study measuring the effects of non-stationarity of the rewards does not provide sufficient evidence that the proposed environment contributes a novel ability to investigate non-stationarity. First, the given study of non-stationarity focuses solely on an alternating reward function, clearly demonstrating the problem of catastrophic forgetting. While this is a motivating demonstration, it is not novel and the issue of catastrophic forgetting in our models has been known since at least the 90s (e.g. French 1999 and related). Carefully and scientifically investigating such an issue is best done in a far less complex environment where more precise results can be drawn. Further, the ability to oscillate a reward function in this way is not unique to this environment and can be trivially done in most environments. Secondly, it is unclear if JBW allows for non-stationarity in the transition probabilities in the MDP. This is a critical component to non-stationarity and would be a necessary feature for me to claim non-stationarity is widely supported in the environment.
The paper starts with a motivating conversation about environment complexity, with interesting insights into measuring the complexity of an environment based on the complexity of the policy used to solve that environment. However this conversation is ignored until the conclusion of the paper, where the paper claims to have built an environment of greater complexity than already existing environments. Without any supporting evidence in the body of the paper, it is impossible to verify the validity of this statement, and it is still an open question to me whether this claim is even falsifiable in the first place. As a concrete counter-claim, I would claim that the Minecraft environment (Malmo) has similar or higher complexity to the proposed environment in most aspects. Minecraft has a far greater diversity of objects, a third dimension of movement, adversarial components, hunger and health, etc. each of which adding a large level of complexity not achievable in the proposed environment. This is not to say that I expect the proposed environment to contain these features, but rather to point out that claims of greater complexity may be ill-founded.
Additional Comments (not affecting score)
I do slightly question if ICLR is the appropriate venue for such work. While I recognize that the scope of this conference has shifted considerably over the past few years, this paper (as written) does not further understanding or study of learning representations. I believe a more careful demonstration of the representation induced by characteristics of the environment is within easy reach of the paper, but is not currently presented.
-----------
After the author response, reading other reviews/responses, and looking at the edited draft:
I am convinced of the utility of the domain, the scope of the engineering effort put into building, and the ease with which it can be configured by the user to test many applicable settings (partial observability, stochasticity in transitions and rewards, etc.). I remain slightly skeptical of the amount of benefit the proposed provides over the Malmo environment for any of the settings discussed in the case-studies.
I specifically feel my concerns about the stochasticity in the transitions and environment complexity have been well addressed. My concerns about the curriculum learning demonstration are partially addressed to a point where I am satisfied. My concerns about the modeling choice are also partially satisfied, with one lingering concern. I am unclear if the environment is trivially solvable by using more computation resources (e.g. bigger networks). However, after reconsideration I decided this concern bares less weight than I previously considered.
All this considered, I am changing my rating from 3 -> 6.