Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance. But in dynamic or sparsely rewarding environments these correlations are often too small, or rewarding events are too infrequent to make learning feasible. Human education instead relies on curricula-the breakdown of tasks into simpler, static challenges with dense rewards-to build up to complex behaviors. While curricula are also useful for artificial agents, handcrafting them is time consuming. This has lead researchers to explore automatic curriculum generation. Here we explore automatic curriculum generation in rich, dynamic environments. Using a setter-solver paradigm we show the importance of considering goal validity, goal feasibility, and goal coverage to construct useful curricula. We demonstrate the success of our approach in rich but sparsely rewarding 2D and 3D environments, where an agent is tasked to achieve a single goal selected from a set of possible goals that varies between episodes, and identify challenges for future work. Finally, we demonstrate the value of a novel technique that guides agents towards a desired goal distribution. Altogether, these results represent a substantial step towards applying automatic task curricula to learn complex, otherwise unlearnable goals, and to our knowledge are the first to demonstrate automated curriculum generation for goal-conditioned agents in environments where the possible goals vary between episodes.
This paper proposes an autocurricula scheme to train a goal-conditional agent in a dynamic and sparse-rewarding environment. The main idea is to train a setter model to sample goals for next-step training, where the setter can make the decision either based on the training history or the environmental observation (conditional case). The paper proposes three criteria which leads to three types of loss to train the setter model, i.e., goal validity (the goal should be achievable by some existing policy), goal feasibility (how probable the current policy can achieve the goal), and goal coverage (the sampled goals by the setter need to cover all possible goals). A judge model is needed to output the feasibility of a given goal. So the autocurricula scheme contains the solver (agent), the setter, and the judge, each having its own combination of loss and they are trained together. Given a desired goal distribution, the paper proposes to additionally train a discriminator whose optimization objective is Wasserstein loss. In experiments, they evaluate the proposed method on three types of tasks in two environments, i.e., 3D color finding and grid world alchemy. The goals in the two environments are similar in that they all aim to achieve some color or color pairs. The difference lies in that the first one finds colors while the second pick up colors. Each environment can be changed between episodes by changing the colors of objects in the scenes. Experimental results show that different combinations of the three types of losses can bring improvements in some scenarios. Making setter and judge conditioned on environment observation can further improve the success rate. Given a desired distribution of goals, the learning becomes more efficient. The paper compares this method with Goal GAN as a baseline and outperforms it on the three tasks.
This paper is an early exploration of the effectiveness of autocurricula on goal-conditional tasks. The experiments to some extent verify the effectiveness of the proposed method. The high-level idea of setter-solver learning can be possibly generalized to other tasks. However, it is not convincing that the detailed techniques proposed in this paper can be easily generalized to more complicated environments and tasks. The writing of this paper is not clear and enough and the whole paper is difficult to understand. The experimental comparison is insufficient since only one baseline of curriculum (goal GAN) is compared (some of the methods mentioned in Section 2 can be easily applied though some of them were not specifically designed for goal-conditional tasks). 
Detailed comments:
1) There are too many details and hyperparameters involved when applied to specific tasks as shown in this paper. Considering the two environments in this paper is easier than most game environments studied in other papers (many of them also uses some types of the curriculum), the experimental result is not very strong. The proposed scheme is possible to be too simple for a slightly more complicated environment or task. 
2) The success of training relies on interactions between the three to four types of losses, but they all have the same weights in the combined loss. Is there any special reason for not setting different weights for different losses?
3) It is not clear how the inverse transform of the setter model (i.e., S^{-1} in L_val) is achieved.
4) There are three random quantities involved in the loss terms, i.e., the noise \xi applied to the sampled goals, the latent input z, the feasibility f. It is not clear whether the randomness on them will dominate the curriculum or not. 
5) The setter model takes z and f as inputs, where z is a vector but f is a scalar. Will it result in a setter model whose output does not change too much when changing f? 
6) How does the proposed method compare to simple goal selection by uncertainty, hardness, or curiosity? How does it compare to hindsight experience reply methods?
7) It is helpful to visualize the generated curriculum, i.e., the trajectory of selected goals during training in the 2D/3D grid.
8) Computing L_judge requires to test whether the agent can finally achieve the sampled goals. Is it too computational expensive during training? 
---------------------------
Update after rebuttal:
Thanks for the detailed reply from the authors! They answer most of my questions. I think this paper has a moderate contribution and thus will keep my "weakly accept" rating.