Reinforcement learning (RL) is a powerful technique to train an agent to perform a task. However, an agent that is trained using RL is only capable of achieving the single task that is specified via its reward function. Such an approach does not scale well to settings in which an agent needs to perform a diverse set of tasks, such as navigating to varying positions in a room or moving objects to varying locations. Instead, we propose a method that allows an agent to automatically discover the range of tasks that it is capable of performing in its environment. We use a generator network to propose tasks for the agent to try to accomplish, each task being specified as reaching a certain parametrized subset of the state-space. The generator network is optimized using adversarial training to produce tasks that are always at the appropriate level of difficulty for the agent. Our method thus automatically produces a curriculum of tasks for the agent to learn. We show that, by using this framework, an agent can efficiently and automatically learn to perform a wide set of tasks without requiring any prior knowledge of its environment 1 . Our method can also learn to accomplish tasks with sparse rewards, which pose significant challenges for traditional RL methods.
Summary:
This paper proposes to use a GAN to generate goals to implement a form of curriculum learning. A goal is defined as a subset of the state space. The authors claim that this model can discover all "goals" in the environment and their 'difficulty', which can be measured by the success rate / reward of the policy. Hence the goal network could learn a form of curriculum, where a goal is 'good' if it is a state that the policy can reach after a (small) improvement of the current policy.
Training the goal GAN is done via labels, which are states together with the achieved reward by the policy that is being learned.
The benchmark problems are whether the GAN generates goals that allow the agent to reach the end of a U-maze, and a point-mass task.
Authors compare GAN goal generation vs uniformly choosing a goal and 2 other methods.
My overall impression is that this work addresses an interesting question, but the experimental setup / results are not clearly worked out. More broadly, the paper does not address how one can combine RL and training a goal GAN in a stable way.
Pro:
- Developing hierarchical learning methods to improve the sample complexity of RL is an important problem.
- The paper shows that the U-maze can be 'solved' using a variety of methods that generate goals in a non-uniform way.
Con:
- It is not clear to me how the asymmetric self-play and SAGG-RIAC are implemented and why they are natural baselines.
- It is not clear to me what the 'goals' are in the point mass experiment. This entire experiment should be explained much more clearly (+image).
- It is not clear how this method compares qualitatively vs baselines (differences in goals etc).
- This method doesn't seem to always outperform the asymm-selfplay baseline. The text mentions that baseline is less efficient, but this doesn't make the graph very interpretable.
- The curriculum in the maze-case consists of regions that just progress along the maze, and hence is a 1-dimensional space. Hence using a manually defined set of goals should work quite well. It would be better to include such a baseline as well.
- The experimental maze-setting and point-mass have a simple state / goal structure. How can this method generalize to harder problems?
-- The entire method is quite complicated (e.g. training GANs can be highly unstable). How do we stabilize / balance training the GAN vs the RL problem?
-- I don't see how this method could generalize to problems where the goals / subregions of space do not have a simple distribution as in the maze problem, e.g. if there are multiple ways of navigating a maze towards some final goal state. In that case, to discover a good solution, the generated goals should focus on one alternative and hence the GAN should have a unimodal distribution. How do you force the GAN in a principled way to focus on one goal in this case? How could you combine RL and training the GAN stably in that case?
Detailed:
- (2) is a bit strange: shouldn't the indicator say: 1( \exists t: s_t \in S^g )? Surely not all states in the rollout (s_0 ... s_t) are in the goal subspace: the indicator does not factorize over the union. Same for other formulas that use \union.
- Are goals overlapping or non-overlapping subsets of the state space? 
Definition around (1) basically says it's non-overlapping, yet the goal GAN seems to predict goals in a 2d space, hence the predicted goals are overlapping? 
- What are the goals that the non-uniform baselines predict? Does the GAN produce better goals?
- Generating goal labels is
- Paper should discuss literature on hierarchical methods that use goals learned from data and via variational methods:
1. Strategic Attentive Writer (STRAW), V. Mnih et al, NIPS 2016
2. Generating Long-term Trajectories Using Deep Hierarchical Networks. S.
Zheng et al, NIPS 2016