Published as a conference paper at ICLR 2018 INTRINSIC MOTIVATION AND AUTOMATIC CURRICULA VIA ASYMMETRIC SELF-PLAY
We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner. Our scheme pits two versions of the same agent, Alice and Bob, against one another. Alice proposes a task for Bob to complete; and then Bob attempts to complete the task. In this work we will focus on two kinds of environments: (nearly) reversible environments and environments that can be reset. Alice will "propose" the task by doing a sequence of actions and then Bob must undo or repeat them, respectively. Via an appropriate reward structure, Alice and Bob automatically generate a curriculum of exploration, enabling unsupervised training of the agent. When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.
The paper presents a method for learning a curriculum for reinforcement learning tasks.The approach revolves around splitting the personality of the agent into two parts. The first personality learns to generate goals for other personality for which the second agent is just barely capable--much in the same way a teacher always pushes just past the frontier of a student’s ability. The second personality attempts to achieve the objectives set by the first as well as achieve the original RL task.  
The novelty of the proposed method is introduction of a teacher that learns to generate a curriculum for the agent.The formulation is simple and elegant as the teacher is incentivised to widen the gap between bob but pays a price for the time it takes which balances the adversarial behavior. 
Prior and concurrent work on learning curriculum and intrinsic motivation in RL rely on GANs (e.g., automatic goal generation by Held et al.), adversarial agents (e.g., RARL by Pinto et al.), or algorithmic/heuristic methods (e.g., reverse curriculum by Florensa et al. and HER Andrychowicz et al.).  In the context of this work, the contribution is the insight that an agent can be learned to explore the immediate reachable space but that is just within the capabilities of the agent. HER and goal generation share the core insight on training to reach goals. However, HER does generate goals beyond the reachable it instead relies on training on existing reached states or explicitly consider the capabilities of the agent on reaching a goal. Goal generation while learning to sample from the achievable frontier does not ensure the goal is reachable and may not be as stable to train. 
As noted by the authors the above mentioned prior work is closely related to the proposed approach. However, the paper only briefly mentions this corpus of work. A more thorough comparison with these techniques should be provided even if somewhat concurrent with the proposed method. The authors should consider additional experiments on the same domains of this prior work to contrast performance.
Questions:
Do the plots track the combined iterations that both Alice and Bob are in control of the environment or just for Bob?