DIVERSITY IS ALL YOU NEED: LEARNING SKILLS WITHOUT A REWARD FUNCTION
Intelligent creatures can explore their environments and learn useful skills without supervision. In this paper, we propose "Diversity is All You Need"(DIAYN), a method for learning useful skills without a reward function. Our proposed method learns skills by maximizing an information theoretic objective using a maximum entropy policy. On a variety of simulated robotic tasks, we show that this simple objective results in the unsupervised emergence of diverse skills, such as walking and jumping. In a number of reinforcement learning benchmark environments, our method is able to learn a skill that solves the benchmark task despite never receiving the true task reward. We show how pretrained skills can provide a good parameter initialization for downstream tasks, and can be composed hierarchically to solve complex, sparse reward tasks. Our results suggest that unsupervised discovery of skills can serve as an effective pretraining mechanism for overcoming challenges of exploration and data efficiency in reinforcement learning.
This paper proposes a method for learning skills in absence of a reward function. These skills are learned so that the diversity of the trajectories produced by each skill is maximised. This is achieved by having a discriminator attempting to tell these skills apart. The agent is rewarded for visiting states that are easy to distinguish and the discriminator is trained to better infer the skills from states visited by the agent. Furthermore, a maximum entropy policy is used to force the skills to be diverse. The proposed method is general and any RL algorithm with entropy maximisation in the objective can be used, the implementation in the paper uses the Soft Actor Critic method.
The problem that they are tackling is interesting and is of clear value for obtaining more generalisable RL algorithms. The paper is overall clear and easy to follow, the results are interesting and potentially useful, although I have some reservations regarding how they assess this usefulness in the current version of this paper.
Structure-wise, I would say that the choice of writing the paper in the form of a Q&A, with very brief explanations and details was more distracting and at times unnecessary than I liked (e.g. Question 7 could move to Appendix as it is quite trivial).
I really appreciated how much care has been taken to discuss differences with the closest prior work, Variational Intrinsic Control (VIC) by Gregor et al. 
One such difference is that their prior distribution over skills is not learnt. While there are good arguments by the authors about why this is appealing (e.g. it prevents collapsing to sampling only a few skills), I feel this could be also quite a limitation of their method. This assumes that you have a good a-priori knowledge and assumptions regarding how many skills are useful or needed in the environment. This is unlikely to be the case in complex environments, where you first need to learn simple skills in order to explore the environment, and later learn to form new more complex skills. During this process, you might want to prune simplistic skills after you learnt more abstract and complex ones, for instance in the context of continual learning. I understand this could be investigated in future work, but I feel they take a rather optimistic take on this problem.
Overall, the use case for the proposed method is slightly unclear to me. While the paper claims to allow diverse set of skills to be learnt, it is highly dependent on learning varied action sequences that help you visit different part of state space, regardless of their usefulness. This means there could be learn a lot of skills that capture part of the state space that is not useful or desirable for downstream tasks. While there is a case made for DIAYN being a stepping stone for imitation learning and hierarchical RL, I don’t find the reported experiments for imitation learning and HRL convincing. In the imitation learning experiment, the distance (KL divergence) between all skills and the expert data is computed and the closest skill is then chosen as the policy imitating the expert. The results are weak and no comparisons with any LfD baselines are reported. The HRL experiments also lack comparisons to any other HRL baseline. I feel that this section is rather weak, especially compared to the rest of the paper, and I am not sure it achieves much.
As a general comment, the choice of reporting the training progress using “hours spent training” is an peculiar choice which is never discussed. I understand that for methods with varying computational costs this might be a fairer comparison but it would be perhaps good to also report progress against number of required environment interactions (including pre-training).
Another assumption made is that the method is valuable in situations where the reward function is expensive to compute and the unsupervised pre-training is free (somewhat easing the large amount of pre-training required). However, it would have been interesting to see examples of such environments in their experiments supporting these claims, as this assumption is not valid for the chosen MuJoCo environments.
Despite these comments, I still feel this is valuable work, that can clearly inspire further relevant work and deserves to be presented at ICLR.
It presents a solid contribution, given its technical novelty, proposed applications and its overall generality. 
However, the paper could use more convincing experiments to support its claims.
Additional comments and typos:
- Figure 5 lack error bars across the 5 random seeds and are crucial to assess whether this performance difference is indeed significant given the amount of pre-training required.
- Figure 7’s title and caption is missing...
- typo: page 3, last paragraph “...mutual information between skills and states, **I(S; Z )**” not I(A; Z)
- typo: page 7 paragraph next to Figure 6 “...whereas DIAYN explicitly **learns** skills that effectively partition the state space”
- typo: page 7 above Figure 8 “...make them **exceedingly** difficult for non- hierarchical RL algorithms.”