Meta-learning is a powerful tool that learns how to quickly adapt a model to new tasks. In the context of reinforcement learning, meta-learning algorithms can acquire reinforcement learning procedures to solve new problems more efficiently by meta-learning prior tasks. The performance of meta-learning algorithms critically depends on the tasks available for meta-training: in the same way that supervised learning algorithms generalize best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We describe a general recipe for unsupervised meta-reinforcement learning, and describe an effective instantiation of this approach based on a recently proposed unsupervised exploration technique and model-agnostic meta-learning. We also discuss practical and conceptual considerations for developing unsupervised meta-learning methods. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design, significantly exceeds the performance of learning from scratch, and even matches performance of meta-learning methods that use hand-specified task distributions in many environments.
The paper considers a particular setting of so-called meta-reinforcement learning (meta-RL) where there is a distribution over reward functions (the transition function is fixed) and with some access to this distribution, the goal is to produce a learning algorithm that  "learns well" on the distribution. At training time, a sample of reward functions are drawn and an algorithm returns a learning program that at test time, can reinforcement learn the test environment as specified by a test reward function. The paper proposes to generate a set of training reward functions instead of relying on some "manual" specification, thus the term "unsupervised" in the title. It also proposes an algorithm, basing on a recent work in skill discovery (DIAYN, not yet peer-reviewed), to find such reward functions. 
Firstly, the exposition is hard to follow. For example, the "Task acquisition via random discriminators" subsection, without first mentioning DIAYN, seems out-of-context: what is D_phi_rand(z|s)? a joint distribution of (reward function, state) makes no sense. It only makes sense when there is a stochastic process, e.g. MDP coupled with a policy. 
Secondly, the reasoning is very informal based on a vague vocabulary (not trying to pick on the authors, these are not uncommon in deep learning literature) are used without rigorous arguments. Section 3.1 brought up a natural objection -- I applaud the authors' self-critique -- based on "no free lunch theorem," but it dismisses it via "the specific choice for the unsupervised learning procedure and meta-learning algorithm can easily impose an inductive bias" without specifying what (and how) choice leads to what "inductive biases." This is crucial as the authors seem to suggest that although the "inductive bias" is important -- task design expresses such -- an unsupervised method, which requires no supervision, can do as well. 
Thirdly, the baseline comparison seems inappropriate to me. The "fair" baseline the authors proposed was to RL a test task from scratch. But this is false as the meta-RL agent enjoys access to the transition dynamics (controlled Markov process, CMP in the paper) during the so-called meta-training (before meta-testing on the test task). In fact, a more appropriate baseline would be initialize an RL agent with with a correct model (if sample complexity in training is not a concern, which seems to be the case as it was never addressed in the paper) or a model estimated from random sample transitions (if we are mindful of sample complexity which seems more reasonable to me). One may object that a (vanilla) policy-gradient method cannot incorporate an environment model but why should we restrict ourselves to these model-free methods in this setting where the dynamics can be accessed during (meta-)training?
Pros:
1. It connects skill discovery and meta-RL. Even though such connection was not made explicitly clear in the writing, its heavy reliance on a recent (not yet peer-reviewed) paper suggests such. It seems to want to suggest it through a kind of "duality" between skills/policies and rewards/tasks (z in the paper denotes the parameter of a reward function and also the parameter of policy). But are there any difference between the two settings? 
Cons:
1. The writing is imprecise and often hard to follow.
2. The setting considered is not well motivated. How does an unsupervised method provide the task distribution before seeing any tasks? 
3. The restriction of tasks to different reward functions made the proposed baseline seem unfairly weak.
In summary, I could not recommend accepting this work as it stands. I sincerely hope that the authors will be more precise in their future writing and focus on articulating and testing their key hypotheses.