Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptuallyspecified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects similarity in controllable aspects of the environment instead of distance in the space of observations. We demonstrate the efficacy of our agent to learn, in an unsupervised manner, to reach a diverse set of goals on three domains -Atari, the DeepMind Control Suite and DeepMind Lab.
Summary:
The authors take up an important problem in unsupervised deep reinforcement learning which is to learn perceptual reward functions for goal-conditioned policies without extrinsic rewards from the environment. The problem is important in order to push the field forward to learning representations of the environment without predicting value functions from scalar rewards and learn more generalizable aspects of the environment (the authors call this mastery) as opposed to just memorizing the best sequence of actions in typical value/policy networks. 
Model-based methods are currently hard to execute as far as mastery is concerned and goal-conditioned value functions are a good alternative. The authors, therefore, propose to learn UVFA (Schaul et al) with a learned perceptual reward function r(s, s_g) where 's' and 's_g' are current and goal observations respectively. They investigate a few choices for deriving this reward, such as pixel-space L2 distance, Auto-Encoder feature space, WGAN Discriminator (as done in SPIRAL - Ganin and Kulkarni et al), and their approach: cosine similarity based log-likelihood for similarity metric (as in Matching Networks).  They show that their approach works better than other alternatives on a number of visual goal-based tasks.
Specific aspects:
1. A slight negative: I find the whole pipeline extremely hacky and raises serious questions on whether this paper/technique is easy to apply on a wide variety of tasks. It gives me the suspicion that the environments were cherry-picked for showing the success of the proposed method, though, that's, in general, true of most deep RL papers. It would be nice if the authors instead wrote the paper from the perspective of proposing a new benchmark (it would be amazing if the benchmark is open sourced so that it will lead to more people working specifically on this setting and a lot more comparisons). 
-- Revision: The pipeline is hacky, but getting GAN based reward learning to work is also not very straightforward. The authors do plan to release the detectors used for the benchmarking.
2. To elaborate on the above, these are the portions I find hacky: 
(i) Need for decoy observations to learn an approximate log-likelihood 
(ii) Using sparse reward for all transitions except the final terminal state: Yes, I am aware of the fact that HER has already shown sparse rewards are easier to learn value functions with, compared to dense rewards. But I am genuinely surprised that you have pretty much the same setting (ie re-label only terminal transition, r(s_T, s_g)) and motivate the need for learning a perceptual metric. If the information bits per transition is similar to HER in terms of the policy network's objective function, I am not sure why you need to learn a perceptual reward then? There's also no baseline comparison with just naive HER on image observations. That will be worth seeing actually. I feel this kind of comparisons are more interesting and important for the message of the paper. Note that in other papers cited in this, such as SPIRAL, UPN, etc, the reward metrics are used for every state transition. 
(iii) In addition to naive image HER, I would really like to see a SPIRAL + HER baseline as is. ie use the GAN reward for all transitions and also use relabeling for successes. My prior belief is that this will work really well. I would really like to know how the reward for each transition in the trajectory works (both for SPIRAL and your approach) and how the naive HER works. 
--Revision: The authors have added HER baselines. Agreed with the authors that comparison of per-timestep perceptual reward vs terminal state perceptual reward is a good topic for future work.
3. Another place I really found confusing throughout the paper is the careless swapping of notations, especially in the xi(h) and e(h). Please use consistent notations especially in equation (3), the pseudocode and the rest of the paper. 
4.  a. Would be nice to know if a VAE feature space metric is bad, but not a strict requirement if you don't have time to do it. But I think showing Euclidean metric baseline on VAE is better than an AE. 
      b. Another baseline that is related is to learn a metric with a triplet loss as in Sermanet's work. Or any noise contrastive loss approach (such as CPC). The matching networks approach is similar in spirit. Just pointing out as reference and something worth trying, but not expecting it to be done for rebuttal. 
     
5. Overall, I think this is a good paper, gives a good overview of an important problem; the matching networks idea is nice and simple; but the paper could be more broader in terms of writing than trying to portray the success of DISCERN specifically. I would be happy accepting it even if the SPIRAL baseline or VAE / AE baseline worked as well as the matching networks because I think those approaches are more principled and likely to require fewer hacks and could be applied to a lot of domains easily. I also hope the authors run the baselines I asked for just to make the paper more scientifically complete. 
6. Not a big deal for me in terms of deciding acceptance, but for the sake of good principles in academics, related work could be stronger, though I can understand it must have been small purely due to page limits. 
Some papers which could be cited are (1) Unsupervised Perceptual Rewards (though it uses AlexNet pre-trained), (2) Time Contrastive Networks (which also uses AlexNet and doesn't really work on single-view tasks but is a good citation to add), (3) Original UVFA  (definitely has to be there given you even use the abbreviation for the keywords description of the paper)
7. Some slightly incorrect facts/wording in the paper: The two papers cited in model-based methods (Oh and Chiappa) are not really unsupervised. They use a ton of demonstrations to learn those world models. Better citation might be David Ha's World Models or Chelsea Finn's Video Prediction.