Reinforcement learning requires manual specification of a reward function to learn a task. While in principle this reward function only needs to specify the task goal, in practice reinforcement learning can be very time-consuming or even infeasible unless the reward function is shaped so as to provide a smooth gradient towards a successful outcome. This shaping is difficult to specify by hand, particularly when the task is learned from raw observations, such as images. In this paper, we study how we can automatically learn dynamical distances: a measure of the expected number of time steps to reach a given goal state from any other state. These dynamical distances can be used to provide well-shaped reward functions for reaching new goals, making it possible to learn complex tasks efficiently. We show that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the environment is used to learn the dynamical distances, while a small amount of preference supervision is used to determine the task goal, without any manually engineered reward function or goal examples. We evaluate our method both on a real-world robot and in simulation. We show that our method can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and just ten preference labels, without any other supervision. Videos of the learned skills can be found on the project website:
https://sites.google.com/view/skills-via-distance-learning.
Summary
The authors propose learning the expected # of time steps to reach a given goal state from any other state which they call: dynamical distances. The motivation suggests that such an ability would help the agent in reaching new goal states and therefore in learning complex tasks. Task goal is given with a small amount of preference supervision. The claim is that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the env is used to learn the dynamical distances. Evaluation is done on a real-world robot and in simulation, the method can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and just ten preference labels, without any other supervision. The ideas presented in this work are interesting, but I have some major concerns (please see detailed comments below). The paper is well written and is mostly easy to understand. 
				
Detailed comments:
The learning process follows two steps: 
-distance evaluation step: learn a policy specific dynamical distance parameterized by \psi: by computing the expected # of time steps it took for \pi to reach the goal state.
-policy improvement step: use the learned distance fn to optimize a policy to reach the goal
One weakness of the proposed approach is that the supervised regression step requires an on-policy experience which can be very expensive as pointed out by the authors as well. The authors suggest that because they use this as an intermediate representation this does not impact learning. Wouldn’t it be possible to *simultaneously* learn the dynamical distance during the off-policy policy-learning step?
The second weakness of the proposed approach is that in the policy improvement step it is assumed that a goal state is given. It is also weird to use goals that are already reachable. Please provide clarifications. 
Algorithm 1 learns the distance corresponding to the current policy. Wouldn’t it be a more generalized approach to learn the dynamical distance irrespective of the policy i.e., not policy-specific? Or even better, would it be possible to instead learn the dd applicable to a class of policies? Assuming behaviourally similar policies can have the same dynamical distance? 
On an intuitive level, it would be useful for the reader to make concrete how is this approach different from learning goal conditioned policies? The dynamical distances reflect the distance to the goals: goals are either 1) sampled from an experience replay buffer, 2) given by user preference, 3) chosen more smartly by the dynamical distance learned.
It is not clear if in sec 4.3 “the optimal dynamical distance will be 2”, but since the Eq.2. use the expected distance, does the above statement hold?
Empirical analysis: The authors aim to address three questions through the experiments: Regarding Q2: Is DDL applicable to real-world, vision-based robotic control tasks? Fig 1 shows interesting tasks and a useful contribution. 
In terms of learning from human preferences; the experiments do not seem sufficient and need more comparative analysis. In particular, the authors should compare to other well-known techniques where learning from human preferences is performed such as [1], [2]. 
The same is the case with unsupervised skill acquisition, the authors only compared to Diayn. I would suggest comparing to other relevant baselines such as [3], [4], [5].  Fig 6 primarily suggests the obvious that Diayn maximizes for diversity while DDL maximizes for distance. Moreover, even this comparison seems incomplete as I am not sure why the top row in Fig 6 only shows the proposed approach and not the baseline (diayn). Unsupervised skill acquisition makes it even more important to understand what is the nature of skills learned. I find that analysis is missing here as well, and would add completeness to these results. 
Considering the paper’s claims are heavily based on the ability to learn skills, there is no qualitative analysis of what kind of skills are learned. For e.g. it is mentioned that DDL method that can learn a 9-DoF real-world dexterous manipulation task, but what are the exact skills learned here per task? It would be useful to do a more rigorous analysis of each skill learnt. “Videos of the learned skills can be found on the project website: https://sites.google.com/view/skills-via-distance-learning.” However, the link throws a 404 error.
Overall:		
The paper is well written, real-world experiments are a ++, interesting mix of ideas along with applicability to multiple problems. The main experiment is shown with preferences where comparisons do not seem enough and the paper could be made stronger by thorough comparisons. 
[1] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in Neural Information Processing Systems. 2017.
[2] Ashesh Jain, Brian Wojcik, Thorsten Joachims, and shutosh Saxena. Learning trajectory preferences for manipulators via iterative improvement. In NIPS, 2013.
[3] Gupta, Abhishek, et al. "Unsupervised meta-learning for reinforcement learning." arXiv preprint arXiv:1806.04640 (2018).
[4] Achiam, Joshua, et al. "Variational option discovery algorithms." arXiv preprint arXiv:1807.10299 (2018).
[5] Karol Gregor, Danilo Rezende, and Daan Wierstra. Variational Intrinsic Control. pages 1–15, 2016.