HIERARCHICAL RL USING AN ENSEMBLE OF PROPRIOCEPTIVE PERIODIC POLICIES
In this work we introduce a simple, robust approach to hierarchically training an agent in the setting of sparse reward tasks. The agent is split into a low-level and a high-level policy. The low-level policy only accesses internal, proprioceptive dimensions of the state observation. The low-level policies are trained with a simple reward that encourages changing the values of the non-proprioceptive dimensions. Furthermore, it is induced to be periodic with the use a "phase function." The high-level policy is trained using a sparse, task-dependent reward, and operates by choosing which of the low-level policies to run at any given time. Using this approach, we solve difficult maze and navigation tasks with sparse rewards using the Mujoco Ant and Humanoid agents and show improvement over recent hierarchical methods.
Brief summary: 
HRL method which uses a 2 level hierarchy for sparse reward tasks. The low level policies are only provided access to proprioceptive parts of the observation, and are trained to maximize change in the non-proprioceptive part of the state as reward. The higher level policy is trained as usual by commanding lower level policies. 
Overall impression:
I think the paper has a major assumption about the separation of internal and external state, thereby setting the form of the low level primitives. This may not be fully general, but is particularly useful for the classes of tasks shown here as seen from the strong results. I would like to see the method applied more generally to other robotic tasks, and a comparison to Florensa et al. And perhaps the addition of a video which shows the learned behaviors. 
Introduction: 
the difficulty of learning a high-level controller when the low-level policies shifts -> look at “data efficient hierarchical reinforcement learning” (Nachum et al)
The basic assumption that we can separate out observations into proprioceptive and not proprioceptive can often be difficult. For example with visual inputs or entangled state representations, this might be very challenging to extract. This idea seems to be very heavily based on what is “internal” and what is “external” to the agent, which may be quite challenging to separate. 
The introduction of phase functions seems to be very specific to locomotion?
 Related work: 
The connection of learning diverse policies should be discussed with Florensa et al, since they also perform something similar with their mutual information term. DeepMimic, DeepLoco (Peng et al) also use phase information in the state, worthwhile to cite. 
Section 3.1:
The pros and cons of making the assumption that representation is disentangled enough to make this separation, should be discussed. 
Also, the internal and external state should be discussed with a concrete example, for the ant for example. 
Section 3.2:
The objective for learning diverse policies is in some sense more general than Florensa et al, but in the same vein of thinking. What are the pros and cons of this approach over that?  The objective is greedy in the change of external state. We’d instead like something that over the whole trajectory maximizes change?
Section 3.3:  How well would these cyclic objectives work in a non-locomotion setting? For example manipulation
Section 3.4: This formulation is really quite standard in many HRL methods such as options framework. The details can be significantly cut down, and not presented as a novel contribution. 
Experiments:
It is quite cool that Figure 2 shows very significant movement, but in some sense this is already supervised to say “move the CoM a lot”. This should be compared with explicitly optimizing for such an objective, as in Florensa et al. I’m not sure that this would qualify as “unsupervised” per se. As in it too is using a particular set of pre-training tasks, just decided by the form of choosing internal and external state.
all of the baselines fail to get close to the goal locations.-> this is a bit surprising? Why are all the methods performing this poorly even when rewarded for moving the agent as much as possible.
Overall, the results are pretty impressive. A video would be a great addition to the paper. 
Comparison to Eysenbach et al isn’t quite fair since that method receives less information. If given the extra information, the HRL method performs much better (as indicated by the ant waypoint plot in that paper).