NEAR-OPTIMAL REPRESENTATION LEARNING FOR HIERARCHICAL REINFORCEMENT LEARNING
We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning. In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach. Accordingly, the choice of representation -the mapping of observation space to goal space -is crucial. To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation. We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice. Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods.
The problem setting considered in this paper is that of the recent wave of "goal-conditioned" formulations for hierarchical control in Reinforcement Learning. In this problem, a low-level controller is incentivized to reach a goal state designated by a higher-level controller. This goal is represented in an abstract (embedding) multi-dimensional vector space. Establishing "closeness to goal" entails the existence of some distance metric (assumed to be given an fixed) and a function $f$ which can project states to their corresponding goal representation. The "representation learning" problem referred to by the authors pertains to this function. The paper is built around the question: how does the choice of $f$ affects the expressivity of the class of policies induced in the lower level controller, which in turn affects the optimality of the overall system. The authors answer this question by first providing a bound on the loss of optimality due to the potential mismatch between the distribution over next states under the choice of primitive actions produced by a locally optimal low-level controller. The structure of the argument mimics that of model compression methods based on bisimulation metrics. The model compression here is with respect to the actions (or behaviors) rather than states (as in aggregation/bismulation methods). In that sense, this paper is a valuable contribution to the more general problem of understanding the nature of the interaction between state abstraction and temporal abstraction and where the two may blend (as discussed by Dietterich and MAXQ or Konidaris for example). Using the proposed bounds as an objective, the authors then derive a gradient-based algorithm for learning a better $f$. While restricted to a specific kind of temporal abstraction model, this paper offers the first (to my knowledge) clear formulation  of "goal-conditioned" (which I believe is an expression proposed by the authors) HRL fleshed out of architectural and algorithmic considerations. The template of analysis is also novel and may even be useful in the more general SMDP/options perspective. I recommend this paper for acceptance mostly based on this: I believe that these two aspects will be lasting contributions (much more than the specifics of the proposed algorithms). 
# Comments and Questions
It's certainly good to pitch the paper as a "representation learning" paper at a Representation Learning conference, but I would be careful in using this expression too broadly. The term "representation" can mean different things depending on what part of the system is considered. Representation learning of the policies, value functions etc. I don't have specific recommendations for how to phrase things differently, but please make sure to define upfront which represention you are referring to. 
Representation learning in the sense of let's say Baxter (1995) or Minsky (1961) is more about "ease of learning" (computation, number of samples etc) than "accuracy". In the same way, one could argue that options are more about learning more easily (faster) than for getting more reward (primitive options achieve the optimal). Rather than quantifying the loss of optimality, it would be interesting to also understand how much one gains in terms of convergence speed for a given $f$ versus another. I would like to see (it's up to you) this question being discussed in your paper. In other words, I think that you need to provide some more motivation as to why think the representation learning of $f$ should be equated with the problem of maximizing the return. One reason why I think that is stems from the model formulation in the first place: the low-level controller is a local one and maximizes its own pseudo-reward (vs one that knows about other goals and what the higher level controller may do). It's both a feature, and limitation of this model formulation; the "full information" counterpart also has its drawbacks.
A limitation of this work is also that the analysis for the temporally extended version of the low-level controller is restricted to open-loop policies. The extension to closed-loop policies is important. There is also some arbitrariness in the choice of distance function which would be important to study. 
Relevant work (it's up to you to include or not): 
- Philip Thomas and Andrew Barto in "Motor Primitive Discovery" (2012) also talk about options-like abstraction in terms of compression of action. You may want to have a look. 
- Still and Precup (2011) in "An information-theoretic approach to curiosity-driven reinforcement learning" also talk about viewing actions as "summary of the state" (in their own words). In particular, they look at minimizing the mutual information between state-action pairs. 
- More generally, I think that the idea of finding "lossless" subgoal representations is also related to ideas of "empowerment" (the line of work of Polani).