Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning. In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach. Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial. To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation. We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice. Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods.
 -- Summary -- The authors proposes a novel approach in learning a representation for HRL. They define the notion of sub-optimality of a representation (a mapping from observation space to goal space) for a goal-conditioned HRL that measure the loss in the value as a result of using a representation. Authors then state an intriguing connection between representation learning and bounding the sub-optimality which results in a gradient based algorithm. -- Clarity -- The paper is very well written and easy to follow. -- novelty -- To the best of my knowledge, this is the first paper that formalizes a framework for sub-optimality of goal-conditioned HRL and I think this is the main contribution of the paper, that might have lasting effect in the field. This works mainly builds on top of Nachum et al 2018 for Data Efficient HRL. -- Questions and Concerns -- [1] Authors discussed the quality of the representation by comparing to some near optimal representation (x,y) in section 7. My main concern is how really “good” is the representation?, being able to recover a representation like x,y location is very impressive, but I think of a “good” representation as either a mapping that can be generalized or facilitate the learning (reducing the sample complexity). For example Figure 3 shows the performance after 10M steps, I would like to see a learning curve and comparison to previous works like Nachum et al 2018 and see if this approach resulted in a better (in term of sample complexity) algorithm than HIRO. Or an experiment that show the learned representation can be generalized to other tasks. [2] What are author*s insight toward low level objective function? (equation 5 for example) I think there could be more discussion about equation 5 and why this is a good objective to optimize. For example third term is an entropy of the next state given actions, which will be zero in the case of deterministic environment, so the objective is incentivizing for actions that reduce the distance, and also has more deterministic outcome (it’s kind of like empowerment - klyubin 2015). I’m not sure about the prior term, would be great to hear authors thoughts on that. (a connection between MI is partially answering that question but I think would be helpful to add more discussion about this) [3] Distance function : In section C1 authors mention that they used L2 distance function for agent’s reward. That’s a natural choice, although would be great to study the effect of distance functions. But my main concern is that author*s main claim of a good representation quality is the fact that they recovered similar to XY representation, but that might simply be an artifact of the distance function, and that (x,y) can imitate reward very well. I am curious to see what the learned representation would be with a different distance function. -- I think the paper is strong, and interesting however i’d like to hear authors thoughts on [2], and addressing [1] and [3] would make the paper much stronger.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning. In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach. Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial. To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation. We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice. Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 -- Summary -- The authors proposes a novel approach in learning a representation for HRL. They define the notion of sub-optimality of a representation (a mapping from observation space to goal space) for a goal-conditioned HRL that measure the loss in the value as a result of using a representation. Authors then state an intriguing connection between representation learning and bounding the sub-optimality which results in a gradient based algorithm. -- Clarity -- The paper is very well written and easy to follow. -- novelty -- To the best of my knowledge, this is the first paper that formalizes a framework for sub-optimality of goal-conditioned HRL and I think this is the main contribution of the paper, that might have lasting effect in the field. This works mainly builds on top of Nachum et al 2018 for Data Efficient HRL. -- Questions and Concerns -- [1] Authors discussed the quality of the representation by comparing to some near optimal representation (x,y) in section 7. My main concern is how really “good” is the representation?, being able to recover a representation like x,y location is very impressive, but I think of a “good” representation as either a mapping that can be generalized or facilitate the learning (reducing the sample complexity). For example Figure 3 shows the performance after 10M steps, I would like to see a learning curve and comparison to previous works like Nachum et al 2018 and see if this approach resulted in a better (in term of sample complexity) algorithm than HIRO. Or an experiment that show the learned representation can be generalized to other tasks. [2] What are author*s insight toward low level objective function? (equation 5 for example) I think there could be more discussion about equation 5 and why this is a good objective to optimize. For example third term is an entropy of the next state given actions, which will be zero in the case of deterministic environment, so the objective is incentivizing for actions that reduce the distance, and also has more deterministic outcome (it’s kind of like empowerment - klyubin 2015). I’m not sure about the prior term, would be great to hear authors thoughts on that. (a connection between MI is partially answering that question but I think would be helpful to add more discussion about this) [3] Distance function : In section C1 authors mention that they used L2 distance function for agent’s reward. That’s a natural choice, although would be great to study the effect of distance functions. But my main concern is that author*s main claim of a good representation quality is the fact that they recovered similar to XY representation, but that might simply be an artifact of the distance function, and that (x,y) can imitate reward very well. I am curious to see what the learned representation would be with a different distance function. -- I think the paper is strong, and interesting however i’d like to hear authors thoughts on [2], and addressing [1] and [3] would make the paper much stronger.
assistant
 -- Summary -- The authors proposes a novel approach in learning a representation for HRL. They define the notion of sub-optimality of a representation (a mapping from observation space to goal space) for a goal-conditioned HRL that measure the loss in the value as a result of using a representation. Authors then state an intriguing connection between representation learning and bounding the sub-optimality which results in a gradient based algorithm. -- Clarity -- The paper is very well written and easy to follow. -- novelty -- To the best of my knowledge, this is the first paper that formalizes a framework for sub-optimality of goal-conditioned HRL and I think this is the main contribution of the paper, that might have lasting effect in the field. This works mainly builds on top of Nachum et al 2018 for Data Efficient HRL. -- Questions and Concerns -- [1] Authors discussed the quality of the representation by comparing to some near optimal representation (x,y) in section 7. My main concern is how really “good” is the representation?, being able to recover a representation like x,y location is very impressive, but I think of a “good” representation as either a mapping that can be generalized or facilitate the learning (reducing the sample complexity). For example Figure 3 shows the performance after 10M steps, I would like to see a learning curve and comparison to previous works like Nachum et al 2018 and see if this approach resulted in a better (in term of sample complexity) algorithm than HIRO. Or an experiment that show the learned representation can be generalized to other tasks. [2] What are author*s insight toward low level objective function? (equation 5 for example) I think there could be more discussion about equation 5 and why this is a good objective to optimize. For example third term is an entropy of the next state given actions, which will be zero in the case of deterministic environment, so the objective is incentivizing for actions that reduce the distance, and also has more deterministic outcome (it’s kind of like empowerment - klyubin 2015). I’m not sure about the prior term, would be great to hear authors thoughts on that. (a connection between MI is partially answering that question but I think would be helpful to add more discussion about this) [3] Distance function : In section C1 authors mention that they used L2 distance function for agent’s reward. That’s a natural choice, although would be great to study the effect of distance functions. But my main concern is that author*s main claim of a good representation quality is the fact that they recovered similar to XY representation, but that might simply be an artifact of the distance function, and that (x,y) can imitate reward very well. I am curious to see what the learned representation would be with a different distance function. -- I think the paper is strong, and interesting however i’d like to hear authors thoughts on [2], and addressing [1] and [3] would make the paper much stronger.