LEARNING ACTIONABLE REPRESENTATIONS WITH GOAL-CONDITIONED POLICIES
Representation learning is a central challenge across a range of machine learning areas. In reinforcement learning, effective and functional representations have the potential to tremendously accelerate learning progress and solve more challenging problems. Most prior work on representation learning has focused on generative approaches, learning representations that capture all the underlying factors of variation in the observation space in a more disentangled or well-ordered manner. In this paper, we instead aim to learn functionally salient representations: representations that are not necessarily complete in terms of capturing all factors of variation in the observation space, but rather aim to capture those factors of variation that are important for decision making -that are "actionable." These representations are aware of the dynamics of the environment, and capture only the elements of the observation that are necessary for decision making rather than all factors of variation, eliminating the need for explicit reconstruction. We show how these learned representations can be useful to improve exploration for sparse reward problems, to enable long horizon hierarchical reinforcement learning, and as a state representation for learning policies for downstream tasks. We evaluate our method on a number of simulated environments, and compare it to prior methods for representation learning, exploration, and hierarchical reinforcement learning.
In this paper, the authors propose a new approach to representation learning in the context of reinforcement learning.
The main idea is that two states should be distinguished *functionally* in terms of the actions that are needed to reach them,
in contrast with generative methods which try to capture all aspects of the state dynamics, even those which are not relevant for the task at hand.
The method of the authors assumes that a goal-conditioned policy is already learned, and they use a Kullback-Leibler-based distance
between policies conditioned by these two states as the loss that the representation learning algorithm should minimize.
The experimental study is based on 6 simulated environments and outlines various properties of the framework.
Overall, the idea is interesting, but the paper suffers from many weaknesses both in the framework description and in the experimental study that make me consider that it is not ready for publication at a good conference like ICLR.
The first weakness of the approach is that it assumes that a learned goal-conditioned policy is already available, and that the representation extracted from it can only be useful for learning "downstream tasks" in a second step. But learning the goal-conditioned policy from the raw input representation in the first place might be the most difficult task. In that respect, wouldn't it be possible to *simultaneously* learn a goal-conditioned policy and the representation it is based on? This is partly suggested when the authors mention that the representation could be learned from only a partial goal-conditioned policy, but this idea definitely needs to be investigated further.
A second point is about unsufficiently clear thoughts about the way to intuitively advocate for the approach. The authors first claim that two states are functionally different if they are reached from different actions. Thinking further about what "functionally" means, I would rather have said that two states are functionally different if different goals can be reached from them. But when looking at the framework, this is close to what the authors do in practice: they use a distance between two *goal*-conditioned policies, not *state*-conditioned policies. To me, the authors have established their framework thinking of the case where the state space and the goal space are identical (as they can condition the goal-conditioned policy by any state=goal). But thinking further to the case where goals and states are different (or at least goals are only a subset of states), probably they would end-up with a different intuitive presentation of their framework. Shouldn't finally D_{act} be a distance between goals rather than between states?
Section 4 lists the properties that can be expected from the framework. To me, the last paragraph of Section 4 should be a subsection 4.4 with a title such as "state abstraction (or clustering?) from actionable representation". And the corresponding properties should come with their own questions and subsection in the experimental study (more about this below).
About the related work, a few remarks:
- The authors do not refer to papers about using auxiliary tasks. Though the purpose of these works is often to supply for additional reward signals in the sparse reward context, then are often concerned with learning efficient representations such as predictive ones.
- The authors refer to Pathak et al. (2017), but not to the more recent Burda et al. (2018) (Large-scale study of curiosity-driven learning) which insists on the idea of inverse dynamical features which is exactly the approach the authors may want to contrast theirs with. To me, they must read it.
- The authors should also read Laversanne-Finot et al. (2018, CoRL) who learn goal space representations and show an ability to extract independently controllable features from that.
A positive side of the experimental study is that the 6 simulated environments are well-chosen, as they illustrate various aspects of what it means to learn an adequate representation. Also, the results described in Fig. 5 are interesting. A side note is that the authors address in this Figure a problem pointed in Penedones et al (2018) about "The Leakage Propagation problem" and that their solution seems more convincing than in the original paper, maybe they should have a look.
But there are also several weaknesses:
- for all experiments, the way to obtain a goal-conditioned policy in the first place is not described. This definitely hampers reproducibility of the work. A study of the effect of various optimization effort on these goal-conditioned policies might also be of interest.
- most importantly, in Section 6.4, 6.5 and 6.6, much too few details are given. Particularly in 6.6, the task is hardly described with a few words. The message a reader can get from this section is not much more than "we are doing something that works, believe us!". So the authors should choose between two options:
* either giving less experimental results, but describing them accurately enough so that other people can try to reproduce them, and analyzing them so that people can extract something more interesting than "with their tuning (which is not described), the framework of the authors outperforms other systems whose tuning is not described either".
* or add a huge appendix with all the missing details.
I'm clearly in favor of the first option.
Some more detailed points or questions about the experimental section:
- not so important, Section 6.2 could be grouped with Section 6.1, or the various competing methods could be described directly in the sections where they are used.
- in Fig. 5, in the four room environment, ARC gets 4 separated clusters. How can the system know that transitions between these clusters are possible?
- in Section 6.3, about the pushing experiment, I would like to argue against the fact that the block position is the important factor and the end-effector position is secundary. Indeed, the end-effector must be correctly positioned so that the block can move. Does ARC capture this important constraint?
- Globally, although it is interesting, Fig.6 only conveys a quite indirect message about the quality of the learned representation.
- Still in Fig. 6, what is described as "blue" appears as violet in the figures and pink in the caption, this does not help when reading for the first time.
- In Section 6.4, Fig.7 a, ARC happens to do better than the oracle. The authors should describe the oracle in more details and discuss why it does not provide a "perfect" representation.
- Still in Section 6.4, the authors insist that ARC outperforms VIME, but from Fig.7, VIME is not among the best performing methods. Why insist on this one? And a deeper discussion of the performance of each method would be much more valuable than just showing these curves.
- Section 6.5 is so short that I do not find it useful at all.
- Section 6.6 should be split into the HRL question and the clustering question, as mentioned above. But this only makes sense if the experiments are properly described, as is it is not useful.
Finally, the discussion is rather empty, and would be much more interesting if the experiments had been analyzed in more details.
typos:
p1: that can knows => know
p7: euclidean => Euclidean