UNSUPERVISED LEARNING OF GOAL SPACES FOR INTRINSICALLY MOTIVATED GOAL EXPLORATION
Intrinsically motivated goal exploration algorithms enable machines to discover repertoires of policies that produce a diversity of effects in complex environments. These exploration algorithms have been shown to allow real world robots to acquire skills such as tool use in high-dimensional continuous state and action spaces. However, they have so far assumed that self-generated goals are sampled in a specifically engineered feature space, limiting their autonomy. In this work, we propose to use deep representation learning algorithms to learn an adequate goal space. This is a developmental 2-stage approach: first, in a perceptual learning stage, deep learning algorithms use passive raw sensor observations of world changes to learn a corresponding latent space; then goal exploration happens in a second stage by sampling goals in this latent space. We present experiments where a simulated robot arm interacts with an object, and we show that exploration algorithms using such learned representations can match the performance obtained using engineered representations.
[Edit: After revisions, the authors have made a good-faith effort to improve the clarity and presentation of their paper: figures have been revised, key descriptions have been added, and (perhaps most critically) a couple of small sections outlining the contributions and significance of this work have been written. In light of these changes, I've updated my score.]
Summary:
The authors aim to overcome one of the central limitations of intrinsically motivated goal exploration algorithms by learning a representation without relying on a "designer" to manually specify the space of possible goals. This work is significant as it would allow one to learn a policy in complex environments even in the absence of a such a designer or even a clear notion of what would constitute a "good" distribution of goal states.
However, even after multiple reads, much of the remainder of the paper remains unclear. Many important details, including the metrics by which the authors evaluate performance of their work, can only be found in the appendix; this makes the paper very difficult to follow.
There are too many metrics and too few conclusions for this paper. The authors introduce a handful of metrics for evaluating the performance of their approach; I am unfamiliar with a couple of these metrics and there is not much exposition justifying their significance and inclusion in the paper. Furthermore, there are myriad plots showing the performance of the different algorithms, but very little explanation of the importance of the results. For instance, in the middle of page 9, it is noted that some of the techniques "yield almost as low performance as" the randomized baseline, yet no attempt is made to explain why this might be the case or what implications it has for the authors' approach. This problem pervades the paper: many metrics are introduced for how we might want to evaluate these techniques, yet there is no provided reason to prefer one over another (or even why we might want to prefer them over the classical techniques).
Other comments:
- There remain open questions about the quality of the MSE numbers; there are a number of instances in which the authors cite that the "Meta-Policy MSE is not a simple to interpret" (The remainder of this sentence is incomplete in the paper), yet little is done to further justify why it was used here, or why many of the deep representation techniques do not perform very well.
- The authors do not list how many observations they are given before the deep representations are learned. Why is this? Additionally, is it possible that not enough data was provided?
- The authors assert that 10 dimensions was chosen arbitrarily for the size of the latent space, but this seems like a hugely important choice of parameter. What would happen if a dimension of 2 were chosen? Would the performance of the deep representation models improve? Would their performance rival that of RGE-FI?
- The authors should motivate the algorithm on page 6 in words before simply inserting it into the body of the text. It would improve the clarity of the paper.
- The authors need to be clearer about their notation in a number of places. For instance, they use \gamma to represent the distribution of goals, yet it does not appear on page 7, in the experimental setup.
- It is never explicitly mentioned exactly how the deep representation learning methods will be used. It is pretty clear to those who are familiar with the techniques that the latent space is what will be used, but a few equations would be instructive (and would make the paper more self-contained).
In short, the paper has some interesting ideas, yet lacks a clear takeaway message. Instead, it contains a large number of metrics and computes them for a host of different possible variations of the proposed techniques, and does not include significant explanation for the results. Even given my lack of expertise in this subject, the paper has some clear flaws that need addressing.
Pros:
- A clear, well-written abstract and introduction
- While I am not experienced enough in the field to really comment on the originality, it does seem that the approach the authors have taken is original, and applies deep learning techniques to avoid having to custom-design a "feature space" for their particular family of problems.
Cons:
- The figure captions are all very "matter-of-fact" and, while they explain what each figure shows, provide no explanation of the results. The figure captions should be as self-contained as possible (I should be able to understand the figures and the implications of the results from the captions alone).
- There is not much significance in the current form of the paper, owing to the lack of clear message. While the overarching problem is potentially interesting, the authors seem to make very little effort to draw conclusions from their results. I.e. it is difficult for me to easily visualize all of the "moving parts" of this work: a figure showing the relationship bet
- Too many individual ideas are presented in the paper, hurting clarity. As a result, the paper feels scattered. The authors do not have a clear message that neatly ties the results together.