Published as a conference paper at ICLR 2018 LEARNING AWARENESS MODELS
We consider the setting of an agent with a fixed body interacting with an unknown and uncertain external world. We show that models trained to predict proprioceptive information about the agent's body come to represent objects in the external world. In spite of being trained with only internally available signals, these dynamic body models come to represent external objects through the necessity of predicting their effects on the agent's own body. That is, the model learns holistic persistent representations of objects in the world, even though the only training signals are body signals. Our dynamics model is able to successfully predict distributions over 132 sensor readings over 100 steps into the future and we demonstrate that even when the body is no longer in contact with an object, the latent variables of the dynamics model continue to represent its shape. We show that active data collection by maximizing the entropy of predictions about the bodytouch sensors, proprioception and vestibular information-leads to learning of dynamic models that show superior performance when used for control. We also collect data from a real robotic hand and show that the same models can be used to answer questions about properties of objects in the real world. Videos with qualitative results of our models are available at https://goo.gl/mZuqAV.
Summary:
The paper describes a system which creates an internal representation of the scene given observations, being this internal representation advantageous over raw sensory input for object classification and control. The internal representation comes from a recurrent network (more specifically, a sequence2sequence net) trained to maximize the likelihood of the observations from training
Positive aspects:
The authors suggest an interesting hypothesis: an internal representation of the world which is useful for control could be obtained just by forcing the agent to be able to predict the outcome of its actions in the world. This hypothesis would enable robots to train it in a self-supervised manner, which would be extremely valuable.
Negative aspects:
Although the premise of the paper is interesting, its execution is not ideal. The formulation of the problem is unclear and difficult to follow, with a number of important terms left undefined. Moreover, the experiment task is too simplistic; from the results, it's not clear whether the representation is anything more than trivial accumulation of sensory input
- Lack of clarity:
-- what exactly is the "generic cost" C in section 7.1?
-- why are both f and z parameters of C? f is directly a function of z. Given that the form of C is not explained, seems like f could be directly computing as part of C.
-- what is the relation between actions a in section 7.1 and u in section 4?
-- How is the minimization problem of u_{1:T} solved?
-- Are the authors sure that they perform gathering information through "maximizing uncertainty" (section 7.1)? This sounds profoundly counterintuitive. Maximizing the uncertainty in the world state should result in minimum information about the worlds state. I would assume this is a serious typo, but cannot confirm given that the relation between the minimize cost C and the Renyi entropy H is not explicitely stated. 
-- When the authors state that "The learner trains the model by maximum likelihood" in section 7.1, do they refer to the prediction model or the control model? It would seem that it is the control model, but the objective being "the same as in section 6" points in the opposite direction
-- What is the method for classifying and/or regressing given the features and internal representation? This is important because, if the method was a recurrent net with memory, the differences between the two representations would probably be minimal.
- Simplistic experimental task:
My main intake from the experiments is that having a recurrent network processing the sensory input provides some "memory" to the system which reduces uncertainty when sensory data is ambiguous. This is visible from the fact that the performance from both systems is comparable at the beginning, but degrades for sensory input when the hand is open. This could be achievable in many simple ways, like modeling the classification/regression problem directly with an LSTM for example. Simpler modalities of providing a memory to the system should be used as a baseline.
Conclusion:
Although the idea of learning an internal representation of the world by being able to predict its state from observations is interesting, the presented paper is a) too simplistic in its experimental evaluation and b) too unclear about its implementation. Consequently, I believe the authors should improve these aspects before the article is valuable to the community