Despite its omnipresence in robotics application, the nature of spatial knowledge and the mechanisms that underlie its emergence in autonomous agents are still poorly understood. Recent theoretical work suggests that the concept of space can be grounded by capturing invariants induced by the structure of space in an agent's raw sensorimotor experience. Moreover, it is hypothesized that capturing these invariants is beneficial for a naive agent trying to predict its sensorimotor experience. Under certain exploratory conditions, spatial representations should thus emerge as a byproduct of learning to predict. We propose a simple sensorimotor predictive scheme, apply it to different agents and types of exploration, and evaluate the pertinence of this hypothesis. We show that a naive agent can capture the topology and metric regularity of its spatial configuration without any a priori knowledge, nor extraneous supervision.
This very interesting paper is based on the sensorimotor contingency theory, which grounds the perception of the agent (motor perception and sensory perception) in the capacity to learn to predict future sensory experience and to build a compact internal representation of proprioception and motor states. They show that through active experience (exploration) of an environment, an agent can encode its motor state in a way that captures both the topology and the relative distances. The authors show that in the case of an environment consistent with sensorimotor transitions and with some changes in the environment that are not inconsistent with the sensorimotor transitions (resets?), the network can learn the representation h of motor that project to the actual position of the agent in the environment (albeit in very contrived 2D toy tasks).
The model does not rely on RL at all; rather, it uses the representation of motor states (proprioception) to make predictions about the sensory observations of the environment. If such an agent were to act, I presume that there would be a search procedure across motor states to make the agent reach a desired state. The paper merits publication at ICLR provided very extensive revisions are made, and I am listing here the improvements to be made.
It is cool to cite Kant, Poincaré and Nicod, whose philosophical work underlies subsequent work on representing space and sensory experience. When citing Kant, please cite the primary source, not the 1998 re-edition.
There are missing references to Wayne et al (2018) "Unsupervised Predictive Memory in a Goal-Directed Agent" arXiv:1803.10760 and Ha & Schmidhuber "World Models" arXiv:1803.10122, where an agent is shown to build a representation of the world that can be decoded into spatial position and even a map of obstacles, on previously unseen environments, using only prediction of images, rewards and actions. Please include them in your work as they considerably change the narrative of section 2 (related work). Essentially, while the claims of the paper are interesting and relevant for the representation learning community, similar work has already been done, at much larger scale, from visual observations, using RL and self-supervised learning.
Section 2 is also somewhat overly critical of previous work: in (Banino et al, 2018; Cueva & Wei, 2018) "rely[ing] on extraneous spatial supervision signals [do not] counter any claim of autonomy", first because these signals can come from sensory perception (e.g., smell) and also because the agent is still autonomous at test time. Similarly, the depth prediction task in (Mirowski et al, 2017) is rather intuitive (stereo-vision).
Part 3 is difficult to parse: it would help to use the word proprioception (or explain why it is inappropriate) when talking about motor states, and exteroception (sensing the environment). I understood the first assumption, which is local continuity in sensory and motor space as well as the ambiguity of redundant motor systems that can generate the same sensory states, but not the second assumption. From what I understand, there are two invariants in building the model of proprioception: invariance to the topology of the environment and to the distances between objects, but then this is hard to reconcile with the setup of (in)consistent transitions in a moving environment and consistent transitions in a static environment. Please rewrite this section in a way that is easier to parse for people who know state-space models and RL for navigation and grasping (who may be your audience). Moreover, all the references point to a single work, which suggests that it is a very peculiar way of approaching a much more general problem of sensorimotor prediction, and therefore begs for a clear and simple explanation.
The architecture of the model is interesting: typically deep RL papers encode the sensory observations s into a hidden representation h, to take actions and produce a motor state m. Here, the current and future motor states m_t and m_{t+1} are embedded into h_t and h_{t+1} using a siamese MLP and used, in combination with the current sensory observation s_t, to make a prediction of s_{t+1}. This is somewhat related to learning the dynamics in model-based RL; please look into and cite Pathak et al (2017) "Curiosity-driven exploration by self-supervised prediction", ICML and other work on intrinsic curiosity.
The experiments are in very simplistic 2D grid world environments, but it makes the analysis and understanding of the 3D representations h much more simpler to follow. On the other hand, the discrete world task is very contrived (especially the weird mapping from m to h and from p to s) and hard to relate to existing work. One difficult problem that is solved by (Banino et al, 2018) is that of path integration in 2D from egocentric velocity. Could the authors present results on such a nonlinear case?
Revision: Score updated from 6 to 7.