Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map. The precise form of this representation is often considered to be a metric representation of space. An internal representation, however, is judged by its contribution to performance on a given task, and may thus vary between different types of navigation tasks. Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment. To focus on internal representations, we split learning into a task-agnostic pre-training stage that modifies internal connectivity and a task-specific Q learning stage that controls the network's output. We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state. These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities. By combining two types of networks in a modular structure, we could get better performance for both regularities. Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes -which can be shaped by pre-training and analyzed using dynamical systems methods. Furthermore, we demonstrate that non-metric representations are useful for navigation tasks, and their combination with metric representation leads to flexibile multiple-task learning.
## Overview
This paper explores how pre-training a recurrent network on different navigational objectives confers different benefits when it comes to solving downstream tasks. First, networks are pretrained on an objective that either emphasizes position (path integration) or landmark memory (identity of the last wall encountered). This pretraining generates recurrent networks of two classes, called PosNets and MemNets (in addition to no pre-training, called RandNets). Surprisingly, the authors found that pre-training confers different benefits that manifests as differential performance of PosNets and MemNets across the suite. Some evidence is provided that this difference has to do with the requirements of the task. Moreover, the authors show how the different pretraining manifests as different dynamical structures (measured using fixed point analyses) present in the networks after pre-training. In particular, the PosNets contained a 2D plane attractor (used to readout position), whereas the MemNets contained clusters of fixed points (corresponding to the previously encountered landmark).
Overall, I thought this was a very interesting paper--it is one of the first papers I have seen that demonstrates how different pre-training requirements both change network dynamics (as measured by fixed points), and how those differences can yield different benefits on downstream navigational tasks.
## Major comments/concerns
- I think the presentation the pretraining objective (eq 3) could be clearer. Is eq 3 what is minimized during pre-training? How are \alpha, \beta, and \gamma chosen? \alpha is used to separate the two types of networks (MemNet from PosNet), which is the critical difference studied in the paper, so it would helpful to go into more detail about what \alpha controls and how it was chosen.
- For the first task, I am surprised that the agent is able to navigate the environment using only the eight neighboring locations. What is the size of the arena? What fraction of the states are simply surrounded on all sides by empty space? It would be informative to show some trajectories of agents solving the basic task.
- For Fig 3A and 3B, it would be nice to show the other network's performance (i.e. show the PosNet on the scaling task in 3A, and the MemNet on the bar task in 3B).
- How come there are no networks that are able to solve both sets of tasks? That is, how come there are no networks in the upper right region of Fig 3C? Does this suggest that an agent needs to combine two separate RNNs to solve the whole suite of tasks?
- What happens if you train recurrent networks with more sophisticated cell architectures (e.g. a GRU or an LSTM)? These are typically easier to train (and using automatic differentiation techniques are also amenable to fixed point analysis).
## Minor comments
- In eq. (1), use `\left(` and `\right)` to make the first set of parentheses have an appropriate height.
- Typo on the first line after eq. (6) (matrices)
- Relevant reference on comparing networks using dynamics around approximate fixed points: https://arxiv.org/abs/1907.08549.