Design Principles of the Hippocampal Cognitive Map
Hippocampal place fields have been shown to reflect behaviorally relevant aspects of space. For instance, place fields tend to be skewed along commonly traveled directions, they cluster around rewarded locations, and they are constrained by the geometric structure of the environment. We hypothesize a set of design principles for the hippocampal cognitive map that explain how place fields represent space in a way that facilitates navigation and reinforcement learning. In particular, we suggest that place fields encode not just information about the current location, but also predictions about future locations under the current transition distribution. Under this model, a variety of place field phenomena arise naturally from the structure of rewards, barriers, and directional biases as reflected in the transition policy. Furthermore, we demonstrate that this representation of space can support efficient reinforcement learning. We also propose that grid cells compute the eigendecomposition of place fields in part because is useful for segmenting an enclosure along natural boundaries. When applied recursively, this segmentation can be used to discover a hierarchical decomposition of space. Thus, grid cells might be involved in computing subgoals for hierarchical reinforcement learning.
Beyond representing the position of an animal in a given environment, the activity of neurons in the hippocampus (areas CA1,3) is known to be influenced by a range of task-dependent factors, for instance the presence of a reward at a specific location in the environment; yet we don't fully understand how these representations emerge and what they are good for. The present paper proposes that these observations are a reflection of the circuit implementing a specific algorithm (using a successor representation, SR, initially proposed by Dayan in 1993) for learning state values for reinforcement learning; moreover it suggests that the representation in an upstream region (medial EC) may provide a basis for a hierarchical decomposition of space.  Overall, some of the ideas put forward here are intriguing and potentially interesting for theoretical neuroscientists studying hippocampal coding, however the link to the neural data is relatively weak and the presentation of the material is difficult to follow in places. Detailed comments1. Content:Since the algorithmic part of the paper is not new, the key contribution of this work is the link between the SR representation  and the activity of neurons in the hippocampus. Unfortunately, this link between the two is not clear in several aspects:a) it is never spelled out exactly  how does the matrix M relate to the firing of the neurons in the corresponding hippocampal circuit.If there is a one-to-one map between firing rates and M(s,s'), how can a downstream circuit compute V(s)?One would need to include at least a cartoon version of how the whole algorithm would work at the level of the circuit.b) in SR the representation is a direct reflection of the geometry of the space on one side and the policy on the other. It is the second component that is particularly important for explaining task-dependent effects on neural activity; still, in section 3 it is barely stated why policy corresponds to which experimental result (there is a general description of the different learning algorithms but it is never spelled out what exactly was used when).  This is particularly important from the perspective of the dynamics of learning in the circuit: in an online setting, the policy changes in parallel to the state representation, which presumably could influence the final result (depending on the time constants for learning the different components).c) On a more computational note, the parallel learning of policy and representation can actually slow down learning in a some settings, for instance when the rewards contingencies change (see Dayan 93). In a constantly changing environment, it is not clear to me why it would be a good idea to have a policy-dependent representation rather than just doing proper model-based RL. Presumably it is the ease of learning that makes such a solution still appealing; these issues should be at least mentioned in the discussion. c) I find the link to the Stevens and Coupe behavioural results somewhat tenuous: in particular the notion of a boundary as defined there is slightly different than a simple spatial boundary (be it a 'semi-permeable' one); also I would not ascribe the processing of state boundaries to the hippocampus in particular; if nothing else, this bit needs to be justified more carefully.d) the activity of grid cells in MEC looks nothing like the fields in fig 6. Although the responses are periodic, the geometry is different (hexagonal grid) and border cells have elongated fields, these are restricted to the actual border and do not show any periodicity; I am unaware of any experimental data corresponding to panel c. More generally, I am unaware of any experimental work showing reward-dependent changes in these responses, which would be expected should these neural responses correspond to an eigendecomposition of the SR.Small technical note: the convolution with the Gaussian filter is used in the data analysis to smoothen out some of the noise due to limited sampling of the neural responses. I don't see a reason to use it here, and in fact smoothening the fields may prove misleading. Also, was this smoothing used consistently? (there does not seem to be any in the plots with boundaries)2. Clarity: I find the structure of the paper is somewhat difficult to follow, in particular the fact that important technical details are lumped together in the supplements, often without even being referenced in the main text. Also some of the details are missing, making it impossible to know exactly what was done to obtain the results in individual figures. 3. Significance: once the link to the neural circuit is made explicit and the presentation is improved, this could be a very interesting paper for the hippocampal coding subfield.