Inspired by neurophysiological discoveries of navigation cells in the mammalian brain, we introduce the first deep neural network architecture for modeling Egocentric Spatial Memory (ESM). It learns to estimate the pose of the agent and progressively construct top-down 2D global maps from egocentric views in a spatially extended environment. During the exploration, our proposed ESM network model updates belief of the global map based on local observations using a recurrent neural network. It also augments the local mapping with a novel external memory to encode and store latent representations of the visited places based on their corresponding locations in the egocentric coordinate. This enables the agents to perform loop closure and mapping correction. This work contributes in the following aspects: first, our proposed ESM network provides an accurate mapping ability which is vitally important for embodied agents to navigate to goal locations.
In the experiments, we demonstrate the functionalities of the ESM network in random walks in complicated 3D mazes by comparing with several competitive baselines and state-of-the-art Simultaneous Localization and Mapping (SLAM) algorithms. Secondly, we faithfully hypothesize the functionality and the working mechanism of navigation cells in the brain. Comprehensive analysis of our model suggests the essential role of individual modules in our proposed architecture and demonstrates efficiency of communications among these modules. We hope this work would advance research in the collaboration and communications over both fields of computer science and computational neuroscience.
Significance of Contributions Unclear
The paper describes a neural network architecture for monocular SLAM that is argued to take inspiration from neuroscience. The architecture is comprised of four components: one that estimates egomotion (HDU) much like prediction in a filtering framework; one that fuses the current image into a local 2D metric map (BVU); one that detects loop closures (PCU); and one that integrates local maps (GU). These modules along with their associated representations are learned in an end-to-end fashion. The method is trained and evaluated on simulated grid environments and compared to two visual SLAM algorithms. 
The contributions and significance of the paper are unclear. SLAM is arguably a solved problem at the scales considered here, with existing solutions capable of performing localization and mapping in large (city-scale), real-world environments. That aside, one can appreciate the merits of representation learning in the context of SLAM and a handful of neural network-based approaches to SLAM and the related problem of navigation have been proposed of-late. However, the paper doesn't do a sufficient job making the advantages of the proposed approach over these methods clear. Further, the paper emphasizes parallels to neuroscience models for navigation as being a contribution, however these similarities are largely hand wavy and one could argue that they also exist for the many other SLAM algorithms that perform prediction (as in HDU), local/global mapping (as in BVU and GU) and loop closure detection (as in PCU). More fundamentally, the proposed method does not appear to account for motion or measurement noise that are inherent in any SLAM problem and, related, does not attempt to model the uncertainty in the resulting map or pose estimates.
The paper evaluates the individual components of the architecture. The results suggest that the different modules are doing something reasonable, though the evaluation is rather limited (in terms of spatial scale) and a bit arbitrary (e.g., comparing local maps to the ground truth at a seemingly arbitrary 32s). The evaluation of the loop closure is limited to a qualitative measure and is therefore not convincing. The authors should quantitatively evaluate the performance of loop closure in terms of precision and recall (this is particularly important given effects of erroneous loop closure detections and the claims that the proposed method is robust). Meanwhile, it isn't clear that much can be concluded from the ablation studies as there is relatively little difference in MSE between the two ablated models.
Additional comments/questions:
* A stated advantage of this method over that of Gupta et al. is that the agent's motion is not assumed to be known. However, it isn't clear whether and how the model incorporates motion or measurement uncertainty, which is fundamental to any SLAM (or navigation) framework.
* Related, an important aspect of any SLAM algorithm is an explicit estimate of the uncertainty in the agent's pose and the map, however it doesn't seem that the proposed model attempts to express this uncertainty.
* The paper claims to estimate the agent's pose as it navigates, but it is not apparent how the pose is maintained beyond estimating egomotion by comparing the current image to a local map.
* Related, it is not clear how the method balances egomotion estimates and exteroceptive measurements (e.g, as are fused with traditional filtering frameworks). There are vague references to "eliminating discrepancies" when merging measurements, but it isn't clear what this actually means, whether the output is consistent, or how the advantages of egomotion estimation and measurements are balanced.
* The BVU module is stated as generating a "local" map, but it is not clear from the discussion what limits the resulting map to the area in the vicinity of the robot vs. the entire environment.
* It is not clear why previous data is transformed to the agent's reference frame as a result of motion vs. the more traditional approach of transforming the agent's pose to a global reference frame.
* The description of loop closure detection and the associated heuristics is confusing. For example, Section 3.4 states that the agent only considers positions that are distant from the most recent visited position as a means of avoiding trival loop closures, however Section 3.4 states that GU provides memory vectors near the current location for loop closure classification.
* The description of the GU module is confusing. How does spatial indexing deal with changes to the map (e.g., as a result of loop closures/measurement updates) or transformations to the robot's frame-of-reference? What are h, H, w, and W and how are they chosen?
* The architecture assumes a discrete (and course) action space, whereas actions are typically continuous. Have the authors tried regressing to continuous actions or experimenting with finer discretizations that are more suitable to real applications?
* It is not clear what is meant by the statement that the PU "learns to encode the representation of visited places".
* The means by which the architecture is trained is unclear. What is the loss that is optimized? How is the triplet loss (Eqn. 3) incorporated (e.g., is it weighted differently than other terms in the loss)?
* Section 3.2 states that the "agent has to learn to take actions to explore its surroundings", however it isn't apparent that the method reasons over the agent's policy. Indeed, this is an open area of research. Instead, the results section suggests that the agent acts randomly.
* Section 4.1 draws comparisons between HDU and Head Direction Cells, however the latter estimate location/orientation whereas the former (this method) predicts egomotion. While egomotion can be integrated to estimate pose (as is done in Fig 4), these are not the same thing.
* The authors are encouraged to tone down claims regarding parallels to navigation models from neuroscience as they are largely unjustified.
* The comparison to existing monocular SLAM baselines is surprising and the reviewer remains skeptical regarding the stated advantages of the proposed method. How much of this difference is a result of testing in simulation? It would be more convincing to compare performance in real-world environments, for which these baselines have proven effective.
* Figure 1: "Border" --> "Boundary"
* Figure 1: The camera image should also go to the BVU block
* Many of the citations are incorrectly not parenthesized
* The paper should be proof-read for grammatical errors