Tasks involving localization, memorization and planning in partially observable 3D environments are an ongoing challenge in Deep Reinforcement Learning. We present EgoMap, a spatially structured neural memory architecture. EgoMap augments a deep reinforcement learning agent's performance in 3D environments on challenging tasks with multi-step objectives. The EgoMap architecture incorporates several inductive biases including a differentiable inverse projection of CNN feature vectors onto a top-down spatially structured map. The map is updated with ego-motion measurements through a differentiable affine transform. We show this architecture outperforms both standard recurrent agents and state of the art agents with structured memory. We demonstrate that incorporating these inductive biases into an agent's architecture allows for stable training with reward alone, circumventing the expense of acquiring and labelling expert trajectories. A detailed ablation study demonstrates the impact of key aspects of the architecture and through extensive qualitative analysis, we show how the agent exploits its structured internal memory to achieve higher performance.
This very well written and executed paper synthesizes several ideas recently published in the field of deep reinforcement learning-based goal-driven navigation. It elegantly combines these ideas together by presenting a neural agent architecture that consists of:
* a perception module (e.g. a convnet) that extracts coarse visual feature maps s_t from an RGBD image
* a differentiable map canvas M_t that is rotated at each step based on affine egocentric velocity (dx_t, dy_y, d \phi_t)
* differentiable inverse projection mapping, which uses known camera parameters, projective geometry and the depth channel of the image to project the visual feature vectors s_t onto a 2D map and add it to the existing canvas M_t
* a recurrent module (GRU) for update a state h_t that is used for computing the policy distribution and value function
* additional inputs to the policy and value function, that include a global map read r_t, as well as a query q_t (produced by the policy head) based retrieval of features from the map
* position indexing of features retrieved from the map
The algorithm is trained end-to-end, without extra supervision, using Advantage Actor-Critic (A2C) RL. Based on the strong inductive biases regarding the map, namely affine transforms of the map given information about relative movement, and projective geometry transformations of visual features in the map frame, it seems that the question of where to write is solved, and that the network only needs to learn what to write in the differentiable map. Evaluation is done on 3 games in VizDoom: finding the exit of the Labyrinth, object retrieval and find and return / Minotaur. 
Criticism:
The authors could justify better the choice of using the projective geometry inductive prior. They use sentences like "We argue that projective geometry is a strong law imposed on any vision system working from egocentric observations" (not quite related to grid and place cells, despite being in that section) and "this inverse mapping operation is second nature to many organisms" without giving any reference.
Several papers have been published in the last two years, focusing on differential memory architectures with a 2D map structure, projective geometry. This paper goes further by building and iteratively updating a 2D occupancy map using visual features and image geometry, just like RGBD-SLAM (which would merit a citation, e.g., [1] and [2]). This paper essentially combines existing ideas (see table 1): projective geometry, reward-based learning of M_t, RL, multitask navigation, semantic features. While this is not novel, seeing all this combined in a single technique does have merit.
What is disappointing, given that this is a combination paper, is that the environment is so simply, and that photorealistic environments were not tested. For example, the VizDoom environment uses 2D sprites for objects, making the visual feature extraction from objects much simpler. Would the method work equally well with the objects in DeepMind Lab, which are seen from multiple view points? And what in an environment like AdobeIndoorNav?
[1] Henry et al (2010) "RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments"
[2] Izadi et al (2011) "KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera"