This work presents a modular and hierarchical approach to learn policies for exploring 3D environments. Our approach leverages the strengths of both classical and learning-based methods, by using analytical path planners with learned mappers, and global and local policies. Use of learning provides flexibility with respect to input modalities (in mapper), leverages structural regularities of the world (in global policies), and provides robustness to errors in state estimation (in local policies). Such use of learning within each module retains its benefits, while at the same time, hierarchical decomposition and modular training allow us to sidestep the high sample complexities associated with training end-to-end policies. Our experiments in visually and physically realistic simulated 3D environments demonstrate the effectiveness of our approach over past learning and geometry-based approaches. 1 1 INTRODUCTION Navigation is a critical task in building intelligent agents. Navigation tasks can be expressed in many forms, for example, point goal tasks involve navigating to a specific coordinates and semantic navigation involves finding path to a specific scene or object. Such tasks may need to be performed in known (already mapped) or unknown environments. Irrespective of the task or the setting, a core problem in navigation is exploration, i.e., how to efficiently visit as much of the environment. This is useful for pre-mapping in known environments, or actually solving tasks in known environments.
Recent work from  has used end-to-end learning to tackle this problem. Their motivation is three fold: a) learning provides flexibility to the choice of input modalities (classical systems rely on observing geometry through use of specialized sensors, while learning systems can infer geometry directly from RGB images), b) use of learning can improve robustness to errors in explicit state estimation, and c) learning can effectively leverage structural regularities of the real world, leading to more efficient behavior in previously unseen environments. This lead to their design of an end-to-end trained neural network based policy that processed raw sensory observations to directly output actions that the agent should execute.
While use of learning for exploration is well motivated, casting the exploration problem as an end-to-end learning problem has its own drawbacks. Learning about mapping, state-estimation and path-planning purely from data in an end-to-end manner can be prohibitively expensive. Consequently, past end-to-end learning work for exploration from  relies on use of imitation learning and many millions of frames of experience, but still performs worse than classical methods that don't require any training at all. This motivates our work. In this paper, we investigate alternate formulations of employing learning for exploration that retains the advantages that learning has to offer, but doesn't suffer from the drawbacks of full-blown end-to-end learning. Our key conceptual insight is that use of learning for leveraging structural regularities of indoor environments, robustness to state-estimation errors, and flexibility with respect to input modalities, happens at different time scales and can thus be factored out. This motivates use of learning in a modular and hierarchical fashion inside of what one may call a 'classical navigation pipeline'. This results in navigation policies that can work with raw sensory inputs such as RGB images, are robust to state estimation errors, and leverage regularities of real 1 See https://sites.google.com/view/active-neural-mapping for visualization videos.
The paper describes a method for visual robot navigation in simulated environments. In terms of overall objectives and targeted reasoning, the current approaches can be roughly divided into two groups: i) learning tasks requiring high-level reasoning for navigation involving the detection and discovery of objects and their affordances and eventually also requiring to process language input, and ii) simpler navigation task involving geometry and the detection of free (navigable space): point goal, maximizing coverage etc. The former target more complex problems but the agents are more difficult (currently up to impossible) to transfer to real environments, whereas the latter directly target problems which can currently realistically used in real world scenarios.
The paper is of the second group, and addresses one of the currently investigated problems in robot navigation and mapping, namely whether learned navigation is superior to traditional planning algorithms, and whether the two different approaches can be integrated. It proposes to separate the task into long-term and short-term goals, which is not new per se, but the proposed formulation is quite interesting. In particular, the integration of the “handcrafted” planar (front propagation) into the learned framework solves a couple of issues with sample efficiency of learned methods, while still keeping some flexibility of learning over the 100% traditional approaches.
I will be upfront – I already reviewed an earlier version of this paper for NeurIPS 2019, where this paper unfortunately did not pass. I was actually a favorable reviewer at this time and was defending it. The paper has been improved since and I would be happy to see it pass. I still have a couple of questions, some of which are similar to the ones I raised in the NeurIPS review (others have been addressed since).
While I do agree that the targeted tasks might be considered less exciting then tasks involving high level semantics, I do also think that these tasks are far from solved as soon as we try to implement them in real life scenarios. I do think that the proposed paper is an interesting step forward.
The advantages of the proposed method are “bought” with a couple of key design choices, in particular the handcrafted non-differentiable long-term path planner. The downside of this is that the loss signals can’t be backpropagated through the planner, which restricts the mapping module to very simple mapping information, basically free /navigational space. End-to-end training of navigation could in principle learn to map objects and affordances which are discovered through the task and not hardcoded or even learned with supervision, which also must be known in advance. This means that the contribution is limited to simpler navigational tasks like the tested exploration and PointGoal. In contrast, other work from the literature uses differentiable planners (eg cited CMP (Gupta et al 2017), using value Iteration Networks (cited Talmar et al. 2016) which allows fine-tuning.
The mapping network, which is learned with supervision, is a general encoder-decoder network which needs to translate from projective first person views to ego-centric bird’s eye views. It thus needs to learn projective geometry from data, although projective geometry could be used as structure for the network, given camera calibration, which has been done in other work:
-	Chen et al., 2019
-	Gupta et la., 2017
-	Henriques et al., 2018
And a couple of others.
Several improvements have been made since the NeurIPS submission, some of which I had addressed in my review. The experiments are quite convincing in their comparisons with the state of the art, in particular the generalization performances:
- generalization from Gibson (training) to Matterport (testing)
- generalization from exploration (training) to PointGoal (testing).
A couple of the results have been removed from the NeurIPS submission, unfortunately, I think they should be kept in.
I appreciated the realistic sensor model fitted to real data measured with a Locobot robot, and the ablation studies, which indicated the contributions of the different planner modules and of pose estimation. The role of the short term planner has been made clearer in the new paper.
I found it interesting that the stellar performance at the Habitat AI challenge was removed from the new paper – this method (or at least a preceding version) won the challenge. But I do understand that this choice was motivated by some remarks of the NeurIPS fellow reviewers regarding the simplicity of the PointGoal task of the challenge.
A couple of less positive aspects, and questions:
On the downside, and following the remarks on literature above, I still think that the results should be compared with CMP, the main competitor of this method. 
I think this is the main short coming of the paper, in particular since CMP is able to perform end-to-end training because the planner is differentiable (value iteration networks, NIPS 2016).
The literature w.r.t. to hierarchical planning is very far from exhaustive and lots of work is missing, consisting of recent work 
Embodied Question Answering, Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra, CVPR 2018 
(and several follow up papers)
but also quite classical work like the literature around the options framework, with the following starting point:
R.S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181–211, 1999.
And many other papers.
The figures 1 and 2 have been completely redone, but they are not completely clear. In particular, several intermediate representations/maps/Images are not commented or labeled, they should be annotated with the symbols from the text. 
The role of the sensor output is not clear. Sensors normally provide relative positions … but the text seems to indicate absolute pose. Some details are lacking.
In “… to predict the pose change between the two maps …” it is unclear what is done here. Is this self-supervision?
The authors mention that unexplored area is considered as free space for planning. What consequences did this have in case of unexplored obstacles? I guess the problem was delegated to the local policy, which needed coping with these issues?
The last paragraph before the conclusions briefly mentions experiments and comparisons but without giving any details. This is unfortunate, since there is still space available (the paper length is 8.5 pages).