LEARNING EXPLORATION POLICIES FOR NAVIGATION
Numerous past works have tackled the problem of task-driven navigation. But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention. In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards. We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms. We find that use of policies with spatial memory that are bootstrapped with imitation learning and finally finetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments. We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques. Finally, we also show how such task-agnostic exploration can be used for down-stream tasks. Videos are available at: https://sites.google.com/view/exploration-for-nav/.
This paper proposes a method for learning how to explore environments. The paper mentions that the “exploration task” that is defined in this paper can be used for improving the well-known navigation tasks. For solving this task, a reward function a network architecture that uses RGBD images + reconstructed map + imitation learning + PPO is designed.
<<Pros>>
-The paper is well-written (except for a few typos).
-The overall approach is simple and does not have much complications. 
-The underling idea and motivation is clearly narrated in the intro and abstract and the paper has a easy-to-understand flow.  
<<Cons>>
**The technical novelty is not significant**
-This paper does not provide significant technical novelty. It is a combination of known prior methods: imitation learning + ppo (prior RL work). The presented exploration task is not properly justified as to how it could be useful for the navigation task. The reconstruction of maps for solving the navigation problem is a well-explored problem in prior SLAM and 3D reconstruction methods. Overall the novelty of the approach and the proposed problem is incremental. 
**The paper has major short comings in the experimental section. The presented experiments do not support the main claim of the paper which is improving the performance in the well-known navigation task. Major baselines are missing. Also, the provided results are not convincing in doing the right comparison with the baselines. **
-Experimental details are missing. The major experimental evaluations (Fig. 2 and Fig. 3) are based on the m^2 coverage after k steps and the plots are cut at 1000 steps. What are the statistical properties of the 3D houses used for training and testing? E.g what is their area in m^2? How big is each step in meters?  Why are the graphs cut at 1000 steps? How would different methods converge after more than 1000 steps, e.g. 2000 steps? I would like to see how would the different methods converge after larger number of steps? How long would each step take in terms of time? How could these numbers convey the significance of the proposed method in a real would problem settings? 
-The experiments do not convey if learning has significantly resulted in improved exploration. Consider a simple baseline that follows a similar approach as explained in the paper for constructing the occupancy map using the depth sensor. A non-learning agent could use this map at each step to make a greedy choice about its next action which greedily maximizes the coverage gain based on its current belief of the map. While the performance of random policy is shown in Fig.2 the performance of this greedy baseline is a better representative of the lower bound of the performance on the proposed task and problem setup.
-What is the performance of a learning-based method that only performs collision avoidance? Collision avoidance methods tend to implicitly learn to do a good map coverage. This simple baseline can show a tangible lower bound of a learning-based approach that does not rely on map.
-The major promise of the paper is that the proposed exploration task can improve navigation. However, the navigation experiment does not compare the proposed method with any of prior works in navigation. There is a huge list of prior methods for navigation some of which are cited in the “learning for Navigation” section of the related works and the comparison provided in Fig. 4 is incomplete compared to the state-of-the-arts in navigation. For example, while the curiosity driven approach is compared for the exploration, the more related curiosity based navigation method which uses both “exploration strategy” and “imitation learning” : “Pathak, Deepak, et al. "Zero-shot visual imitation." International Conference on Learning Representations. 2018.
“ is missed in navigation comparison. The aforementioned paper is also missed in the references.  
-Algorithmic-wise, it would make the argument of the paper clearer if results were conducted by running different exploration strategies for navigation to see if running RL with a good exploration strategy could solve the exploration challenge of the navigation problem without needing an explicit exploration stage (similar to the proposed method) which first explores and constructs the map and then does navigation by planning.
-The navigation problem as explained in section is solved based on planning approach that uses a reconstructed map. This is a fairly conventional approach that SLAM based methods use. Therefore, comparison with a SLAM method that constructs the map and then does navigation would be necessary. 
** Technical details are missing or not explained clearly**
- Section 3.1 does not clearly explain the map construction. It seems that the constructed map is just a 2D reconstruction of the space (and not 3D) using the depth sensor which does not need transformation of the 3D point cloud. What is the exact 3D transformation that you have done using the intrinsic camera parameters? This section mentions that there can be error in such map reconstruction because of robot noise but alignment is not needed because the proposed learning method provides robustness against miss-alignment. How is this justified? Why not using the known loop closure techniques in SLAM? 
-The technical details about the incorporated imitation learning method are missing. What imitation learning method is used? How is the policy trained during the imitation learning phase? 
-Last paragraph of intro mentions that the proposed method uses 3D information efficiently for doing exploration. The point of this sentence is unclear. What 3D information is used efficiently in the paper? Isn’t it only 2.5D (information obtained by depth sensor) used in the proposed method?
**Presentation can be improved**
-The left and right plots of the Figure 3 contains lots of repetitions which brings in confusion in comparing the performance of runs with different settings. These two plots should be presented in a single plot. 
- Interpretation of “green vs white vs black” in the reconstructed maps is left to the reader in Fig. 1. 
- Last line in page 5: there is no need for reiteration. It is already clear.
**Missing references**
-Since the paper is about learning to explore, discussion about “exploration techniques in RL” is recommended to be added in at least the related work section. 
-A big list papers for 3D map reconstruction is missing. Since the proposed method relies on a map reconstruction, those papers are relevant to this work and can potentially be used for comparison (as explained above). It is highly recommended that relevant prior 3D map reconstruction papers be added to the related work sections.