Building deep reinforcement learning agents that can generalize and adapt to unseen environments remains a fundamental challenge for AI. This paper describes progresses on this challenge in the context of man-made environments, which are visually diverse but contain intrinsic semantic regularities. We propose a hybrid model-based and model-free approach, LEArning and Planning with Semantics (LEAPS), consisting of a multi-target sub-policy that acts on visual inputs, and a Bayesian model over semantic structures. When placed in an unseen environment, the agent plans with the semantic model to make high-level decisions, proposes the next sub-target for the sub-policy to execute, and updates the semantic model based on new observations. We perform experiments in visual navigation tasks using House3D, a 3D environment that contains diverse human-designed indoor scenes with real-world objects. LEAPS outperforms strong baselines that do not explicitly plan using the semantic content.
The paper proposes a hybrid model-free and model-based RL agent for the task of navigation. Reaching the target is decomposed into a set of sub-goals, and the plan is updated as the agent explores the environment. The method has been tested in the House3D environment for the task of RoomNav, where the goal is to navigate towards a certain room. 
The idea of integrating RL agents with semantic knowledge is interesting. However, the paper has several major issues that should be addressed in the rebuttal:
(1) The experiment results in Figure 3 and Figure 4 are based on groundtruth room information. The only experiment that is fully automatic is the one in Figure 5. However, there is no difference between the proposed method and the baselines in that case. So the proposed method is not effective without groundtruth information.
(2) The only evaluation metric that is used is "Success Rate". That metric is not sufficient for evaluation of navigation agents since it does not include episode length information. All of the results should be based on the protocol mentioned in "On Evaluation of Embodied Navigation Agents", arXiv 2018. 
(3) There is no termination action according to Appendix B. So the agent does not know if it is at the target or not. It seems the agent will stop if it issues "stay still" three times. That is different from termination action. Also, it is confusing what 450 pixels means for a scene classifier that works on the image.
(4) The paper is written in a convoluted way:
   (a) It is not clear if the semantic model is trained along with the RL model end-to-end or not.
   (b) Regarding multi-target sub-policies, is there a separate policy for each pair of intermediate targets? 
   (c) Regarding inference and planning on M, what is \tau exactly? How is the length of the plan determined? 
   (d) Why is the model updated only after a fixed number of steps? That increases the episode length. 
(5) The number of T_i's is manually set to 8. That causes serious generalization issues. How do we know how many T_i's exist in a new environment?
Minor comments:
- The paper mentions "An example of such environments is House3D which contains 45k real-world 3D scenes". House3D includes only synthetic scenes. They should not be called real-world scenes.
- How is the reward shaping done?
****
Final comments after reading the response and the reviews:
Regarding the fairness of the review, success rate is not sufficient to evaluate navigation agents. A random agent can achieve 100% success if it is given enough time. So it is totally fair to ask for a metric (such as SPL) that is a function of both success rate and episode length. 
I am going to increase the rating to 5 since some of my concerns have been addressed. There are still a number of issues:
- The authors did not run the experiments with the termination action. I disagree that this is orthogonal to the focus of the paper. This is not just an additional action. It indicates whether the agent has learned anything or it is just a combination of better obstacle avoidance and luck. The SPL numbers are so low (maximum SPL is 6.19%) so adding the termination action will probably make the method similar to random.
- There is a huge gap between success rate and SPL numbers. For instance, success rate is 66.4, while SPL is 5.84 (note that for some reason the SPL numbers are multiplied by 10 in the table). I doubt that the agent has learned anything meaningful in comparison to the baseline. I understand that the task is hard, but this gap is so huge.
- A separate policy is trained for each sub-target. This doesn't scale. There should be one policy for all targets.