Value Propagation Networks
We present Value Propagation (VProp), a set of parameter-efficient differentiable planning modules built on Value Iteration which can successfully be trained using reinforcement learning to solve unseen tasks, has the capability to generalize to larger map sizes, and can learn to navigate in dynamic environments. We show that the modules enable learning to plan when the environment also includes stochastic elements, providing a cost-efficient learning system to build low-level size-invariant planners for a variety of interactive navigation problems. We evaluate on static and dynamic configurations of MazeBase grid-worlds, with randomly generated environments of several different sizes, and on a StarCraft navigation scenario, with more complex dynamics, and pixels as input.
Update:
I thank the authors for the response. Unfortunately, the response does not mention modifications made to the paper according to the comments. According to pdfdiff, modifications to the paper are very minor, and none of my comments are addressed in the paper. I think the paper shows good results, but it could very much benefit from improved presentation and evaluation. I do recommend acceptance, but if the authors put more work in improving the paper, it could have a larger impact.
------
The paper proposes a learnable planning model based on value iteration. The proposed methods can be seen as modifications of Value Iteration Networks (VIN), with some improvements aimed at improving sample efficiency and generalization to large environment sizes. The method is validated on gridworld-type environments, as well as on a more complex StarCraft-based domain with raw pixel input.
Pros:
1) The topic of the paper is interesting: combining the advantages of learning and planning seems like a promising direction to achieving adaptive and generalizable systems.
2) The presentation is quite good, although some details are missing.
3) The proposed method can be effectively trained with reinforcement learning and generalizes well to much larger environments than trained on. It beats vanilla VIN by a large margin. The MVProp variant of the method is especially successful.
Cons:
1) I would like to see a more complete discussion of the MVProp method. Propagation of only positive rewards seems like somewhat of a hack. Is this a general solution or is it only applicable to gridworld navigation-type tasks? Why? If not, is the area of applicability of MVProp different from VProp? Also, is the area of applicability of VProp different from VIN? It’s important to discuss this in detail.
2) I wonder how would the method behave in more realistic gridworld environments, for instance similar in layout to those used in RL navigation literature (DMLab, ViZDoom, MINOS, etc). The presented environments are quite artificial and seem to basically only require “obstacle avoidance”, not so much deliberate long-distance planning.
3) Some details are missing. For instance, I was not able to find the exact network architectures used in different tasks. 
Related to this, I was confused by the phrase “As these new environments are not static, the agent needs to re-plan at every step, forcing us to train on 8x8 maps to reduce the time spent rolling-out the recurrent modules.” I might be misunderstanding something, but is there any recurrent network in VProp? Isn’t it just predicting the parameters once and then rolling our value iteration forward without any learning? Is this so time-consuming?
4) Why does the performance even of the best method not reach 100% even in the simpler environments in Figure 2? Why is the performance plateauing far from 100% in the more difficult case? It would be interesting to see more analysis of how the method works, when it fails, and which parts still need improvement. On a related topic, it would be good to see more qualitative results both in MazeBaze and StarCraft - in the form of images or videos.
5) Novelty is somewhat limited: the method is conceptually similar to VIN. 
To conclude, I think the paper is interesting and the proposed method seems to perform well in the tested environments. I am quite positive about the paper, and I will gladly raise the rating if my questions are addressed satisfactorily.