We present Value Propagation (VProp), a parameter-efficient differentiable planning module built on Value Iteration which can successfully be trained in a reinforcement learning fashion to solve unseen tasks, has the capability to generalize to larger map sizes, and can learn to navigate in dynamic environments. We evaluate on configurations of MazeBase grid-worlds, with randomly generated environments of several different sizes. Furthermore, we show that the module enables to learn to plan when the environment also includes stochastic elements, providing a cost-efficient learning system to build low-level size-invariant planners for a variety of interactive navigation problems.
ORIGINALITY & SIGNIFICANCE
The authors build upon value iteration networks: the idea that the value function can be computed efficiently from rewards and transitions using a dedicated convolutional network. The authors point out that the original "value iteration network” (Tamar 2016) did not handle non-stationary dynamics models or variable size problems well and propose a new formulation to extend the model to this case which they call a value propagation network.  It seems useful and practical to compute value iteration explicitly as this will propagate values for us without having to learn the propagated form through extensive gradient update steps. Extending to the scenario of non-stationary dynamics is important to make the idea applicable to common problems. The work is therefore original and significant.
The algorithm is evaluated on the original obstacle grids from Tamar 2016 and larger grids generated to test scalability. The authors Prop and MVProp are able to solve the grids with much higher reliability at the end of training and converge much faster.  The M in MVProp in particular seems to be very useful in scaling up to the large grids. The authors also show that the algorithm handles non-stationary dynamics in an avalanche task where obstacles can fall over time.
QUALITY
The symbol d_{rew} is never defined — what does “new” stand for? It appears to be the number of latent convolutional filters or channels generated by the state embedding network. 
Section 2.2 Sentence 2: The final layer representing the encoding is given as ( R^{d_rew  x d_x x d_y }.
Based on the description  in the first paragraph of section 2, it sounds like d_rew might be the number of channels or filters in the last convolutional layer. 
In equation 1, it wasn’t obvious to me that the expression max_a q_{ij}^{k-1} q^{k} corresponds to an actual operation?
The h( \Phi( x ), v^{k-1} ) sort of makes sense …  value is only calculated with respect to only the observation of the maze obstacles but the policy \pi is calculated with respect to the joint  observation and agent state. 
The expression 
   h_{aid} ( \phi(0), v )   =   <  Wa,   [ \phi(o) ; v ]   >   +   b
makes sense and reminds me of the Value Iteration network work where we take the previous value function, combine it with the reward function and use convolution to compute the expectation (the weights Wa encode the effect of transitions). I gather the tensor Wa = R^{|A| x (d_{rew} x d_x x d_y } both converts the feature embedding \phi{o} to rewards and represents the transition / propagation of reward across states due to transitions and discounts at the same time? 
I didn’t understand the r^in, r&out representation in section 4.1. These are given by the domain?
I did get the overall idea of efficiently creating a local value function in the neighborhood of the current state and passing this to the policy so that it can make a local decision.
A bit more detail defining terms, explaining their intuitive role and how the output of one module feeds into the next would be helpful.
POST REVISION COMMENTS:
- I didn't reread the whole thing -  just used the diff tool.  
- It looks like the typos in the equations got fixed
- The new phrase "enables to learn to plan" seems pretty awkward