QMDP-Net: Deep Learning for Planning under Partial Observability
This paper introduces the QMDP-net, a neural network architecture for planning under partial observability. The QMDP-net combines the strengths of model-free learning and model-based planning. It is a recurrent policy network, but it represents a policy for a parameterized set of tasks by connecting a model with a planning algorithm that solves the model, thus embedding the solution structure of planning in a network learning architecture. The QMDP-net is fully differentiable and allows for end-to-end training. We train a QMDPnet on different tasks so that it can generalize to new ones in the parameterized task set and "transfer" to other similar tasks beyond the set. In preliminary experiments, QMDP-net showed strong performance on several robotic tasks in simulation. Interestingly, while QMDP-net encodes the QMDP algorithm, it sometimes outperforms the QMDP algorithm in the experiments, as a result of end-to-end learning.
The paper proposes a novel policy network architecture for partially observable environments. The network includes a filtering module and a planning module. The filtering module mimics computation of the current belief of the agent given its previous belief, the last action and the last observation. The model of the environment and the observation function are replaced with trainable neural modules. The planning module runs value iteration for an MDP, whose transition function and reward function are also trainable neural modules. The filtering model and the planning module are stacked to resemble the QMDP algorithm for solving POMDP. All the modules are trained jointly according to an imitation learning criterion, i.e. the whole network is trained to predict the next action of an expert given the past observations and actions. The experimental evaluation shows better generalization to new tasks.
The approach can be as an extension of Value Iteration Networks (VIN) to partially observable environments by means of adding the filtering network. The paper is in general well written. I did not understand though why the internal model of the network does not use the same observation and action spaces as the real model of the world. Sentences like "While the belief image is defined in M, action inputs are defined in \hat{M}" were therefore very hard to understand. The choice of notation f^t_T is quite confusing. 
The experimental evaluation seems correct. I would suggest a convolutional LSTM as another baseline for the grid world. A fully-connected LSTM is obviously worse eqipped than the proposed model to handle their local structure. I am somewhat suspicious if the fact that expert trajectories came from QMDP could give QMDP-net an unfair advantage, would be good if the paper clearly argued why it is not the case. 
One limitation is this line of research (including VIN) is that it assumes discrete state spaces and the amount of computation is proportional to the number of states. As far as I can I see this makes this approach not applicable to real-world problems with high-dimensional state spaces.
Overall, the paper proposes a novel and elegant approach to build policy networks and validates it experimentaly. I recommend to accept the paper.