A key feature of intelligent behavior is the ability to learn abstract strategies that transfer to unfamiliar problems. Therefore, we present a novel architecture, based on memory-augmented networks, that is inspired by the von Neumann and Harvard architectures of modern computers. This architecture enables the learning of abstract algorithmic solutions via Evolution Strategies in a reinforcement learning setting. Applied to Sokoban, sliding block puzzle and robotic manipulation tasks, we show that the architecture can learn algorithmic solutions with strong generalization and abstraction: scaling to arbitrary task configurations and complexities, and being independent of both the data representation and the task domain.
This paper introduces a neural controller architecture for learning abstract algorithmic solutions to search and planning problems. By combining abstract and domain-specific components, the model is able to mimic two classical algorithms quite closely across several domains. The precision of the learning is very high; verified by generalization to substantially larger problem sizes and different domains. One notable conclusion is that Evolutionary Strategies is able to learn algorithmic solutions whose precision is on par with deterministic algorithms. The method of triggering learning based on curriculum level performance is a notable feature that nicely couples generalization progress with learning, and yields insightful learning curves.
Although the paper shows that learning abstract algorithmic solutions is possible, it is not clear that the framework is able to learn such solutions when the training signals have not already been broken down into their smallest constituent parts. In other words, it is not clear what this framework could be used for, since it appears the experimenter must already possess a complete specification of the target algorithm and breakdown of the domain. Is there a setting where this framework could be used to learn something the experimenter is not already aware of? Or is the main point that it is technically possible to get an NN to learn this behavior?
Although it is clear that models are achieved that satisfy R1-R3, it is not clear exactly what problem formulation is being considered. It would be very helpful if the paper included a formal problem definition so that the purpose of each framework component and differences w.r.t. prior work are clear. 
Similarly, the motivation for each of the data dependent modules is not clear. Is there something fundamental about this particular decomposition into modules? Or are these just the modules that were necessary given the specifics of the algorithms that were learned in experiments? How does this framework generalize to other kinds of algorithms?
Are the comparison methods (DNC and stackRNN) unable to generalize to larger problem sizes? Including the full comparisons on generalization would give a more complete picture. Similarly, the figures are missing the comparisons for Learning to Plan for DNC and stackRNN.
Is the comparison w.r.t. training time in Figure 3c fair, since the proposed framework pretrains the submodules?
Is there a fundamental problem of DNC being addressed here? E.g., are there some critical types of submodules where making them differentiable is not an option?
Is the algorithm limited to cases where the number of actions at each state is equal? I.e., could it be applied to algorithms like shortest path in the DNC paper?
Finally, the last line talks about intriguing applications to the real world, but the running example in the paper is sorting. Is there some hypothetical but concrete example of how this framework could help in the real world, and do something better than a hard-coded classical algorithm? Or discover a new algorithm?
----------
After the rebuttal and discussion, I am convinced the paper is a useful contribution, and have increased my rating. However, I still think the presentation can be much improved to improve the clarity of the contribution. I would hope to see the following addressed in the final version:
1. More detailed and precise discussion of how the approach relates to prior work. As is, this discussion is informal and scattered throughout the paper. Enumerating the distinctions in one place would make the contribution much more clear.
2. The above would also help make the explicit contribution more clear. E.g., the "Contribution" paragraph currently does not contain the fact that the machine has only been applied to "planning" problems. This should be included to avoid making the contribution seem overly general.
3. More formal description of what each module does. Right now, they are described informally. Actually seeing the equations of what each computes would make it easier to understand how and why they all fit together.
I still see the main contribution as "It is technically possible to train the abstract controller of a neural computer for planning using NES, so that R1-R3 are satisfied." Ideally, the above clarifications would make it more clear that this is the main contribution, or that there are some other key contributions beyond this.