Recent research developing neural network architectures with external memory have often used the benchmark bAbI question and answering dataset which provides a challenging number of tasks requiring reasoning. Here we employed a classic associative inference task from the memory-based reasoning neuroscience literature in order to more carefully probe the reasoning capacity of existing memoryaugmented architectures. This task is thought to capture the essence of reasoningthe appreciation of distant relationships among elements distributed across multiple facts or memories. Surprisingly, we found that current architectures struggle to reason over long distance associations. Similar results were obtained on a more complex task involving finding the shortest path between nodes in a path. We therefore developed MEMO, an architecture endowed with the capacity to reason over longer distances. This was accomplished with the addition of two novel components. First, it introduces a separation between memories/facts stored in external memory and the items that comprise these facts in external memory. Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of 'memory hops' before the answer is produced. MEMO is capable of solving our novel reasoning tasks, as well as match state of the art results in bAbI.
Summary:
This paper presents a new task (paired associate inference), drawn from cognitive psychology, which requires linking many pieces of information together to make inferences with long range dependencies. Experimental results show that standard memory architectures fail on these tasks. To redress this, the paper proposes a new memory architecture with several new features that allow for much better performance on the paired associate task. Finally, the paper undertakes systematic experiments on more traditional domains like shortest path problems, showing that the new architecture achieves modest improvements.
Major comments:
Overall this is an interesting and useful work which uses a task from cognitive psychology to illuminate reasoning limitations in prior memory augmented neural networks. It then proposes several architectural and algorithmic fixes to improve performance. The resulting solution appears to work based on the experimental evaluation, and although this does not lead to substantial improvements in the state of the art on the bAbI dataset, it seems likely that these improvements could be useful for even harder task settings. 
The new algorithmic improvements are justified mainly by their performance in experiments, which is completely acceptable if the experiments have been done to a high standard. Nevertheless, it would be wonderful if more insight could be gained about why exactly this configuration of architectural changes was selected and what they contribute. The ablation experiments go some way to addressing this question and perhaps the paper would benefit from a greater discussion of these results in the main text.
The experiments appear to have been done to a high standard, and it is promising that the method’s advantages are more pronounced as the task gets harder (eg by demanding longer chains of inference).
The paper could also be improved by more clearly describing how hyper parameters were selected for each experiment (including the different ablation studies). Why were hyperparamters sometimes chosen based on training loss and other times chosen based on validation loss? In general it would be useful to explain the selection procedure very carefully for each result, since it appears to have differed between evaluations.
The paper is reasonably easy to follow, but could be streamlined to enable more discussion of material that is currently in the appendix.
Typos:
ENM->EMN on pg2