Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
Neural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks. These models appear promising for applications such as language modeling and machine translation. However, they scale poorly in both space and time as the amount of memory grows -limiting their applicability to real-world domains. Here, we present an end-to-end differentiable memory access scheme, which we call Sparse Access Memory (SAM), that retains the representational power of the original approaches whilst training efficiently with very large memories. We show that SAM achieves asymptotic lower bounds in space and time complexity, and find that an implementation runs 1,000⇥ faster and with 3,000⇥ less physical memory than non-sparse models. SAM learns with comparable data efficiency to existing models on a range of synthetic tasks and one-shot Omniglot character recognition, and can scale to tasks requiring 100,000s of time steps and memories. As well, we show how our approach can be adapted for models that maintain temporal associations between memories, as with the recently introduced Differentiable Neural Computer.
The authors use fast approximate-nearest-neighbors structures and arg-top-K operations for computing the attention weights in an Neural Turing Machine.  They describe experiments showing that this does not degrade training performance, and in fact allows training on larger problems.
The novelty in this work is thin, see e.g. section 3.3. in the memory networks paper http://arxiv.org/pdf/1410.3916.pdf; there are now several memory network papers that use memories with millions of items and hashing methods to make the lookups scalable.    On the other hand,  I have not seen such tools used for writeable memories, and even if there is not a conceptual leap in this paper, certainly the practical aspects of making it work are worth reporting.   Nevertheless, I do recommend accepting the paper, but would like to see some changes.  1: I think it would be quite difficult to replicate the results of this paper from the descriptions in the paper.  Especially the descriptions of the tasks are lacking; considering the code for the tasks from the original NTM paper have not been released, and there is no "standard" version of the tasks, it is crucial that the paper give careful descriptions of the construction of the tasks.   I would also ask that the authors  commit to releasing the code for their experiments.     2: Where will this fail? does it always work? The authors often discuss their model as being smooth, but note that the argmax (or K-argmax) operation is *not* smooth.   With $K=1$, in the read only setting, their models are much like the MemNN-WSH in  in http://arxiv.org/pdf/1503.08895.pdf (although that model does not use hashing), and that model was reported to work less well than the fully smooth model.  Is it because of the $K=1$?  Because of the task?  I think this paper would be more valuable if there was some analysis of failure cases and some discussion of what kinds of tasks/setups allow their training to work: their model is making a discrete action, and they train without taking into consideration (as opposed to http://arxiv.org/abs/1511.07275).  I think that they get results in this way is a good contribution, but it would be better to understand what are the limits.  Also, I would suggest that the authors remove section 3.6, and I would also suggest the authors not call the omniglot data "non-synthetic" or "real-world".