Using Fast Weights to Attend to the Recent Past
Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These "fast weights" can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proved very helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.
This paper presents a method for using a rapidly changing secondary matrix of temporary weight values, referred to as "fast weights", in recurrent neural networks to maintain a trace of past inputs and network states. Presented experiments show that incorporating these fast weights in recurrent neural networks improve performance. RNNs with fast weights are shown to outperform other RNN variants with equivalent numbers of recurrent units on associative retrieval of characters from strings and image recognition tasks (MNIST and facial expression).
The presented method for incorporating fast weights into recurrent neural networks is elegant and produces noticeable improvements in performance in the experiments presented. The paper is very well written, for the most part. A small amount of related prior work exists on fast weights; however, it seems that research on this topic never caught on. This paper will hopefully change this; fast weights should be of interest to the machine learning community. Additionally, the developments and results presented in this paper have a biological inspiration and may inspire work in computational neuroscience and cognitive science.  Some feedback, questions and suggestions arose on reading:  The performance improvements shown in experimental results indicate that work in fast weights has significant promise. It would have been interesting to see some analysis or an example of the low-level outcomes of fast weight operation, such as a resulting fast weight matrix (in preference to the description of a second image recognition task, perhaps). An appendix describing implementation details was mentioned in the paper but was not submitted with the paper for review.  The contents of the Conclusion section might be better described as, and converted into, a Discussion section (an additional Conclusion section might not be necessary). Suggestions and comments on the Conclusion(/Discussion) follow:  1) The authors may disagree and disregard this suggestion; however, rather than outright state that the paper contributes to a field of study, my preference is to refer to the specific developments as "contributions" and then describe how these contributions are significant or relevant to the fields of study. For example, in the case of the paper under review, the contributions of the paper are a method for adapting fast weights in recurrent neural networks and the experimental results demonstrating improved RNN performance. The demonstration of improved performance of RNNs is evidence of the significance of the contributions to machine learning. These contributions are also relevant to computational neuroscience and cognitive science as a model and evidence that fast mechanisms of synaptic plasticity may contribute to sequential processing of stimuli and working memory.  2) Line 288 has the statement "Layer normalization makes this kind of attention work better", which is mentioned earlier but not actually shown in the paper, possibly making the statement unsuitable as a conclusion in the paper. A statement reminding the reader of the use of layer normalization might be more suitable.  3) References are made to "sequence-to-sequence RNNs used in machine translation" (line 290) and "[t]he ability of people to recursively apply the very same knowledge and processing apparatus to a whole sentence and to an embedded clause within that sentence has long been used as an argument against neural networks as a model of higher-level cognitive abilities" (line 294-297). Ideally, both of these references would be accompanied by a citation.     A few suggestions for presentation and typographical corrections follow:  1) The black-and-white printability of the paper could be improved by choosing thicker line weights for figures 1, 3, and 5, and by using different line styles or arrow heads to distinguish signals. Figure 5 could benefit from a larger font size for plots.  2) Line 203: remove "their". Page 8 has a number of instances of the opening quotation mark being a closing quotation mark. Line 287: "unis" was probably meant to be "units". References [8] and [25] have initialisms that should be capitalised.