Unsupervised learning of an efficient short-term memory network
Learning in recurrent neural networks has been a topic fraught with difficulties and problems. We here report substantial progress in the unsupervised learning of recurrent networks that can keep track of an input signal. Specifically, we show how these networks can learn to efficiently represent their present and past inputs, based on local learning rules only. Our results are based on several key insights. First, we develop a local learning rule for the recurrent weights whose main aim is to drive the network into a regime where, on average, feedforward signal inputs are canceled by recurrent inputs. We show that this learning rule minimizes a cost function. Second, we develop a local learning rule for the feedforward weights that, based on networks in which recurrent inputs already predict feedforward inputs, further minimizes the cost. Third, we show how the learning rules can be modified such that the network can directly encode non-whitened inputs. Fourth, we show that these learning rules can also be applied to a network that feeds a time-delayed version of the network output back into itself. As a consequence, the network starts to efficiently represent both its signal inputs and their history. We develop our main theory for linear networks, but then sketch how the learning rules could be transferred to balanced, spiking networks.
This submission describes a novel autoencoder method, that uses unsupervised learning to configure a recurrent network to encode both the current and past states of an input.  I am not a mathematician nor machine learning expert, and thus am not qualified to review the work for technical merit.  However, I have extensive experience in neural network modeling, and thus appreciate both the objective and purported accomplishments:  the ability to train a recurrent network to store input sequences in an efficient manner using non-supervised learning.  The authors describe a mechanism that addresses the problem by breaking it into two stages — autoencoding, and then optimization — that are carried out over different times scales.  The assumptions are reasonable and, of the derivations I could follow, they seemed reasonable.  This approach is clever and, to the best of my knowledge, is novel.  However, I would like to have seen more extensive evaluation of the network performance, including a benchmark comparison with other standard approaches to the problem using supervised learning (e.g., simple recurrent networks; back propagation through time).  While I think such results would strengthen the submission, and its appeal to the audience at NIPS, the results strike me as being of sufficient interest and potential importance to be accepted for presentation.