Pre-training of Recurrent Neural Networks via Linear Autoencoders
We propose a pre-training technique for recurrent neural networks based on linear autoencoder networks for sequences, i.e. linear dynamical systems modelling the target sequences. We start by giving a closed form solution for the definition of the optimal weights of a linear autoencoder given a training set of sequences. This solution, however, is computationally very demanding, so we suggest a procedure to get an approximate solution for a given number of hidden units. The weights obtained for the linear autoencoder are then used as initial weights for the inputto-hidden connections of a recurrent neural network, which is then trained on the desired task. Using four well known datasets of sequences of polyphonic music, we show that the proposed pre-training approach is highly effective, since it allows to largely improve the state of the art results on all the considered datasets.
Quality and originality: The paper is well written. The introduction motivates the general problem. Section 2 is very easy to read, providing both explanation and necessary linear algebra.  and does a good job of supporting the proposed model. I see two important issues in the paper. First is the motivation of the importance of pre-training. The authors are convincing here. The second is whether they authors’ particular version of pre-training for RNNs is the right one. Rather they show that they can outperform a particular RNN approach which uses no pre-training. Here it’s a bit more confusing, as they compare only to the results from a single paper on music data.  Furthermore there’s some confusion as to how much better the proposed model works compared to the one from Sperduti in 2007 (the “conceptual framework” extended here by the authors).  Does the new model actually work better than the Sperduti one on music data? Or is it just that the theoretical approach is reworked? The authors address this directly in the closing paragraph of Section 1, but I wish they had spent more time pulling apart their contributions from those of previous work on pre-training. Furthermore, I wish they had at least implemented Sperduti’s method in the experiments section.Clarity and Significance: It’s not clear why the authors chose to use music datasets. No analyses of the performance of the musical performance of the models is given, nor do we get any musical examples.  This adds some confusion to the paper: is this about sequence prediction in general or about music sequence prediction? I presume it’s about sequence prediction in general, but would have appreciated either some non-music benchmarks or some more exploration of the music space. Of course space is an issue. However I would have preferred that the lemmas and some of the algorithmic details be moved out to supporting material, with more time spent exploring how pre-training does anything other than raise accuracy.