Modeling Deep Temporal Dependencies with Recurrent "Grammar Cells"
We propose modeling time series by representing the transformations that take a frame at time t to a frame at time t+1. To this end we show how a bi-linear model of transformations, such as a gated autoencoder, can be turned into a recurrent network, by training it to predict future frames from the current one and the inferred transformation using backprop-through-time. We also show how stacking multiple layers of gating units in a recurrent pyramid makes it possible to represent the "syntax" of complicated time series, and that it can outperform standard recurrent neural networks in terms of prediction accuracy on a variety of tasks.
Review of submission 1062:Modeling sequences with a predictive gating networkSummary: The authors use BPTT to predict future observations from previous observations, and build a stack of such predictors. Applications: sine waves, rotating 3D objects. The approach outperforms certain traditional RNNs. Comments:An interesting approach, but relations to similar previous work are missing:1. Submission: "We show that we can build a deep model by stacking multiple layers of the transformation model, so that higher layers capture higher-oder transformations, in other words, transformations between transformations."The first very similar hierarchy of RNN predictors was described in 1992 [17]. Each RNN predicts its next input. Each is trained by BPTT. Higher-level RNNs see only inputs that lower levels could not predict. The approach in the submission seems very similar, although it is not quite the same - what exactly are the differences?[17] J. Schmidhuber. Learning complex, extended sequences using the principle of history compression, Neural Computation, 4(2):234-242, 1992One difference seems to be: "after extracting the transformation from two frames in the time series, the model predicts the next frame(s) by applying the transformation passed down by a higher layer (in lower layers of the model), or by assuming constancy of the transformation through time (in the top-most layer)."That is, unlike the apparently more general 1992 model, only 2 frames are used to predict the third?  Perhaps that's why the authors are drawing this analogy: "One may think of the bilinear model as performing a first-order Taylor approximation of the input sequence, where the hidden representation models the partial first-order derivatives of the inputs with respect to time." Another difference seems to be that the 1992 hierarchy compresses the observation sequence in unsupervised fashion, while the system of the authors does not. "More importantly, we use predictive training using backprop through time for our model, and show that this is crucial for achieving good prediction performance." This, however, also applies to the deep RNN hierarchy [17].The relation to the previous work should be made very clear!2. Submission says: "Multiplicative interactions (bilinear models) were recently shown to be useful in recurrent neural networks by [4]."In fact, ref [8] of the authors used multiplicative interactions for RNNs much earlier, with great success. 3. Experiments:Chirp signal: "The performances of the DPGM were compared with that of a standard RNN (trained with BPTT) and a CRBM (trained with contrastive divergence) [12] on a dataset containing chirps (sinusoidal waves that increase or decrease in frequency over time)."It is well known that standard RNNs trained by BPTT don't do well on (superimposed) sine waves, e.g., [18]. But there are RNN training algorithms where the connections to output units are trained by the pseudoinverse, and connections to hidden units are evolved to minimize the residual error. This works very well on (even superimposed) sine waves [18]. To compare to the state of the art, one should test against such systems.[18] J. Schmidhuber, D. Wierstra, M. Gagliolo, F. Gomez. Training Recurrent Networks by Evolino. Neural Computation, 19(3): 757-779, 2007The other experiments are interesting.4. Discussion"A major long-standing problem in sequence modeling is how to deal with long range correlations." Here one should cite the work [19] that described and analyzed this problem first:[19] S. Hochreiter. Diploma thesis, TUM, 1991.5. General comment: Interesting and publishable, but only provided the missing relations to similar previous work are made very clear.