Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
Recurrent Neural Networks (RNNs) have become the state-of-the-art choice for extracting patterns from temporal sequences. However, current RNN models are ill-suited to process irregularly sampled data triggered by events generated in continuous time by sensors or other neurons. Such data can occur, for example, when the input comes from novel event-driven artificial sensors that generate sparse, asynchronous streams of events or from multiple conventional sensors with different update intervals. In this work, we introduce the Phased LSTM model, which extends the LSTM unit by adding a new time gate. This gate is controlled by a parametrized oscillation with a frequency range that produces updates of the memory cell only during a small percentage of the cycle. Even with the sparse updates imposed by the oscillation, the Phased LSTM network achieves faster convergence than regular LSTMs on tasks which require learning of long sequences. The model naturally integrates inputs from sensors of arbitrary sampling rates, thereby opening new areas of investigation for processing asynchronous sensory events that carry timing information. It also greatly improves the performance of LSTMs in standard RNN applications, and does so with an order-of-magnitude fewer computes at runtime.
LSTMs and related models (e.g GRUs) have become the workhorses of neural network research and applications involving sequences. The key idea of LSTMs is to use gating over the memory cells to mitigate the "vanishing gradients" problem that occurs in standard recurrent networks, allowing the network to learn tasks that require persisting memories over many timesteps.  This work proposes an extension to LSTMs, adding "time gates" which control updates to the memory. These gates are controlled by oscillators, which only allow updates to the memory during certain points in the oscillation. The periodicity and phase-shift of the gates are learned during training. One additional property these time gates have, is that as they are controlled by a continuous-time oscillator, the value of the gates can be sampled at irregular time intervals.  These phased LSTMs are tested in 4 experiments: classifying sinusoidal signals (a task tailored to the properties of this model), adding task (similar to the adding task in the original LSTM paper), event-based encoding of MNIST and a lip-reading dataset. In all of these tasks the phased LSTMs outperform the original LSTM.
LSTMs (and GRUs) are increasingly used as basic building blocks in neural network architectures, both in inherently sequential problems but also in other applications as many other problems can usefully be decomposed into sequential problems using mechanisms such as attention. Despite being devised some time ago, LSTMs have proved to be difficult to beat as a general purpose tools for modeling sequential structures (e.g. [1]).  This paper presents an interesting idea for improving the performance of LSTMs, particularly on tasks which contain cyclical structure. It is novel and explains the model and motivations well. There are aspect of the analysis and experimental results which could be improved on, but it is a novel approach that will be of interest to the field. I have several suggestions for improvements below, but these do not significantly detract from the work, which is of a high standard.  One aspect that should be clarified is exactly how time-sampling was performed in the cases where sampling was "asynchronous." It's clear that k_t can be computed for any time t, but the remainder of the LSTM still appears to be a discrete-time update (for example, two updates in quick succession when k_t ~ 1 will affect c_t differently than only 1 update is sampled during the k_t~1 phase).  It would be very interesting if the authors could include an any analysis or insight into the solution their model is learning on simple tasks such as adding or classifying sinusoids.  The manuscript mentions the potential for reduced computational load, particularly for high-frequency signals, due to the gating alleviating the need to compute update to the cell when k_t = 0. It would be ideal if this was supported by empirical analysis in one of the experiments at test time (where alpha = 0).  Although realized in a different way, these ideas bear some resemblance to reservoir computing. In particular, both maintain memories using fixed oscillators while learning the connections into/out of the oscillators. It would be helpful to cite some of this work and mention this connection/distinctions.  The choice of experimental comparisons is somewhat eclectic. It certainly makes sense to compare with one of the original LSTM tasks (adding). It would help to assess the potentially impact of this work if it were used on a larger, standard sequential benchmark that doesn't necessarily contain obvious oscillatory structure. For example, a language modeling or machine translation task. would improve the model substantially to compare with a more commonly used sequential benchmark (e.g. [3]).  One aspect of the model which the authors mention is the tendency to preserve memories over many timesteps due the gating effect of the time gates (particularly if tau is large). One might hope to obtain something similar in a standard LSTM by heavily biasing some of the forget gates to 1 and the input gates to 0. It would be a useful comparison to check that simple initialization tricks can't reproduce the results.  [1] LSTM: A Search Space Odyssey http://arxiv.org/pdf/1503.04069v1.pdf [2] A clockwork RNN. http://arxiv.org/abs/1402.3511 [3] Recurrent neural network regularization http://arxiv.org/pdf/1409.2329v5.pdf   ** After feedback ** Due to the large number of reviews, there wasn't space to address my comments in depth. Overall, my ratings are unchanged (they were already high), but I agree with the bulk of the other reviewers this is a good paper.