summaries: The work takes inspiration from a recent work on phased LSTM, and proposes to add a Gaussian gate based on time to LSTM cells. With this additional gate, the network can skip updating the states by closing the time-gate, as a result enabling longer memory persistence, and better gradient flow. The authors also propose to add a budget term to force the time-gate to be closed most of the time as a way to save compute. Empirical results suggest the Gaussian-gated LSTMs perform better than regular LSTMs on tasks with long temporal dependencies. The authors also propose to use a curriculum training schedule in which the variance of the gaussian gates is continuously increased to speed up training of LSTMS.   This paper focuses on the reduction of training time by various mechanisms. By introducing a time gate during training, it controls when a neuron (weights?) can be updated during training. By introducing and additional budget term in the loss function, training costs (number of computations) are reduced by one order of magnitude. 
abstractText: Recurrent neural networks can be difficult to train on long sequence data due to the well-known vanishing gradient problem. Some architectures incorporate methods to reduce RNN state updates, thereby allowing the network to preserve memory over long temporal intervals. We propose a timing-gated LSTM RNN model, called the Gaussian-gated LSTM (g-LSTM) for reducing state updates. The time gate controls when a neuron can be updated during training, enabling longer memory persistence and better error-gradient flow. This model captures long temporal dependencies better than an LSTM on very long sequence tasks and the time gate parameters can be learned even from a non-optimal initialization. Because the time gate limits the updates of the neuron state, the number of computes needed for the network update is also reduced. By adding a computational budget term to the training loss, we obtain a network which further reduces the number of computes by at least 10×. Finally, we propose a temporal curriculum learning schedule for the g-LSTM that helps speed up the convergence time of the equivalent LSTM on long sequences.
references.authors: Victor Campos Brendan Jou Xavier Giro i Nieto Jordi Torres Shih-Fu Chang
references.title: Skip RNN: learning to skip state updates in recurrent neural networks
references.venue: CoRR, abs/1708.06834,
references.authors: Shiyu Chang Yang Zhang Wei Han Mo Yu Xiaoxiao Guo Wei Tan Xiaodong Cui Michael J. Witbrock Mark Hasegawa-Johnson Thomas S. Huang
references.title: Dilated recurrent neural networks
references.venue: CoRR, abs/1710.02224,
references.authors: Kyunghyun Cho Bart van Merriënboer Çalar Gülçehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio
references.title: Learning phrase representations using RNN Encoder–Decoder for statistical machine translation
references.venue: In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Tim Cooijmans Nicolas Ballas Cesar Laurent Aaron C. Courville
references.title: Recurrent batch normalization
references.venue: CoRR, abs/1603.09025,
references.authors: Xavier Glorot Yoshua Bengio
references.title: Understanding the difficulty of training deep feedforward neural networks
references.venue: In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,
references.authors: Sepp Hochreiter Jürgen Schmidhuber
references.title: Long short-term memory
references.venue: Neural Comput.,
references.authors: Rafal Jozefowicz Wojciech Zaremba Ilya Sutskever
references.title: An empirical exploration of recurrent network architectures
references.venue: In International Conference on Machine Learning,
references.authors: Diederik P. Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: CoRR, abs/1412.6980,
references.authors: Alex Krizhevsky Vinod Nair Geoffrey Hinton
references.title: The CIFAR-10 dataset
references.venue: online: http://www. cs. toronto. edu/kriz/cifar. html,
references.authors: David Krueger Tegan Maharaj János Kramár Mohammad Pezeshki Nicolas Ballas Nan Rosemary Ke Anirudh Goyal Yoshua Bengio Aaron Courville Chris Pal
references.title: Zoneout: Regularizing RNNs by randomly preserving hidden activations
references.venue: arXiv preprint arXiv:1606.01305,
references.authors: Yann LeCun Leon Bottou Yoshua Bengio Patrick Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE,
references.authors: Shuai Li Wanqing Li Chris Cook Ce Zhu Yanbo Gao
references.title: Independently recurrent neural network (IndRNN): Building a longer and deeper RNN
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Zachary Chase Lipton
references.title: A critical review of recurrent neural networks for sequence learning
references.venue: CoRR, abs/1506.00019,
references.authors: Daniel Neil Michael Pfeiffer Shih-Chii Liu
references.title: Phased LSTM: Accelerating recurrent network training for long or event-based sequences
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jürgen Schmidhuber
references.title: Deep learning in neural networks: An overview
references.venue: Neural Networks,
references.authors: Ohad Shamir
references.title: Distribution-specific hardness of learning neural networks
references.venue: CoRR, abs/1609.01037,
references.authors: Corentin Tallec Yann Ollivier
references.title: Can recurrent neural networks warp time
references.venue: In Proceedings of International Conference on Learning Representation,
references.authors: Tijmen Tieleman Geoffrey Hinton
references.title: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning
references.authors: Trieu H. Trinh Andrew M. Dai Thang Luong Quoc V. Le
references.title: Learning longer-term dependencies in RNNs with auxiliary losses
references.venue: CoRR, abs/1803.00144,
references.authors: Li Wan Matthew Zeiler Sixin Zhang Yann Le Cun Rob Fergus
references.title: Regularization of neural networks using DropConnect
references.venue: In International Conference on Machine Learning,
references.authors: Adams Wei Yu Hongrae Lee Quoc V. Le
references.title: Learning to skim
references.venue: text. CoRR,
sections.text: Recurrent neural networks can be difficult to train on long sequence data due to the well-known vanishing gradient problem. Some architectures incorporate methods to reduce RNN state updates, thereby allowing the network to preserve memory over long temporal intervals. We propose a timing-gated LSTM RNN model, called the Gaussian-gated LSTM (g-LSTM) for reducing state updates. The time gate controls when a neuron can be updated during training, enabling longer memory persistence and better error-gradient flow. This model captures long temporal dependencies better than an LSTM on very long sequence tasks and the time gate parameters can be learned even from a non-optimal initialization. Because the time gate limits the updates of the neuron state, the number of computes needed for the network update is also reduced. By adding a computational budget term to the training loss, we obtain a network which further reduces the number of computes by at least 10×. Finally, we propose a temporal curriculum learning schedule for the g-LSTM that helps speed up the convergence time of the equivalent LSTM on long sequences.
sections.heading: 1 INTRODUCTION
sections.text: Numerous methods and architectures have been proposed to mitigate the vanishing gradient problem in RNNs, with LSTMs (Hochreiter & Schmidhuber, 1997) as one of the first prominent solutions doing so by including gating structures in the computation. Although the LSTM has excelled at handling many tasks (Schmidhuber, 2015; Lipton, 2015), it still has difficulties in learning complex and long time dependencies (Neil et al., 2016; Chang et al., 2017; Trinh et al., 2018).
  In the last few years, various methods which reduce the state updates of an RNN (LSTM) have been explored to better learn long time dependencies from data. Clockwork RNNs (Koutnik et al., 2014) group the hidden units of the RNN into “modules,” where each module is executed at pre-specified time steps thereby skipping time steps which helps learn longer time dependencies. Recently, various other methods have been proposed which can be characterized by the use of additional “time gates,” kn, that control the information flow from one time step to the next (Krueger et al., 2016; Campos et al., 2017). Phased LSTM (PLSTM) (Neil et al., 2016) learns a parameterized function, kn, from the the time input of the current state and was proven to be successful at learning over long sequences.
  The PLSTM time gate, parameterized by period, phase, and ratio parameters for each hidden unit, is defined through a modulo function with an ill-defined gradient. Furthermore, with periodic functions being hard to learn using gradient-based methods (Shamir, 2016) and with kn being periodic, the PLSTM was unable to learn the time gate parameters and hence relied on careful initialization. In order to offset these difficulties, this work proposes a new LSTM variant called the Gaussian-gated LSTM (g-LSTM). Similar to the PLSTM, it is an LSTM model with a parameterized kn but with only two parameters per hidden unit. Unlike the PLSTM which uses a periodic formulation for kn, the g-LSTM uses a Gaussian function.
  We show in this work that the g-LSTM can provide a number of possible advantages over the LSTM, in particular, on long sequence tasks that pose convergence problems during training:
  • The g-LSTM network can process very long sequences by reducing the time over which the neurons can be updated. It converges faster than the LSTM, especially on sequences that are over 500 steps.
  • The “openness” of the neuron for an update can be adapted according to the task during training, even for extreme, non-optimal initializations of the time gate parameters.
  • By introducing a computational budget term into the loss function during training, the “openness” of the neuron can be optimized for a reduced computational budget. This reduction can be achieved with little or no degradation to the network performance and is useful for network pruning.
  • A “temporal curriculum” training schedule can be set up for the g-LSTM so that it helps to speed up the convergence of a normal LSTM.
  The paper is structured as follows: In section 2, we discuss briefly the related work. Then, in section 3, we present the formulation of the g-LSTM, the datasets used in this work and details about the experimental hyperparameters. In section 4, we present experiments demonstrating the usefulness of the g-LSTM with respect to the four claims listed above. We provide gradient analysis in section 5 to further explain the faster convergence results of the g-LSTM. Finally in section 6, we conclude with a brief discussion of the results.
sections.heading: 2 RELATED WORK
sections.text: There have been a multitude of proposed methods to improve the training of RNNs, especially for long sequences. Apart from incorporating additional gating structures, for example the LSTM and the GRU (Cho et al., 2014), more recently various techniques were proposed to further increase the capabilities of recurrent networks to learn on sequences of length over 1000. Proposed initialization techniques such as the orthogonal initialization of kernel matrices (Cooijmans et al., 2016), chrono initialization of the biases (Tallec & Ollivier, 2018), and diagonal recurrent kernel matrices (e.g. Li et al. (2018)) have demonstrated success. Trinh et al. (2018) propose using truncated backpropagation with an additional auxiliary loss to reconstruct previous events.
  Methods that enable more efficient learning on long temporal sequences use solutions that preserve memory over longer timescales. Such solutions were first explored by Koutnik et al. (2014) in the Clockwork RNN (CW-RNN). This network skips state updates by allowing different neurons to be “activated” on different, modulated clock cycles. More recently proposed models for skipping updates include the Phased LSTM (PLSTM) (Neil et al., 2016) which uses a modulo-periodic timing gate to limit state updates; the Zoneout network (Krueger et al., 2016) which skips state updates in a random manner; and the Skip RNN (Campos et al., 2017) which learns a state skipping scheme from the data to shorten the effective sequence length for the task. Additionally, the LSTM-Jump (Yu et al., 2017) uses a reinforcement learning algorithm to learn when to skip state updates, showing a method to more quickly process (long) sequential data with an RNN while maintaining an accuracy comparable to a baseline LSTM.
  It has been suggested but not yet demonstrated in the literature that the parameters of the CW-RNN clock cycle and PLSTM timing gate could be learned in training. Currently, the implementation of these networks requires a careful initialization of these parameters. With the Gaussian-gated LSTM (g-LSTM) in this work we present a time gated RNN network that converges on long sequence tasks and also has the ability to learn its time gate parameters even when initialized in a nonoptimal way.
sections.heading: 3 METHODS
sections.heading: 3.1 G-LSTM
sections.text: The g-LSTM is an LSTM model with an additional time gate (Fig. 1). This time gate is used to regulate the information flow in time. Equations 1 - 3 describe the update equations for the hidden and cell states of the LSTM. Equations 4 and 5 describe the gating mechanism of the time gate, kn.
  in = σ(xnWxi + hn−1Whi + bi), fn = σ(xnWxf + hn−1Whf + bf ) (1) c̃n = fn cn−1 + in σ(xnWxg + hn−1Whg + bg) (2)
  on = σ(xnWxo + hn−1Who + bo), h̃n = on tanh(c̃n) (3) cn = kn c̃n + (1− kn) cn−1 (4) hn = kn h̃n + (1− kn) hn−1 (5)
  In a standard LSTM, the gating functions in, fn, on, represent the input, forget, and output gates respectively at sequence index n. cn is the cell activation vector, and xn and hn represent the input feature vector and the hidden output vector, respectively. The cell state cn is updated with a fraction of the previous cell state that is controlled by fn, and a new input state created from the element-wise (Hadamard) product, denoted by , of in and the candidate cell activation as in Eq. 2. In the g-LSTM, we further control the cell state and the output hidden state through the kn gate which is independent of the input data and hidden states, and is purely dependent on the time input corresponding to the sequence index n. The use of the Hadamard product ensures that each hidden unit is independently controlled by the corresponding time gate unit, thus enabling the different units in the layer to process the input at different time scales.
  The time gate kn is defined based on a Gaussian function as: kn = e−(tn−µ) 2/σ2where the mean parameter, µ, defines the time when the hidden unit is “open” and the standard deviation, σ, controls the openness of the time gate for each unit around its corresponding µ. The time inputs t = {t1, t2, ..., tn, ..., tN} for the sequence x = {x1, x2, ..., xn, ..., xN} can correspond to the physical notion of time at the respective sequence input. In the absence of a standard notion of time, we use the sequence indices as the time input, i.e. t = {1, 2, ..., n, ..., N}. In this work, we assume this notion of time by default. The “openness” of kn for a neuron is defined by the parameterization of its Gaussian function.
sections.heading: 3.2 BACK PROPAGATION FOR G-LSTM
sections.text: An important characteristic of the g-LSTM is reduced gradient flow in back propagation training methods. By having the gating structure as in Eqs. 4 and 5 there are fewer gradient product terms, which reduces the likelihood of vanishing or exploding gradients. In a gradient descent learning scheme for a given loss function, L, when training the recurrent parameters, Wh (from Eqs. 1 - 3), the gradient as in Eq. 6 is used.
  ∂L
  ∂Wh =
  ∂L
  ∂hN ∂hN ∂Wh
  (6)
  By the chain rule ∂hN∂Wh expands for all time steps of the sequence, n ∈ {1, ..., N}. Because each output state is gated by the time gate, kn, the gradient terms in the expansion of ∂hN∂Wh are scaled by kn. When the time gate is open less often, i.e. with a small σ value, then there are fewer influential gradient terms. More details are given in Appendix A.
sections.heading: 3.3 DATASETS
sections.text: The experiments described in the paper are carried out on the adding task and two standard long sequence datasets: the sequential MNIST and the sequential CIFAR-10 datasets.
  Adding task: In order to test the long sequence learning capability of the g-LSTM, we use the adding task (Hochreiter & Schmidhuber, 1997). In this task, the network is presented with two sequences of length N , x = {x1, ..., xN}, xn ∼ U(0, 1)) and m = {m1, ...,mN},mn ∈ {0, 1},∑Nt=1mn = 2. The sequence m has exactly two values of 1 and the remaining values of the sequence are 0. The indices of the “1” values are chosen at random. For each pair in the sequence (x,m), the associated label value, y, is the sum of the two values in x corresponding to the “1” values of m. The objective of this task is to minimize the mean squared error between the predicted sum from the network, ŷ, and the labeled sum, y. A new training set of 5000 sequence samples is presented in every epoch during training in order to avoid overfitting. The test set consists of a separate 5000 samples. For N > 1000, it is known that LSTMs have difficulty learning the task and hence we focus on values of N > 1000 in this work.
  sMNIST: The sequential MNIST dataset is widely used to analyze the performance of a recurrent model. This dataset consists of 60,000 training samples and 10,000 test samples, each a single vector sequence of length 784 corresponding to the 28 × 28 pixel images in the MNIST dataset (LeCun et al., 1998). We also use permuted MNIST (pMNIST), a permuted variant of the sMNIST dataset where the sequences are processed with a fixed random permutation, making the task harder.
  sCIFAR-10: The sequential CIFAR-10 dataset is another long sequence dataset based on CIFAR10 (Krizhevsky et al., 2014) with 10 classes. The 32 × 32 RGB pixel images are reshaped into sequences of length 1024 with 3 dimensional features corresponding the RGB channels at every time step. Like in the sMNIST dataset, the dataset consists of 60,000 training samples and 10,000 test samples.
sections.heading: 3.4 EXPERIMENTAL HYPERPARAMETERS
sections.text: For the adding task, a mean squared error (MSE) loss was used with the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 10−3. The g-LSTM time gate parameters were trained using a learning rate of 100. For both sMNIST and sCIFAR-10 datasets, the cross-entropy loss function was used along with the RMSProp optimizer (Tieleman & Hinton, 2012) with a learning rate of 10−3. Decay parameters of 0.5 and 0.9 were used for sMNIST and sCIFAR-10, respectively. The bias of the forget gate is initialized to 1 following (Jozefowicz et al., 2015).
sections.heading: 4 RESULTS
sections.text: Section 4.1 presents results that demonstrate the faster convergence properties of the g-LSTM on long sequence tasks. Section 4.2 shows the trainability of the time gate parameters of the g-LSTM even when the parameters are initialized in a non-optimal way. Section 4.3 presents a modified loss function used during training to reduce the number of computes for the network update and Section 4.4 presents a new “temporal curriculum” learning schedule that allows g-LSTMs to help LSTMs converge faster.
sections.heading: 4.1 FAST CONVERGENCE PROPERTIES OF G-LSTM
sections.text: First, we look at the convergence properties of the g-LSTM on the long-sequence adding task, the sMNIST task and the sCIFAR-10 task. Table 1, above, details the network architectures used in the experiments in this section. Similar to the architecture from Trinh et al. (2018), the recurrent layer of the sCIFAR-10 network is followed by two 256 unit fully-connected (FC) layers, where DropConnect (Wan et al. (2013)) (p = 0.5) is applied to the second FC layer. The kernel matrices in the LSTM networks were initialized in an orthogonal manner as described in (Cooijmans et al., 2016).
  The test performances of these networks during the course of the training on different datasets are shown in Fig. 2, while the corresponding final performance metrics at the end of training are shown in Table 1. From Fig. 2, it is evident that the test loss of the g-LSTM decreases faster in training than the LSTM across all datasets. Further experiments show that this trend is maintained with different training optimizers, LSTM initializations including the bias initialization following Tallec & Ollivier (2018), and network sizes as shown in Appendix D.
  Table 2 compares the performance of various networks including the g-LSTM and the baseline LSTM on sMNIST and sCIFAR-10 (from Table 1). The results show that the g-LSTM consistently performs better than the LSTM and has a similar performance to other state-of-the-art networks. Different network sizes were also investigated for the sMNIST task, see Appendix D.
sections.heading: 4.2 TRAINABILITY OF THE TIME GATE PARAMETERS OF G-LSTM
sections.text: To demonstrate that the g-LSTM can be trained even with non-optimal initializations, we look at the performance of the g-LSTM on the adding task with different time gate parameter initializations. We concern ourselves with sequences of length 1000 that are difficult for the LSTM. The time gate parameters are initialized in a way to temporally constrain the network so that it can only process for a short period of time. For example, a network with time gate parameters initialized with µ = 500 and σ = 40 as in Figure 3 (a) can only process a short period of time around the middle of sequence. It follows that the network would be unable to learn with these parameters because in the adding task the input data is distributed equally across the sequence length (T = 1000). Therefore, in order to learn the task from this initialization, the time gate parameters must learn a distribution such that the gates over all hidden units are open across the entirety of the sequence.
  We observe that the time gate parameters do learn, as shown in Figure 3 (b), thereby enabling the network to solve the task. Independent of various time gate initializations, the network reaches an MSE of around 3.9×10−5 at the end of 700 epochs; details of which could be found in Appendix B. The ability of the network to learn the time gate parameters necessary to cover the entire sequence is especially significant because it shows that even with this narrow time window initialization that requires learning of the time gates, the g-LSTM learns the task, whereas the PLSTM does not learn the task as well. An example of this is shown in Appendix C.
sections.heading: 4.3 REDUCTION IN COMPUTATION
sections.text: Although the formulation of the g-LSTM appears to require more computes, it offers substantial speedup as a large proportion of the neurons can be skipped in a timestep at runtime. We can set a threshold on the time gate so that we skip all corresponding computations for time steps where kn is below this threshold. To further reduce the number of operations, it is preferred that the σ of the kn for different neurons should be small but the network performance should not be significantly degraded. To achieve this goal, we included a “computational budget” loss term during the optimization of the gate parameters, µ and σ. The loss equation for updating the kn parameters is given by:
  L = Ldata + λLbudget.
  Similar to the Skip RNN network (Campos et al., 2017), a budget loss term which minimizes the average openness of the time gate over time is applied:
  Lbudget = E[kn] ≈ N∑ n=1 J∑ j=1 k(j)n
  for every neuron j of the g-LSTM. The study was carried out on sMNIST using a network with 110 units, σ initialized to 50, µ initialized uniformly at random between 1 to 784, and a λ value of 1.
  The network’s performance of 2.2% LER was comparable to the network’s performance of 1.3% when no additional budget constraint was imposed. The final σ range for the budgeted g-LSTM is much smaller compared to that of the g-LSTM as shown in Fig. 4. There is only a slight increase in LER for the budgeted g-LSTM versus the g-LSTM (see Table 2), even though there is a significant decrease in the average time gate openness across all hidden units.
  In order to reduce the number of computes, we set a threshold, vT for kn so that the update steps are carried out only if kn > vT , if kn < vT the previous neuron state can be copied over to the current state. By increasing vT , the number of computes decreases as shown in Fig. 5. In the case of vT = 0.01, only 8.2% of the time gates are open on average across all hidden units and all time steps. Furthermore, the LER increased only slightly to 2.3% from 2.2%.
  We give a quantitative estimate for the number of operations (Ops) corresponding to the number of update equations for a g-LSTM. In the estimate, we count a multiply and an add operation as 1 Op and non-linear functions as 5 Ops. For an LSTM, the number of operations is given by
  NLSTM = T H(8D + 8H + 29)
  where T is the number of time steps, H is the number of hidden units, and D is the dimension of the input data. For a g-LSTM, the number of operations is given by
  Ng−LSTM = NLSTM +Ngate
  where Ngate = 13T H is the total number of operations for computing the time gate. The total number of operations for the g-LSTM network on the sMNIST dataset is around 80 MOps for N = 110 and T = 784, after thresholding the budgeted g-LSTM this number is reduced to 7.6 MOps. Additional λ hyperparameters were also investigated for the sMNIST task, see Appendix D.
sections.heading: 4.4 TEMPORAL CURRICULUM TRAINING SCHEDULE FOR LSTMS
sections.text: We demonstrate that it is possible to train an LSTM network to converge faster on a difficult task by using a “temporal curriculum” training schedule for the equivalent g-LSTM network. According to this schedule, the initial σ values of the g-LSTM network are increased continuously throughout the
  training period ending up with high values by the end of training. With such high values, the time gates are essentially open, resulting in an LSTM network. At every training epoch, the lowest ρ% of the σ values, σ̂ in the layer are updated as: σ̂ −→ (1 + α) · σ̂. We analyze the impact of this training schedule for training an LSTM network on sCIFAR-10. For the equivalent g-LSTM network with 110 units, µ is initialized uniformly at random between 1 and 1024 and σ is initialized to 50. An α value of 1/6 and ρ value of 15% are chosen. To ensure that the time gate is fully open by the end of training, σ is set to 5000 across all units during the last 10 epochs of training. The learning rate of the time gate parameters is set to 0, i.e. µ and σ are no longer updated. Figure 6 shows that the temporal curriculum training schedule allows for faster convergence of an LSTM network. The final weights of the trained g-LSTM network can then be copied over to a LSTM network for inference.
sections.heading: 5 GRADIENT FLOW
sections.text: We present results regarding backpropagation flow through the LSTM and g-LSTM networks. Following the hypothesis presented in Section 3.2 on the reduced likelihood of vanishing or exploding gradients in the g-LSTM, we investigate the average gradient norms across time steps, similar to the work in (Krueger et al., 2016). We compute the gradient norms of the loss with respect to the hidden activations, the exact definition is given in Appendix E. Comparing the error propagation of the g-LSTM and LSTM networks on the SMNIST task (as in Sec. 4.1), Figure 7 shows the gradient norms at each time step after training for two different epochs.
  Interpreting the gradient flow from higher to lower time steps (right to left), the gradients of the g-LSTM shown in Fig. 7 show higher gradient values in earlier time steps than the LSTM. It is possible that one of the reasons the g-LSTM converges more quickly (as in Fig. 2 (c)) is that this back-propagated gradient information is more consistent across time steps and does not vanish at early time steps.
sections.heading: 6 CONCLUSION
sections.text: This work proposes a novel RNN variant with a time gate which is parameterized by the input in time. The convergence speeds of the g-LSTM and LSTM are similar for short sequence tasks but the g-LSTM shows faster convergence and produces higher accuracies than LSTM networks on long sequence tasks, as demonstrated for adding task sequences which are longer than 1000 timesteps; and on the sMNIST and sCIFAR-10 datasets. We also demonstrate that the time gate parameters of the g-LSTM (unlike those of the PLSTM) are learnable even when the time gates are initialized in an extreme non-optimal manner for the adding task. The time gate of the g-LSTM can reduce the number of computes that is needed for the updates of the LSTM equations and with an additional loss term to reduce the compute budget, the σ values of the time gate are reduced leading to a 10× decrease in the number of actual computes and with little loss in network accuracy, for the sMNIST dataset. The observation that the budgeted g-LSTM has neurons which are closed by the timing gate suggests that this method can be used to prune a network. We also show that our proposed temporal curriculum training schedule for the g-LSTM can help a corresponding LSTM network to converge during training on long sequence tasks. For future work, it will be of interest to investigate whether these properties carry over to larger or domain-specific datasets.
sections.heading: A BACK PROPAGATION IN GAUSSIAN-GATED RNN
sections.text: For ease of illustration we analyze the gradient of a plain RNN with a Gaussian time gate (Eqs. 7 and 8).
  hn = kn · h̃n + (1− kn) · hn−1 (7)
  h̃n = f(Wxxn +Whhn−1) (8)
  ∂hN ∂Wh = ∂h̃0 ∂Wh N∏ n=1 (knWhf ′ n + (1− kn)) + N∑ n=1 (knf ′ nhn−1) N∏ s=n+1 (ksWhf ′ s + (1− ks)) (9)
  where ∂h̃0∂Wh = 1, h0 = h̃0 = 0.
  From Eq. 9 we can deduce some information about the advantages of the Gaussian time gate in gradient flow for two simple cases of the function kn.
  In Case 1 we choose a timing gate openness which corresponds to a very small σ for the Gaussian gate, i.e. the gate is only open for 1 time step.
  k5 = 1, kn = 0 ∀ n ∈ {1, ..., N}\{5} ∂hN ∂Wh = f ′5 + f ′ 5f ′ 5h4 = f ′ 5Wh (1 + f ′ 5h4)
  In Case 2 we choose a timing gate openness which corresponds to a slightly larger σ for the Gaussian gate, i.e. it is open for 5 time steps.
  k2 = 1, k3 = 1, k4 = 1, k5 = 1, k6 = 1, kn = 0 ∀ n ∈ {1, ..., N}\{2, 3, 4, 5, 6}
  ∂hN ∂Wh = f ′2Whf ′ 3Whf ′ 4Whf ′ 5Whf ′ 6Wh (1 + f ′ 2h1 + f ′ 3h2 + f ′ 4h3 + f ′ 5h4 + f ′ 6h5)
  These cases show that there are fewer terms in the gradient for a timing gate that is opened for only a small fraction of the sequence.
sections.heading: B COMPARING VARIOUS G-LSTM INITIALIZATIONS
sections.heading: C COMPARING TIME GATE PARAMETER TRAINABILITY IN G-LSTM AND
sections.heading: D HYPERPARAMETER INVESTIGATION
sections.text: We look at the network performance for different hyperparameter values, focusing on the sMNIST task.
  Network Initialization and Optimizer In Fig. 2(c) of Section 4.1, we show that the g-LSTM network converges faster than the LSTM for the sMNIST task using the RMSProp optimizer and with an orthogonal initialization of LSTM kernels of both networks, as in (Cooijmans et al., 2016). In addition to using this initializer and optimizer we include results using the ADAM initializer (learning rate of 10−3) and a random weight initialization, “Xavier” as in (Glorot & Bengio, 2010). Across all of these different training techniques we consistently observe that the g-LSTM converges more quickly than the LSTM.
  We ran further experiments to compare the chrono initialization of the LSTM forget and input biases from (Tallec & Ollivier, 2018). The forget and input biases are set as bf ∼ log(U([1, Tmax − 1])) and bi = −bf where Tmax = 784 for the sMNIST task. The use of the time gate with the g-LSTM shortens the effective sequence length for each unit; to account for this, we also provide the results of using a smaller Tmax value for the chrono initialization, Tmax = σ = 250. The comparison of both g-LSTM and LSTM with the chrono initialization and with the “constant initialization” (bf = 1) in Fig. 12 shows that the g-LSTM with the constant initialization converges the fastest. We hypothesize that the g-LSTM can converge faster when using the constant initialization over the chrono initialization because the time gate’s effect of sequence-length-shortening reduces the necessity for long memory, for which chrono initialization seeks to provide. We see that when we reduce the maximum temporal dependency for the chrono initialization (to Tmax = 250, “chrono-gLSTM-250”) this g-LSTM network converges more quickly, similar to the g-LSTM with a constant bias initialization. This suggests that these two techniques, chrono initialization and a Gaussian time gate, could be used together to improve convergence in LSTM networks.
  Network Size Aside from the network size of 110 hidden units, we investigated the training convergence for two additional network sizes: 25 and 220 hidden units. Note that the LSTM for network size 25 is trained for 100 additional epochs until convergence was observed. Across all different network sizes the g-LSTM converges much faster than the LSTM network. With fewer hidden units,
  as seen in Fig. 13 (a), an even more dramatic speed-up in convergence is seen for the g-LSTM compared with the LSTM. The final LERs (g-LSTM, LSTM) for each network size are: 25 units (2.79%, 3.58%), 110 units (1.35%, 1.81%) , 220 units (1.10%, 1.34%).
  Budgeted g-LSTM We provide additional results of the budgeted network (subsection 4.3) for 2 additional λ values, λ = 0.1 and λ = 10, comparing with the original result, for λ = 1. The final LERs: 2.4% (λ = 0.1), 2.2% (λ = 1), 2.8% (λ = 10). The number of computes used by the network trained with λ = 10 is significantly lower than the network that was trained for both λ = 0.1 and 1.
sections.heading: E AVERAGE GRADIENT NORM DEFINITION
sections.text: The average gradient norm in Section 5 is defined as:
  Γ ∈ RN+ where N is the number of time steps of the sequence (for SMNIST, N = 784).
  Γ = 1
  K E ∑ (k) ∥∥∥∥ ∂L ∂h (k) n ∥∥∥∥  ≈ 1 LK ∑ (l,k) ∥∥∥∥ ∂L ∂h (k) n ∥∥∥∥ summing over all L samples of the training set and all K hidden units.
title: GAUSSIAN-GATED LSTM: IMPROVED CONVERGENCE BY REDUCING STATE UPDATES