Recurrent neural networks can be difficult to train on long sequence data due to the well-known vanishing gradient problem. Some architectures incorporate methods to reduce RNN state updates, thereby allowing the network to preserve memory over long temporal intervals. We propose a timing-gated LSTM RNN model, called the Gaussian-gated LSTM (g-LSTM) for reducing state updates. The time gate controls when a neuron can be updated during training, enabling longer memory persistence and better error-gradient flow. This model captures long temporal dependencies better than an LSTM on very long sequence tasks and the time gate parameters can be learned even from a non-optimal initialization. Because the time gate limits the updates of the neuron state, the number of computes needed for the network update is also reduced. By adding a computational budget term to the training loss, we obtain a network which further reduces the number of computes by at least 10×. Finally, we propose a temporal curriculum learning schedule for the g-LSTM that helps speed up the convergence time of the equivalent LSTM on long sequences.
In the vein of recent work on learning “ticking” behaviour for LSTMs such as Phased LSTM, this paper proposes to add additional data independent gates to LSTM units that are defined as Gaussian functions of time indices.
The performance of the modified g-LSTM is compared to LSTM on the Addition, sequential MNIST and sequential CIFAR-10 tasks. The authors argue that g-LSTM results in better performance and has faster convergence on these tasks. 
Additionally, it is proposed that one can reduce the amount of computations performed by the network by adding a computation budget term to the optimized loss that encouraged the cells to update less often. Finally, a technique for gradually transitioning from a g-LSTM to an LSTM during training is proposed, with the objective of speeding up training over a regular LSTM.
The paper is well written and easy to understand in general. However, the main results of this paper are experimental, and I am not entirely convinced by the experiments that g-LSTM is an improvement over the LSTM baseline for certain scenarios.
One broad reason for my doubts is that the comparisons don’t seem to utilise proper hyperparameter tuning for the baseline LSTM. Network sizes, learning rates, decay schedules, initialisations etc. all appear to be fixed, so one can not be sure of the “real” performance or convergence behavior of the models. Biased gate initializations are not used, though they have been used successfully in past work to aid in long term memory.
I should note that for long term memory problems such as those proposed by Hochreiter and Schmidhuber (1997), the proposed LSTM did not use a forget gate (or even BPTT) and used biased gate initialisations. However, these features are useful for more realistic tasks, and popular LSTM designs are biased towards them instead of toy problems.
I would consider the addition problem and sequential MNIST and CIFAR-10 to be interesting and difficult toy tasks for initial validation of ideas (and more extensive hyperparameter searches). It is unclear if the proposed techniques will perform provide  improvements over a well-tuned baseline for some realistic tasks, or are they suitable only for toy problems.