h-DETACH: MODIFYING THE LSTM GRADIENT TO- WARDS BETTER OPTIMIZATION
Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps. We introduce a simple stochastic algorithm (h-detach) that is specific to LSTM optimization and targeted towards addressing this problem. Specifically, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed. Based on the hypothesis that these components carry information about long term dependencies (which we show empirically), their suppression can prevent LSTMs from capturing them. Our algorithm 1 prevents gradients flowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better. We show significant improvements over vanilla LSTM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization using our modification of LSTM gradient on various benchmark datasets.
The results are intriguing. However, similar methods like BN-LSTM [3] and Variational RNNs [4] achieve arguably the same with very similar mechanisms. We do not think they can be considered as orthogonal. This should be addressed by the authors. Also, hard long-term experiments like sequentially predicting pixels (like through MDLSTM-based PixelRNN) or language modelling should be favoured over short sentence image captions. 
It is possible that we will improve our ratings once our concerns are addressed.
Paper Summary:
The authors claim that the gradient along the computational path that goes through the cell state (the linear temporal path or A gradient) of an LSTM carries information about long-term dependencies. Those gradients can be corrupted by the gradient of all other computational paths (i.e. the B gradient). They claim that this makes it hard to learn long-term dependencies and has, therefore, significant negative effects on the convergence speed, training stability, and generalisation performance. They propose a method called h-detach and run experiments on the delayed copy task, sequential MNIST, permuted sequential MNIST (pMNIST), and caption generation on the MS COCO dataset. All show either somewhat improved performance or much more stable learning curves. At every step, h-detach randomly drops all gradients that flow through the h of the standard LSTM, the B gradients, and only keeps the ones from the linear temporal path, the A gradients. Experiments also suggest that the A gradients carry more long-term information than B gradients and that LSTMs with h-detach do not need gradient clipping for successful training.
Positive:
The paper is written clearly. It is well structured and well motivated. H-detach is simple, effective, and somewhat novel (see below). Experiments indicate that its main benefit is training stability as well as minor performance improvements.
Negative:
We are not sure how significant these results are for the following reasons:
- MS COCO image caption generation is the only more challenging dataset, but it seems a bit misplaced as it has very short sentences, while the authors motivate their work through a focus on long-term dependencies. Why not apply h-detach to a language model such as [1] with official online implementations, e.g., [2]. A setting with PixelRNN [6] based on MD-LSTM [7] would also be a great testbed for h-detach.
- The purpose of h-detach is to scale down the B gradients. However, methods which apply e.g. BatchNorm to the hidden state learn a scale parameter which could be learned by the network explicitly. For the backward pass, this has the effect of scaling down the B gradient. Consider e.g. [3] which also achieves similar training stability on sequential MNIST and pMNIST with little overhead. 
- Another very related method is [4] which properly applies a random dropout mask over the recurrent inputs that is shared across timesteps of an RNN. We think that h-detach is essentially achieving the same in a similar way.
Problems with Introduction and Related Work Section:
- The vanishing gradient problem was first described by Hochreiter in 1991 [5] (not by Bengio in 1994). 
- Intro mentions GRU as if it was separate from LSTM. Clarify that GRU is essentially a variant of vanilla LSTM with forget gates [8]. Since one gate is missing, GRU is less powerful than the original LSTM [9]. 
[1] Zaremba et al. "Recurrent neural network regularization." arXiv:1409.2329 (2014).
[2] https://www.tensorflow.org/tutorials/sequences/recurrent
[3] Cooijmans et al. "Recurrent batch normalization." arXiv:1603.09025 (2016).
[4] Gal et al. "A theoretically grounded application of dropout in recurrent neural networks." NIPS 2016.
[5] Hochreiter, Sepp. "Untersuchungen zu dynamischen neuronalen Netzen." Diploma thesis, TUM (1991)
[6] Oord et al. "Pixel recurrent neural networks." arXiv preprint arXiv:1601.06759 (2016).
[7] Graves et al. "Multi-Dimensional Recurrent Neural Networks" arXiv preprint arXiv:0705.2011 (2011).
[8] Gers et al. “Learning to Forget: Continual Prediction with LSTM.“ Neural Computation, 12(10):2451-2471, 2000. 
[9] Weiss et al. On the Practical Computational Power of Finite Precision RNNs for Language Recognition. arXiv:1805.04908.
Comments after rebuttal:
The  paper has clearly improved. 
It leaves a few questions open though. For example, it is surprising that h-detach doesn't work on language modelling since Dropout-LSTM and BN-LSTM clearly improve over vanilla LSTM in this case (if not every case). In the new version, the authors only reference it in one or two sentences but don't discuss this in detail. 
When dropout is mentioned, one should also mention that dropout is a variant of the old stochastic delta rule:
Hanson, S. J. (1990). A Stochastic Version of the Delta Rule, PHYSICA D,42, 265-272.  See also arXiv:1808.03578 
Nevertheless, we now think that this is a very interesting LSTM regularization paper that people who study this field should probably know. We are increasing the score by 2 points!