Long short-term memory networks (LSTMs) were introduced to combat vanishing gradients in simple recurrent neural networks (S-RNNs) by augmenting them with additive recurrent connections controlled by gates. We present an alternate view to explain the success of LSTMs: the gates themselves are powerful recurrent models that provide more representational power than previously appreciated. We do this by showing that the LSTM's gates can be decoupled from the embedded S-RNN, producing a restricted class of RNNs where the main recurrence computes an element-wise weighted sum of context-independent functions of the inputs. Experiments on a range of challenging NLP problems demonstrate that the simplified gate-based models work substantially better than S-RNNs, and often just as well as the original LSTMs, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients.
This paper presents an analysis of LSTMS showing that they have a from where the memory cell contents at each step is a weighted combination of the “content update” values computed at each time step. The weightings are defined in terms of an exponential decay on each dimension at each time step (given by the forget gate), which lets the cell be computed sequentially in linear time rather than in the exhaustive quadratic time that would apparently be necessary for this definition. Second, the paper offers a simplification of LSTMs that compute the value by which the memory cell at each time step in terms of a deterministic function of the input rather than a function of the input and the current context. This reduced form of the LSTM is shown to perform comparably to “full” LSTMs.
The decomposition of the LSTM in terms of these weights is useful, and suggests new strategies for comparing existing quadratic time attention-based extensions to RNNs. The proposed model variations (which replaces the “content update” that has a recurrent network in terms of context-independent update) and their evaluations seem rather more arbitrary. First, there are two RNNs present in the LSTM- one controls the gates, one controls the content update. You get rid of one, not the other. You can make an argument for why the one that was ablated was “more interesting”, but really this is an obvious empirical question that should be addressed. The second problem of what tasks to evaluate on is a general problem with comparing RNNs. One non-language task (e.g., some RL agent with an LSTM, or learning to execute or something) and one synthetic task (copying or something) might be sensible. Although I don’t think this is the responsibility of this paper (although something that should be considered).
Finally, there are many further simplifications of LSTMs that could have been explored in the literature: coupled input-forget gates (Greff et al, 2015), diagonal matrices for gates, GRUs. When proposing yet another simplification, some sense for how these different reductions is useful, so I would recommend comparison to those.
Notes on clarity:
Before Eq 1 it’s hard to know what the antecedent of “which” is without reading ahead.
For componentwise multiplication, you have been using \circ, but then for the iterated component wise product, \prod is used. To be consistent, notation like \odot and \bigodot might be a bit clearer.
The discussion of dynamic programming: the dynamic program is also only available because the attention pattern is limited in a way that self attention is not. This might be worth mentioning.
When presenting Eq 11, the definition of w_j^t elides a lot of complexity. Indeed, w_j^t is only ever implicitly defined in Eq 8, whereas things like the input and forget gates are defined multiple times in the text. Since w_j^t can be defined iteratively and recursively (as a dynamic program), it’s probably worth writing both out, for expository clarity.
Eq 11 might be clearer if you show that Eq 8 can also be rewritten in the same wheat, provided, you make h_{t-1} an argument to output and content.
Table 4 is unclear. In a language model, the figure looks like it is attending to the word that is being generated, which is clearly not what you want to convey since language models don’t condition on the word they are predicting. Presumably the strong diagonal attention is attending to the previous word when computing the representation to generate the subsequent word? In any case, this figure should be corrected to reflect this. This objection also concerns the right hand figure, and the semantics of the meaning of the upper vs lower triangles should be clarified in the caption (rather than just in the text).