They address this in the paper under section 3, &quot;Related work&quot;.<p>From the paper (there&#x27;s more on other things besides LSTM, but here&#x27;s the LSTM specific mention):<p>Hochreiter and Schmidhuber [20] introduced the Long Short Term Memory network (LSTM) architecture.
While this model was orginally developed to address the vanishing and exploding gradient
problems, LSTM is also able to learn simple context-free and context-sensitive grammars [17, 37].
This is possible because its hidden units can choose through a multiplicative gating mechanism to
be either linear or non-linear. The linear units allow the network to potentially count (one can easily
add and subtract constants) and store a finite amount of information for a long period of time. These
mechanisms are also used in the Gated Recurrent Unit network [7, 9]. In our work we investigate
the use of a similar mechanism in a context where the memory is unbounded and structured. As
opposed to previous work, we do not need to “erase” our memory to store a new unit.<p>Edit: They also provide a table in the paper showing improvements over LSTM for certain types of algorithms.