3.3 Transformers
While recurrence in RNNs (Section 3.1) is a boon for neural networks to model sequential data, it is
also a bane as the recurrence in time to update the hidden state intrinsically precludes parallelization.
Additionally, although dedicated gated RNNs such as LSTM and GRU have helped to mitigate
the vanishing gradient problem to some extent, it can still be a challenge to maintain long-term
dependencies in RNNs.
AReviewofDeepLearningTechniquesforSpeechProcessing