has several issues when applied to ASR. RNN, with its overall training speed (i.e., convergence)
and better WER because of effective joint training and decoding methods, is still the best option.
The authors in [116] propose the Speech Transformer, which has the advantage of faster iteration
time, but slower convergence compared to RNN-based ASR. However, integrating the Speech
Transformer with the naive language model (LM) is challenging. To address this issue, various
improvements in the Speech Transformer architecture have been proposed in recent years. For
example, [245] suggests incorporating the Connectionist Temporal Classification (CTC) loss into
the Speech Transformer. CTC is a popular technique used in speech recognition to align input and
output sequences of varying lengths and one-to-many or many-to-one mappings. It introduces
a blank symbol representing gaps between output symbols and computes the loss function by
AReviewofDeepLearningTechniquesforSpeechProcessing