(20)
(21)
SelfAttention(.) denotes attention module with Q = K = V = X, where X is the output of the
previous layer.
Transformer-based architecture turned out to be better than many other architectures such
as RNN, LSTM/GRU, etc. One of the major difficulties when applying a Transformer to speech
applications that it requires more complex configurations (e.g., optimizer, network structure, data
augmentation) than the conventional RNN-based models. Speech signals are continuous-time
signals with much higher dimensionality than text data. This high dimensionality poses significant
computational challenges for the Transformer architecture, originally designed for sequential text
data. Speech signals also have temporal dependencies, which means that the model needs to be
able to process and learn from the entire signal rather than just a sequence of inputs. Also, speech
signals are inherently variable and complex. The same sentence can be spoken differently and even
AReviewofDeepLearningTechniquesforSpeechProcessing