Various other improvements have also been proposed to enhance the performance of Speech
Transformer architecture and integrate it with the naive language model, as the use of the trans-
former directly for ASR has not been effective in exploiting the correlation among the speech
frames. The sequence order of speech, which the recurrent processing of input features can repre-
sent, is an important distinction. The degradation in performance for long sentences is reported
using absolute positional embedding (AED) [85]. The problems associated with long sequences
can become more acute for transformer [672]. To address this issue, a transition was made from
absolute positional encoding to relative positional embeddings [672]. Whereas authors in [537]
replace positional embeddings with pooling layers. In a considerably different approach, the authors
in [383] propose a novel way of combining positional embedding with speech features by replacing
AReviewofDeepLearningTechniquesforSpeechProcessing