• Speech representation learning frameworks, such as wav2vec, have enabled significant ad-
vancements in speech processing tasks. One recent framework, w2v-BERT [585], combines
contrastive learning and MLM to achieve self-supervised speech pre-training on discrete
tokens. Fine-tuning wav2vec models with limited labeled data has also been demonstrated
to achieve state-of-the-art results in speech recognition tasks [25]. Moreover, XLS-R [20],
another model based on wav2vec 2.0, has shown state-of-the-art results in various tasks,
domains, data regimes, and languages, by leveraging multilingual data augmentation and
contrastive learning techniques on a large scale. These models learn universal speech rep-
resentations that can be transferred across languages and domains, thus representing a
significant advancement in speech representation learning.
• Transformers have been increasingly popular in the development of frameworks for learn-
AReviewofDeepLearningTechniquesforSpeechProcessing