• Wav2Vec 2.0 [26] is a framework for self-supervised learning of speech representations that
is one of the current state-of-the-art models for ASR [26]. The training of the model occurs
in two stages. Initially, the model operates in a self-supervised mode during the first phase,
where it uses unlabelled data and aims to achieve the best speech representation possible.
The second phase is fine-tuning a particular dataset for a specific purpose. Wav2Vec 2.0
takes advantage of self-supervised training and uses convolutional layers to extract features
from raw audio.

In the speech field, researchers have explored different approaches to avoid overfitting, including
augmentation techniques like Speech SimCLR [220] and the use of positive and negative pairs
through methods like Contrastive Predictive Coding (CPC) (Ooster and Meyer [404]), Wav2vec
(v1, v2.0) (Schneider et al. [483]), VQ-wav2vec (Baevski et al. [25]), and Discrete BERT [23]." "In the
AReviewofDeepLearningTechniquesforSpeechProcessing