detailed architecture for generative models with three different variants is shown in Figure 11. The
earliest self-supervised method, predicting masked inputs using surrounding data, originated from
the text field in 2013 with word2vec. The continuous bag of words (CBOW) concept of word2vec
predicts a central word based on its neighbors, resembling ELMo and BERT’s masked language
modeling (MLM). These non-autoregressive generative approaches differ in their use of advanced
structures, such as bidirectional LSTM (for ELMo) and transformer (for BERT), with recent models
producing contextual embeddings. In the context of the speech, Mockingjay [330] applied masking
to all feature dimensions in the speech domain, whereas TERA [329] applied to mask only to a
particular subset of feature dimensions. The summary of generative self-supervised approaches
along with the data used for training the models are outlined in Table 2. We further discuss different
AReviewofDeepLearningTechniquesforSpeechProcessing