Transfer learning has proven to be crucial in advancing the state of speech and natural language processing research in recent years. In speech, a model pre-trained by self-supervised learning transfers remarkably well on multiple tasks. However, the lack of a consistent evaluation methodology is limiting towards a holistic understanding of the efficacy of such models. SUPERB was a step towards introducing a common benchmark to evaluate pre-trained models across various speech tasks. In this paper, we introduce SUPERB-SG , a new benchmark focused on evaluating the semantic and generative capabilities of pre-trained models by increasing task diversity and difficulty over SUPERB . We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain and quality across different types of tasks. It entails freezing pre-trained model parameters, only using simple task-specific trainable heads. The goal is to be inclusive of all researchers, and encourage efficient use of computational resources. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation. Transfer learning is a paradigm in machine learning that has been very effective for natural language processing (NLP) (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Lan et al., 2019; Dong et al., 2019; Yang et al., 2019; Raffel et al., 2020; Lewis et al., 2019; Conneau et al., 2020), and speech processing (van den Oord et al., 2018; Rivire et al., 2020; Chung et al., 2019; Schneider et al., 2019; Baevski et al., 2020b; Hsu et al., 2021; Liu et al., 2020c,b; Ravanelli et al., 2020; Ling et al., 2020; Ling and Liu, 2020). Self-supervised learning (SSL) is the main driver of this paradigm, an effective and scalable way to learn high-level representation of language that transfers to a variety of tasks. SSL entails learning from the input or some perturbation of it without the need for labelled data. This has unlocked the usage of large amounts of cheaply available unlabelled data. It lends naturally to neural network models that have been shown to possess impressive scaling characteristics such that it is often enough to increase the model and data sizes to improve downstream performance (Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Radford et al., 2019). Speech signal consists of acoustic, linguistic, prosodic, and speaker characteristics. SSL algo-8479 rithms in speech must be evaluated in their ability to produce representations that are useful for tasks that demand understanding of linguistic, speaker, and prosodic elements of spoken language as well as high-level semantics. Researchers have used auto-regressive, contrastive, discriminative and multi-task learning objectives to pre-train models, and have investigated their capabilities across tasks like phoneme recognition (van den Oord et al., 2018; Chung et al., 2019), automatic speech recognition (ASR) (Liu et al., 2020b; Schneider et al., 2019; Ling and Liu, 2020; Ravanelli et al., 2020; Hsu et al., 2021; Chang et al., 2021), speaker verification (Fan et al., 2020), speaker identification (Chung et al., 2019; Liu et al., 2020c), emotion recognition (Macary et al., 2021), speech translation (Chung et al., 2019), voice conversion (Lin et al., 2020; Huang et al., 2021a), spoken language understanding (Lai et al., 2021), and text-to-speech (lvarez et al., 2019). However, the methodologies in such studies vary in the use of datasets, fine-tuning strategies and task-specific model architectures. To bridge this gap, SUPERB (Yang et al., 2021) introduced a standardized benchmark of 10 speech tasks to compare 13 pre-trained models and a Log Mel-Filterbank baseline. It studied the models' performance in tasks focusing on linguistic (phoneme recognition and automatic speech recognition, keyword spotting and query by exam-ple), shallow semantic (intent classification and slot filling), speaker (speaker identification, speaker verification and speaker diarization), and prosodic (emotion recognition) characteristics. In this paper, we introduce SUPERB-SG , a benchmark with 5 new tasks, which are speech translation, out-of-domain ASR, voice conversion, speech separation, and speech enhancement, with an emphasis on evaluating the semantic and generative capabilities of pre-trained models that require high-level representations to capture linguistic, semantic, and speaker characteristics. These tasks go beyond speech recognition by focusing on various other aspects that are essential to building intelligent speech interfaces. Further, we show that while SSL models achieve close to state-of-the-art performance on many tasks, there isn't one model that outperforms all others, and that a simple Log Mel-Filterbank can perform competitively on some tasks. We also demonstrate the robustness of our methodology with an ablation study over different task-specific model architectures and data sizes. The introduction of these new tasks of varying difficulty takes us closer to a more comprehensive unified standard speech benchmark. We hope that this will motivate the development of more powerful, generalizable, and reusable pre-trained models to democratize the advancement of speech research. To facilitate this, we released the codes 1 and integrated the tasks with the SUPERB benchmark. As more powerful SSL models are proposed with promising performance on various tasks, researchers continually try to find extensive evaluation methods to assess model performance, and wish to holistically understand the capability of the learned representations in these models. SUPERB (Yang et al., 2021) is a framework to benchmark the SSL models on 10 speech tasks by learning task-specific prediction heads on top of the frozen shared SSL models. Although the tasks in SUPERB span across different domains, most of them are simple classification problems, or only require utilization of shallow semantics. In contrast, we focus on harder semantic and generative tasks. Another recently proposed benchmark is the LeBenchmark (Evain et al., 2021), investigating the performance of SSL models trained on French data with three semantic tasks. However, they only consider wav2vec 2.0 (Baevski et al., 2020b) with 1 https://github.com/s3prl/s3prl: Tasks in SUPERB-SG are open-sourced and reproducible in the S3PRL toolkit which supports benchmarking the most existing and customized pre-trained models. different architectures as their upstream models (i.e., networks pre-trained with SSL). Here, we evaluate a diverse set of SSL models, and offer a more comprehensive analysis. The Zero Resource Speech Benchmark 2021 (Nguyen et al., 2020) introduces unsupervised speech processing tasks, particularly the spoken language modeling problem. They evaluate the SSL models via zero-shot probings at four linguistic levels. While their benchmark task is specific for certain domain, we use various tasks to evaluate different aspects of SSL models. The HEAR 2021 Challenge 2 aims to develop general-purpose audio representation by focusing on audio tasks beyond speech that include sound event detection, speech commands and pitch & chroma classification. We specifically focus on various aspects of speech processing, thus providing a wide variety of spoken language tasks. This section introduces the tasks in SUPERB-SG , including why we choose these tasks and how we design the task-specific heads for fine-tuning. Following SUPERB 's methodology, we use a lightweight fine-tuning approach wherein we freeze the pre-trained model parameters and only keep the task-specific head's parameters trainable. This setting serves the dual purpose of evaluating the robustness as well as the generalizability of the speech representations, and provides a resource-efficient way of fine-tuning the models that is inclusive of participants with constrained compute resources. We call the pre-trained model as upstream model and the task-specific heads as downstream model. We now discuss the newly added tasks in SUPERB-SG in the following sub-sections. Speech translation (ST) involves translating the acoustic speech signals in the source language into the words in the target language. We use it to evaluate the semantic capability of SSL models, and how they benefit the translation task. We use the CoVoST2 En  De (Wang et al., 2020) dataset (CC0 Licensed) with their official train, validation, and test splits while removing all the samples containing "REMOVE", resulting in 425.8, 25.9 2 https://neuralaudio.ai/hear2021-holistic-evaluation-of-audio-representations.html and 24.5 hours respectively. For text, we keep original case, normalize punctuation, and build character vocabulary with 100% train-set coverage. We report case-sensitive de-tokenized BLEU using sacreBLEU (Post, 2018). Our downstream model has an encoder-decoder architecture with 3 layers of Transformers (Vaswani et al., 2017) each with hidden dimension of 512. A convolutional sub-sampler is used to reduce the sequence length of the input before feeding it to the encoder. We train our model with label-smoothing using a probability of 0.1. A beam size of 20 is used for inference. Although an ASR is included in SUPERB , it only examines SSL models on read English corpus LibriSpeech (Panayotov et al., 2015). Therefore, we introduce out-of-domain ASR (OOD-ASR), which aims to evaluate the models' capabilities across languages, and out-of-domain scenarios. The OOD-ASR tasks are categorized into cross-lingual and spontaneous speech tasks. For the cross-lingual tasks, we choose the Mexican Spanish (es), Mandarin (zh), and Arabic (ar) subsets from Common Voice 7.0 (Ardila et al., 2020) (CC0 Licensed) containing 21.5, 31.2, and 30.7 hours of training data respectively. The validation set sizes are 1.2 hours, 14.4 hours and 12.24 hours, and the test set sizes are 0.6 hour, 15.3 hours and 12.5 hours for es, zh and ar respectively. For the spontaneous speech task (spon), we use the Santa Barbara Corpus of Spoken American English (SBCSAE) (Du Bois et al., 2000  2005) (CC BY-ND 3.0 Licensed), consisting of 60 conversations over different topics spanning 16.7 hours of data. The validation and test set sizes are 1.6 hours and 2.2 hours respectively. For evaluation, we use word error rate (WER) as the metric except for Mandarin which character error rate (CER) is used. The error rates are averaged across all sub-tasks to offer an overall score. The ASR model is a 2-layer BLSTM (Hochreiter and Schmidhuber, 1997) with hidden states of 1024 dimension. The training objective is to minimize the Connectionist Temporal Classification (CTC) loss (Graves et al., 2006). During inference, we use CTC greedy decoding without language model re-scoring to simplify the process and to highlight the impact of the learned acoustic representations. For voice conversion (VC), we consider the intralingual VC task in VCC2020 (Zhao et al., 2020) (ODbL Licensed) under the any-to-one (A2O) setting. A2O VC aims to convert speech from any arbitrary speaker into that of a predefined target speaker. We use the task to evaluate the speaker transferability as well as the generalizability of the SSL models. We use 60 utterances from the target speaker that spans 5 minutes for training, and 25 utterances for testing that span 2 minutes. No validation set was used. We use the commonly used mel-cepstrum distortion (MCD), word error rate (WER) and automatic speaker verification (ASV) accept rate from off-the-shelf ASR and ASV models as evaluation metrics. The downstream model is trained to reconstruct the acoustic feature from the upstream representations in a target-speaker-dependent manner. In the conversion phase, given the representations extracted by the upstream, the model generates the converted acoustic features, which are then sent to a neural vocoder to synthesize the converted waveform. We adopted Tacotron2 (Shen et al., 2018) as the downstream model, which is an autoregressive network consisting of convolutional and LSTM layers. For the neural vocoder, we used the Hifi-GAN (Kong et al., 2020). We follow an implementation described in (Huang et al., 2021b). Speech separation (SS) is the task of separating target speech from background interference (Wang and Chen, 2018). It is an important step in speech processing, especially for noisy and multi-speaker scenarios. We investigate the speech separation problem on a dataset simulated from LibriSpeech (Cosentino et al., 2020) (CC BY 4.0 Licensed) and WHAM! (Wichern et al., 2019) (CC BY-NC 4.0 Licensed) noise. We use 16kHz version of the dataset containing 2 speakers, and focus on the mix_clean condition. The train and evaluation sets contain 43.3 and 4.2 hours of speech simulated from LibriSpeech's train-clean-100 and test-clean . This task is used to evaluate the generative capability of SSL models when input is a mixture of acoustic signals. We use the scale-invariant signal-to-distortion ratio improvement (SI-SDRi) as the evaluation metric. For the downstream model, we use a 3-layer BLSTM model with dimension of 896 for each direction to predict the short-time Fourier transform (STFT) masks for each speaker, and the predictions are transformed back to the time domain using inverse short-time Fourier transform (iSTFT). Permutation invariant training (PIT) (Yu et al., 2017) is performed to optimize the mean square error between the predicted mask and Ideal Non-negative Phase Sensitive Mask (INPSM) (Er-dogan et al., 2015; Kolbk et al., 2017). We choose frequency domain method instead of a time domain based method because of the stride size constraint and computational cost. Speech enhancement (SE) is the task of removing background noise from a degraded speech signal, and it aims to improve the perceived quality and intelligibility of the signal. We include this 8482 Upstream ST OOD-ASR VC SS SE BLEU  WER  MCD  WER  ASV  SI-SDRi  PESQ  STOI  FBANK 2.32 63.58 8.47 38.3 77.25 9.23 2.55 93.6 PASE+ 3.16 61.56 8.66 30.6 63.20 9.87 2.56 93.9 APC 5.95 63.12 8.05 27.2 87.25 8.92 2.56 93.4 VQ-APC 4.23 63.56 7.84 22.4 94.25 8.44 2.56 93.4 NPC 4.32 61.66 7.86 30.4 94.75 8.04 2.52 93.1 Mockingjay 4.45 65.27 8.29 35.1 79.75 9.29 2.53 93.4 TERA 5.66 58.49 8.21 25.1 83.75 10.19 2.54 93.6 DeCoAR 2.0 9.94 53.62 7.83 17.1 90.75 8.54 2.47 93.2 Modified CPC 4.82 62.54 8.41 26.2 71.00 10.40 2.57 93.7 wav2vec 6.61 55.86 7.45 10.1 98.25 9.30 2.53 93.8 vq-wav2vec 5.66 60.66 7.08 13.4 100.00 8.16 2.48 93.6 wav2vec 2.0 Base 14.81 46.95 7.50 10.5 98.00 9.77 2.55 93.9 wav2vec 2.0 Large 12.48 44.69 7.63 15.8 97.25 10.02 2.52 94.0 HuBERT Base 15.53 46.69 7.47 8.0 98.50 9.36 2.58 93.9 HuBERT Large 20.01 44.08 7.22 9.0 99.25 10.45 2.64 94.2 Table 2: Evaluating various SSL representations on new semantic and generative downstream tasks. In SUPERB-SG , we discuss the speech enhancement problem on the Voicebank-DEMAND (Veaux et al., 2013) (CC BY 4.0 Licensed) corpus. The train, validation, and test sets contain 8.8, 0.6 and 0.6 hours of speech respectively. Our evaluation metrics are Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI). For the downstream model, we follow the mask-based speech enhancement pipeline in (Kolbk et al., 2017). A 3-layer BLSTM model similar to the speech separation task is trained to predict the spectral mask for the clean signal. The mean square error between the predicted mask and INPSM is used as the objective. We evaluate the tasks on 15 upstream models, which are PASE+ (Ravanelli et al., 2020), APC (Chung et al., 2019), VQ-APC (Chung et al., 2020), NPC (Liu et al., 2020a), Mockingjay (Liu et al., 2020c), TERA (Liu et al., 2020b), DeCoAR 2.0 (Ling and Liu, 2020), Modifile CPC (Rivire et al., 2020), wav2vec family (Schneider et al., 2019) (Baevski et al., 2020a) (Baevski et al., 2020b) and HuBERT (Hsu et al., 2021). They span across different architectures, sizes and learning objectives. Some models also use vector quantization which has an added benefit of signal compression. For grounding, we use Log Mel Filterbank as our baseline. The detailed properties of upstream models are shown in Table 1. 4 Experimental Setup Following SUPERB , we fix upstream models parameters for all downstream tasks' training. We extract the frame-level representations for each hidden layer of the upstream models from raw waveform, and use a trainable task-specific weighted-sum mechanism to summarize all layers' representations into a sequence of vectors. The summarized representations are then used as the downstream model's input. An overview of the training procedure is demonstrated in Figure 1. Each experiment is done by one single run with the same seed. This procedure is consistent for all experiments, offering a fair and simple evaluation strategy for all upstream models. The results of the upstream models evaluated on SUPERB-SG are shown in Table 2. We only report the averaged WER for OOD-ASR. Full results can be found in Appendix A. For speech-to-text tasks (ST and OOD-ASR), wav2vec 2.0 and HuBERT offer competitive results, while DeCoAR 2.0 shows some improvements. In speech generation tasks (VC, SS, and SE), FBANK yields comparable or superior performance than some SSL models, especially for those metrics that take the quality of the output signal into account. For VC, the 3 reported metrics have the same trend for respective 8483 STOOD-ASRVC ( MCD ) VC ( WER ) VC ( ASV ) SSSE ( PESQ ) SE ( STOI ) PRASRSIDASVICERST OOD-ASR VC (MCD) VC (WER) VC (ASV) SS SE (PESQ) SE (STOI) PR ASR SID ASV IC ER 1.0 .86 .74 .85 .69 .38 .10 .51 .89 .92 .86 .75 .92 .83 .86 1.0 .69 .83 .65 .46 .02 .66 .86 .92 .79 .70 .83 .79 .74 .69 1.0 .90 .98 -.08-.15 .31 .84 .75 .61 .66 .87 .63 .85 .83 .90 1.0 .85 .24 .13 .54 .91 .85 .75 .70 .90 .79 .69 .65 .98 .85 1.0 -.13-.11 .30 .79 .71 .58 .66 .84 .57 .38 .46 -.08 .24 -.13 1.0 .52 .78 .23 .40 .31 .04 .15 .37 .10 .02 -.15 .13 -.11 .52 1.0 .46 .08 -.01 .21 .10 .05 .25 .51 .66 .31 .54 .30 .78 .46 1.0 .52 .54 .42 .41 .44 .53 .89 .86 .84 .91 .79 .23 .08 .52 1.0 .89 .89 .85 .98 .94 .92 .92 .75 .85 .71 .40 -.01 .54 .89 1.0 .83 .71 .88 .83 .86 .79 .61 .75 .58 .31 .21 .42 .89 .83 1.0 .81 .88 .88 .75 .70 .66 .70 .66 .04 .10 .41 .85 .71 .81 1.0 .85 .81 .92 .83 .87 .90 .84 .15 .05 .44 .98 .88 .88 .85 1.0 .89 .83 .79 .63 .79 .57 .37 .25 .53 .94 .83 .88 .81 .89 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Figure 2: Spearman's  between tasks. models. Here, vq-wav2vec achieves the best performance on MCD and ASV, while HuBERT performs the best on WER. For SS, Hubert-Large achieves the best performance, followed by Modified CPC. PASE+, which is pre-trained with denoising tasks, performs better than half the SSL models, but this observation doesn't transfer to the other tasks. For SE, all upstream models perform comparably. The largest gap is only 0.17 in PESQ and 1.1 in STOI. Overall, no model outperforms all others on all tasks. However, HuBERT-Large performs most competitively on all downstream tasks, especially those requiring linguistic and semantic signals. We analyze the correlations between tasks in SUPERB-SG to understand the similarity between tasks, and verify if the experimental results agree with the common understanding of related tasks based on shared representation they require. To compute the correlation, we first change all metrics into a higher-better manner. Then, we compute the Spearman's rank correlation coefficients (Spearman's  ) between all pairs of tasks. For multiple metrics contained in a single task, such as MCD/WER/ASV in VC as well as PESQ/STOI in SE, we compute each of them separately. To make our analysis more representative and generalized to all speech domains, we bring back the six tasks from SUPERB (Yang et al., 2021) that are considered representative of the following four domains: (i) Content recognition tasks contain( A ) ST ( A ) OOD-ASR ( A ) PR ( A ) VC ( WER ) ( A ) ASR ( A ) IC ( B ) SID ( B ) ASV ( B ) ER ( C ) VC ( MCD ) ( C ) VC ( ASV ) ( D ) SS ( E ) SE ( PESQ ) ( F ) SE ( STOI ) ST (A) OOD-ASR (A) PR (A) VC (WER) (A) ASR (A) IC (A) SID (B) ASV (B) ER (B) VC (MCD) (C) VC (ASV) (C) SS (D) SE (PESQ) (E) SE (STOI) (F) 1.0 .86 .89 .85 .92 .92 .86 .75 .83 .74 .69 .38 .10 .51 .86 1.0 .86 .83 .92 .83 .79 .70 .79 .69 .65 .46 .02 .66 .89 .86 1.0 .91 .89 .98 .89 .85 .94 .84 .79 .23 .08 .52 .85 .83 .91 1.0 .85 .90 .75 .70 .79 .90 .85 .24 .13 .54 .92 .92 .89 .85 1.0 .88 .83 .71 .83 .75 .71 .40 -.01 .54 .92 .83 .98 .90 .88 1.0 .88 .85 .89 .87 .84 .15 .05 .44 .86 .79 .89 .75 .83 .88 1.0 .81 .88 .61 .58 .31 .21 .42 .75 .70 .85 .70 .71 .85 .81 1.0 .81 .66 .66 .04 .10 .41 .83 .79 .94 .79 .83 .89 .88 .81 1.0 .63 .57 .37 .25 .53 .74 .69 .84 .90 .75 .87 .61 .66 .63 1.0 .98 -.08-.15 .31 .69 .65 .79 .85 .71 .84 .58 .66 .57 .98 1.0 -.13-.11 .30 .38 .46 .23 .24 .40 .15 .31 .04 .37 -.08-.13 1.0 .52 .78 .10 .02 .08 .13 -.01 .05 .21 .10 .25 -.15-.11 .52 1.0 .46 .51 .66 .52 .54 .54 .44 .42 .41 .53 .31 .30 .78 .46 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Figure 3: Spearman's  between tasks rearranged by clustering result. ing Phoneme Recognition (PR), Automatic Speech Recognition (ASR) (ii) Speaker identity tasks including Identification (SID), Automatic Speaker Verification (ASV) (iii) Semantics task which is Intent Classification (IC) and (iv) Prosodic task which is Emotion Recognition (ER). Together with the 5 tasks introduced in this paper, we show the results of total 11 downstream tasks with the 14 corresponding metrics in Figure 2. Overall, results show that all tasks except SS and SE have strong positive correlation among them. One possible explanation for SS and SE not showing strong correlation is that the low-level information closely related to audio signals is more critical as they need to reconstruct clean speech from interfering speakers and background noise by estimating the STFT masks. As a result, high-level information extracted from SSL models has little benefit for these tasks but is helpful for other tasks. As noted earlier, there is only a small gap in performance between FBANK and SSL models. If we leave SS and SE out, all correlation coefficients are greater than 0.58, showing that the SSL model representations are useful for multiple domains. Although the Spearman's  are large in general in Figure 2, differences between tasks are observable. Here, we focus on the relation between correlation and similarity of tasks. We list the most and the least two correlated tasks comparing with ST, OOD-ASR, VC, SS, and SE. SS and SE are skipped as candidates for for the least correlated 8484 Tasks Top 2 Last 2 ST ASR(0.92) IC (0.92) ASV(0.75) VC (0.76) OOD-ASR ASR(0.92) PR (0.86) ASV(0.70) VC (0.72) VC PR (0.84) ASR(0.77) SID(0.64) ER (0.66) SS SE (0.65) OOD-ASR(0.46) VC (0.01) ASV(0.04) SE SS (0.65) ER (0.39) VC (0.17) IC (0.25) Table 3: Top 2 and last 2 tasks correlated with the five SUPERB-SG tasks ranked by Spearman's  . tasks since they dominate the results. For VC, we average the correlation coefficients across the three metrics. The results are shown in Table 3. ST and OOD-ASR are highly correlated with ASR since they both transform speech signals into discrete text tokens. IC is also correlated with ST since semantic information is required to perform both tasks. Moreover, ASV and VC are the least correlated tasks since they primarily focus on the speaker information with lesser regard to the semantic content. However, the absolute correlation values are still larger than 0.7. For VC, the speaker information needs to be removed while the content has to be kept, similar to PR and ASR but different from SID. SS and SE are correlated with each other and have a much lower correlation with speaker identity and semantics tasks, supporting our assumption. Overall, we find that empirically highly-correlated tasks require similar knowledge or understanding ability. To give a broader view of our correlation results, we further cluster the downstream tasks by their correlation with each other using K-means. In this way, all the tasks are considered simultaneously, and the grouping is driven automatically by the empirical correlation results. If more than one metric are used in a downstream task, we cluster them independently. The clustering results are shown in Table 4 and a rearranged correlation map is shown in Figure 3. The result shows that the clusters of the tasks align with our empirical knowledge. Cluster A includes tasks that require content information, while tasks in cluster B are more sensitive to speaker and prosodic features. Cluster C contains metrics MCD and ASV of VC, which are used to evaluate the signal quality and the rates of speaker transfer. It is worth noting that WER in VC belongs to cluster A, showing that it is more similar to content-related tasks. Furthermore, clusters D, E, and F each contain one of the metrics in SS and SE, aligning with our assumption that these tasks utilize different types of information compared to other tasks. With the analysis of the correlation between tasks, we empirically confirm the reliability of the results, and show that we increase the heterogeneity among speech tasks over SUPERB . We further discover shared properties between tasks with clustering, and the result is aligned with our common understanding of related tasks. To study the impact of downstream model architecture and the data sizes used in SUPERB-SG we evaluate the robustness of SUPERB-SG with variations in downstream model as well as training data size, and show that our conclusions still hold true. We choose ST, OOD-ASR and SS as the downstream tasks for evaluation with an aim to cover semantic, content recognition, and generative task types. For the upstream models, FBANK, TERA, CPC, wav2vec 2.0 Base and HuBERT Base are used to cover different SSL algorithms. For each task, 2 additional downstream architectures are created by modifying the number of layers and the hidden dimensions compared to our default setting. We create small and large models that are roughly the half and twice of default in terms of the number of trainable parameters. A detailed comparison of the downstream architectures is shown in Table 5. The results are shown in Table 6. We show that the ranking of the upstream models is almost fixed when the model sizes are varied. As expected, the small architecture has worse perfor-8485 Architecture ST OOD-ASR SS architecture #params architecture #params architecture #params default 3-layer encoder 3-layer decoder Transformer(dim512) 28.8M 2-layer BLSTM (dim 1024) 53.4M 3-layer BLSTM (dim 896) 51.4M small no encoder 1-layer decoder Transformer(dim512) 10.9M(  0.38) 1-layer BLSTM (dim 1024) 24.1M(  0.45) 2-layer BLSTM (dim 768) 24.4M(  0.47) large 12-layer encoder 6-layer decoder Transformer(dim512) 69.8M(  2.42) 4-layer BLSTM (dim 1024) 112.2M(  2.10) 4-layer BLSTM (dim 1152) 114.50M(  2.23) Table 5: A detailed comparison of downstream model architectures. Moreover, the scores causing the change in ranking are negligible, e.g., TERA/CPC in SS and wav2vec 2.0 Base/HuBERT Base in OOD-ASR with large . The results show that the relative performance achieved by different upstream models is agnostic to the downstream architecture, confirming the robustness of the framework used in SUPERB-SG . To study the effect of data size, we create 3 pseudo datasets per task by sub-sampling 10%, 5% and 1% from the original training set while fixing the validation and test sets. The statistics of the datasets are shown in Table 7, and the results are in Table 8. The ranking of the upstream models remains almost the same for 10% of training data. When that is further reduced to 5%, there is a change in ranking in SS due to a performance drop in Modified CPC. Excluding Modified CPC, the ranking is still fixed showing that the relative performance of the upstream models is agnostic to data size. Furthermore, when using only 1% of training data, most of the SSL models fail on the 3 downstream tasks. This phenomenon is caused by in-sufficient task-specific knowledge due to limited training data size. Although SSL models learn high-level representations from the unlabeled speech signal, acquisition of task-specific knowledge such as translingual ability in ST, text-level token mapping in OOD-ASR, and mask prediction in SS, requires non-trivial supervision. We introduce SUPERB-SG , a set of 5 new tasks that include speech translation, out-of-domain ASR, voice conversion, speech separation, and speech enhancement to evaluate the deep semantic and generative capabilities of SSL models. We evaluate 15 SSL models, and do a comprehensive analysis of the task correlations to demonstrate the reliability of our methodology. We test and confirm the robustness of SUPERB-SG in terms of the downstream model architecture as well as the training data size. The latest introduction of the semantic and generative tasks increases the diversity and difficulty of SUPERB , which can boost a more comprehensive understanding of the capability of various SSL models' representations, and help researchers discover the hidden properties of SSL techniques in development. We have open-sourced all the codes 1 and released a challenge 3 to encourage further research of SSL in speech. We welcome the community to participate and advance the research frontier together. This work fully adheres to the ACL code of ethics. For more details, we provide a checklist in Appendix B. References David lvarez et al. 2019.
abstain abstain abstain abstain objective method abstain abstain result other abstain abstain abstain abstain abstain abstain other abstain abstain abstain objective abstain result objective objective objective abstain other other other objective other other other method other other method other method method method abstain method method abstain method other method method abstain abstain method abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain method abstain abstain method method abstain method abstain abstain method method method abstain abstain abstain abstain method abstain abstain method method abstain method abstain other abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain result result result abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective method abstain method abstain abstain method