2 Department of Computing, The Hong Kong Polytechnic University 3 Department of Computer Science and Technology, Tongji University 4 Microsoft 5 Peng Cheng Laboratory Abstract Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification. We release our code and model at https://github.com/microsoft/ SpeechT5 . Starting with ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), substantial work has shown that pre-trained models can significantly improve in various natural language processing (NLP) tasks (Radford et al., 2019; CONNEAU and Lample, 2019; Yang et al., 2019; Dong et al., 2019; Lewis et al., 2020). Following the pre-training techniques in NLP, self-supervised speech representation learning has also been investigated and shown promising results, benefiting from richly learned representations (Chung and Glass, 2018; Chuang et al., 2020; Song et al., 2019; Baevski et al., 2020; Wang et al., 2021; Hsu et al., 2021; Chung et al., 2021a), such as wav2vec 2.0 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021). However, previous speech pre-training work suffers from two problems: (1) most of them learn the speech representation with only unlabeled speech data but ignore the importance of textual data to spoken language tasks (e.g., automatic speech recognition) which require the modality transformation; (2) most of these models solely rely on a pre-trained speech encoder for various downstream tasks, leaving the decoder not pre-trained for the sequence-to-sequence generation tasks. How to design a unified encoder-decoder model that can take advantage of both unlabeled speech and text data to improve various spoken language processing tasks is not well explored. we attempt to formulate each spoken language processing task as a speech/text to speech/text problem via an encoder-decoder framework, which enables us to use the same pre-trained model with bimodal data across diverse tasks, as shown in Figure 1. To achieve this, we propose a unified-modal pre-training framework, SpeechT5, containing an encoder-decoder backbone network and modal-specific pre/post-nets. With the pre-nets, the input speech/text is embedded in a shared space, and the encoder-decoder backbone network models the sequence-to-sequence conversion, from which the model-specific post-nets generate the speech/text output. Particularly, SpeechT5 is mainly pre-trained with a denoising sequence-to-sequence method by leveraging large-scale unlabeled text and speech corpus. To align the textual and acoustic information into a unified semantic space, the proposed SpeechT5 model (1) maps text and speech representations into a shared vector quantization space, and (2) randomly mixes up the quantized latent representations and the contextual states, which can better guide the quantizer to learn the cross-modal features. We fine-tune SpeechT5 on a wide variety of downstream spoken language processing tasks, including automatic speech recognition (ASR), text-to-speech (TTS), speech translation (ST), voice conversion (VC), speech enhancement (SE), and speaker identification (SID). Massive experiments show that the proposed SpeechT5 model achieves a significant improvement on these spoken language processing tasks compared with the state-of-the-art baselines. Specifically, the proposed SpeechT5 outperforms wav2vec 2.0 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021) with the BASE model on the ASR task and also performs better than the state-of-the-art voice Transformer network (Huang et al., 2021) on the VC task. Besides, SpeechT5 is significantly superior to SpeechNet (Chen et al., 2021b) and pre-trained models from SUPERB (Yang et al., 2021) and achieves the state-of-the-art performance (i.e., 96.49%) on the SID task. We further provide an empirical comparison of the pre-training tasks and modules, and the ablation study demonstrates the effectiveness of the proposed joint speech-text pre-training method. The contributions of this paper are summarized as follows. framework for various spoken language processing tasks. We propose a cross-modal vector quantization approach, which learns the implicit alignment between acoustic and textual representation with large-scale unlabeled speech and text data. Extensive experiments on spoken language processing tasks demonstrate the effectiveness and superiority of the proposed SpeechT5 model. In this section, we propose SpeechT5, a unified-modal framework for learning joint contextual representations for speech and text data via a shared encoder-decoder structure. Figure 2(a) shows the model architecture of the proposed SpeechT5 model. It consists of an encoder-decoder module and six modal-specific pre/post-nets. The pre-nets convert the input speech X s  D s or text X t  D t to a unified space of hidden representations and then feed them into the shared encoder-decoder to perform the sequence-to-sequence conversion. Finally, the post-nets generate the output in the speech or text modality, based on the decoder output. Input/Output Representations To train a single model for a diverse set of spoken language processing tasks, we formulate them as speech/text to speech/text tasks, where the model is fed with speech/text as the input and generates the corresponding output in the speech/text format. Specifically, a text is split into a sequence of characters X t = ( x t 1 , ..., x tN t ) as the input and output. For speech modality, the raw waveform X s = ( x s 1 , ..., x sN s ) is used as the input, and a sequence of the log Mel-filterbank features X f = ( x f 1 , ..., x fN f ) extracted from raw audio using librosa tool 1 is adopted as the target output. A vocoder (Kong et al., 2020) is leveraged to generate the final waveform from the generated features. refer to Vaswani et al. (2017) for more details. We employ the relative position embedding (Shaw et al., 2018) to help capture the relative position differences between elements in the input. Specifi-cally, we only add the relative position embedding to the dot-product weights of the self-attention. Speech Pre/Post-Net The convolutional feature extractor of wav2vec 2.0 (Baevski et al., 2020) serves as the speech-encoder pre-net to downsam-ple raw waveform X s and produce a sequence of a speech utterance H = ( h 1 , ..., h N h ) . The speech-decoder pre-net is a neural network composed of three fully connected layers with the ReLU activation, fed with the log Mel-filterbank X f . To support multi-speaker TTS and VC, the speaker embedding extracted with the x-vector (Snyder et al., 2018) is concatenated with the output of the speech-decoder pre-net followed by a linear layer. The speech-decoder post-net consists of two modules. The first module uses a linear layer fed with the decoder output to predict the log Mel-filterbank Y f = ( y f 1 , ..., y fN f ) , followed by five 1-dimensional convolutional layers to produce a residual to refine the predicted Y f . Another linear module is added to project the decoder output to a scalar for predicting the stop token. Text Pre/Post-Net We use shared embeddings as the text-encoder pre-net and text-decoder pre/post-nets. The pre-net transforms a token index into an embedding vector. The post-net transforms the hidden state into the probability distribution of tokens, normalized by the softmax function. The proposed SpeechT5 model can be pre-trained with large-scale collections of unlabeled speech and text corpus. The proposed joint pre-training method can align the textual and acoustic information into a unified semantic space. Speech Pre-Training Leveraging unlabeled speech data D s to learn general speech representations for both classification and generation tasks, SpeechT5 is trained with two types of tasks: bidirectional masked prediction and sequence-to-sequence generation. Following HuBERT (Hsu et al., 2021), the bidirectional masked prediction leverages a masked language model similar to BERT (Devlin et al., 2019) for the encoder, in which an acoustic unit discovery model provides the frame-level targets Z = ( z 1 , ..., z N h ) 2 . Specifically, we apply span mask strategies to the output H from speech-encoder pre-net, where 8% of timesteps are randomly selected as start indices, and spans of 10 steps are masked. The Transformer encoder takes masked H as the input and produces hidden representations U = ( u 1 , ..., u N h ) . Based on these hidden representations, the cross-entropy loss is computed over masked timesteps as L smlm = (cid:88) n M log p ( z n |  H , n ) , (1) where  H denotes the masked version of H , M 2 The target labels are generated by clustering outputs of the 6-th Transformer layer in the first iteration HuBERT BASE model via the k -means clustering method with 500 clusters. Furthermore, we propose to reconstruct the original speech via a sequence-to-sequence generation task, given the randomly masked input as introduced in bidirectional masked prediction. Following seq2seq TTS models (Li et al., 2019), we enforce the corresponding predicted output Y f , which is generated through the speech-decoder pre-net, Transformer decoder, and speech-decoder post-net, to be close to the original X f by minimizing their L 1 -distance as L s 1 = N f (cid:88) n =1 (cid:107) y fn  x fn (cid:107) 1 , (2) where x fn denotes n -th an 80-dimensional log Mel-filterbank from X f . Besides, we use the binary cross-entropy (BCE) loss L s bce for the stop token. Text Pre-Training With unlabeled text data D t , SpeechT5 is trained to reconstruct the model output Y t = ( y t 1 , ..., y tN t ) to the original text X t , using the corrupted text  X t = ( x t 1 , ...,  x tM ) as the input generated with a mask-based noising function. Following the text infilling approach in BART 3 (Lewis et al., 2020), we randomly sample 30% of text spans to mask, where the span length of text spans draws from a Poisson distribution (  = 3 . 5 ), and each span is replaced with a single mask token. Formally, SpeechT5, including text-encoder pre-net, encoder-decoder model, and text-decoder pre/post nets, is optimized to generate the original sequence with the maximum likelihood estimation as L tmle = N t (cid:88) n =1 log p ( y tn | y t<n ,  X t ) , (3) Joint Pre-Training The above pre-training methods can only leverage speech or text data to model acoustic or language information individually. To build a cross-modality mapping between speech and text, which is essential for tasks such as ASR and TTS, we propose a cross-modal vector quantization method to learn representations capturing the modality-invariant information. Specifically, we utilize vector quantized embeddings as a bridge to align the speech representation and text representation through a shared codebook, 3 We conducted experiments to compare the BART (Lewis et al., 2020) and T5 (Raffel et al., 2020) mask strategies, which can be found in Appendix A. as shown in Figure 2(b). Inspired by VQ-VAE (Oord et al., 2017) and SemFace (Ren et al., 2021), we first use the quantizer to convert these continuous speech/text representations u i from the output of the encoder into discrete representations c i from a fixed-size codebook CK , which contains K learnable embeddings. Then, the nearest neighbor search is performed between the encoder output and the embedding of each latent code via the L 2 distance as c i = arg min j  [ K ] (cid:107) u i  c j (cid:107) 2 , (4) where c j is the j -th quantized vector in the codebook. Note that we do the same operation for the output of the speech and text encoders with a shared codebook. Then, we randomly replace a proportion (10%) of the contextual representations with quantized latent representations in the corresponding time steps and calculate the cross-attention upon the mixed representations, which can explicitly guide the quantizer to utilize the cross-modal information. The diversity loss is used to encourage sharing more codes by maximizing the entropy of the averaged Softmax distribution as L d = 1 KK (cid:88) k =1 p k log p k , (5) where p k is the averaged probability of choosing the k -th code in the codebook. The final pre-training loss with unlabeled speech and text data can be formulated as L = L smlm + L s 1 + L sbce + L tmle +  L d . (6) where  is set to 0 . 1 during pre-training. After pre-training, we fine-tune the encoder-decoder backbone via the loss of the downstream task. The goal is to measure the learning abilities of SpeechT5, and we study the performance on a diverse set of downstream tasks such as ASR, TTS, ST, VC, SE, and SID. All of the spoken language processing tasks that we consider can be learned by concatenating the outputs of the encoder-decoder backbone and corresponding pre-net and post-net. Taking ASR as an example, the final model consists of the speech-encoder pre-net, encoder-decoder, text-decoder pre-net, and text-decoder post-net, 5726 Model LM dev-clean dev-other test-clean test-other wav2vec 2.0 BASE (Baevski et al., 2020) -6.1 13.5 6.1 13.3 HuBERT BASE (Hsu et al., 2021)  -5.5 13.1 5.8 13.3 Baseline (w/o CTC) -5.8 12.3 6.2 12.3 Baseline -4.9 11.7 5.0 11.9 SpeechT5 (w/o CTC) -5.4 10.7 5.8 10.7 SpeechT5 -4.3 10.3 4.4 10.4 DiscreteBERT (Baevski et al., 2019) 4-gram 4.0 10.9 4.5 12.1 wav2vec 2.0 BASE (Baevski et al., 2020) 4-gram 2.7 7.9 3.4 8.0 HuBERT BASE (Hsu et al., 2021) 4-gram 2.7 7.8 3.4 8.1 wav2vec 2.0 BASE (Baevski et al., 2020) Transf. which are initialized by SpeechT5 and fine-tuned via the cross-entropy loss on the corresponding training data. The baseline systems have the same architecture as SpeechT5, but the weights of the baseline encoder are initialized by the HuBERT BASE model (Hsu et al., 2021) if the input data of the downstream tasks is speech. It allows raw waveform as the model input and can provide a strong baseline. All models are implemented in Fairseq 4 (Ott et al., 2019). The encoder-decoder backbone contains 12 Transformer encoder blocks and 6 Transformer decoder blocks, where the model dimension is 768, the inner dimension (FFN) is 3,072, and the number of attention heads is 12. The above encoder setting is the same as that in wav2vec 2.0 BASE and HuBERT BASE . The speech-encoder pre-net contains 7 blocks of temporal convolutions, each of which is composed of 512 channels with strides (5 , 2 , 2 , 2 , 2 , 2 , 2) and kernel sizes (10 , 3 , 3 , 3 , 3 , 2 , 2) . For the speech-decoder pre-net and post-net, we use the same setting as the pre-net and post-net in Shen et al. (2018) except that the number of channels of the post-net is 256. For text-encoder/decoder pre/post-net, a shared embedding layer with dimension 768 is used. For the vector quantization, we use two codebooks with 100 entries for the shared codebook module, resulting in a theoretical maximum of K = 10 4 code entries. For text pre-training, we use the normalized language model training text of LibriSpeech as unlabeled data, which contains 400M sentences. 5 We optimize the model with Adam (Kingma and Ba, 2014) by warming up the learning rate for the first 8% of updates to a peak of 2  10  4 , which is linear decayed for the following updates. We pre-train the proposed SpeechT5 model on 32 V100 GPUs with a batch size of around 90s samples per GPU for speech and 12k tokens per GPU for text and set the update frequency to 2 for 500k steps. We fine-tune the ASR model with the LibriSpeech 100/960 hours data and train the language model (LM) with the LibriSpeech LM text data, which is used for shallow fusion (Gulcehre et al., 2015) during the ASR inference. Besides the cross-entropy loss for the decoder, we add an extra linear layer to calculate the connectionist temporal classification (CTC) loss on the top of the encoder (Watan-abe et al., 2017), so that we can apply the joint CTC/attention decoding (Hori et al., 2017) to boost the performance. We measure the performance of ASR by the word error rate (WER). The implementation details can be found in Appendix B.1. The results of ASR on the 100 hours set of LibriSpeech are reported in Table 1. We compare with several state-of-the-art self-supervised approaches, including DiscreteBERT (Baevski et al., 2019), wav2vec 2.0 (Baevski et al., 2020), and HuBERT (Hsu et al., 2021). Without LM fusion, the baseline outperforms wav2vec 2.0 BASE and HuBERT BASE with the help of the joint CTC/attention decoding, which shows the importance of the decoder. The proposed SpeechT5 model achieves significant improvements on all settings compared to wav2vec 2.0 BASE , HuBERT BASE and our strong baselines, demonstrating the superiority of the proposed pre-training method. Furthermore, when decoding with LM fusion, SpeechT5 obtains the lower WERs than wav2vec 2.0 BASE on all sets and achieves the state-of-the-art performance. Due to space constraints, the results of 960h fine-tuning experiments are reported in Appendix C. 3.3 Evaluation on TTS We fine-tune the pre-trained model on the 460-hours LibriTTS clean sets (Zen et al., 2019) with the L 1 loss, L sbce loss, and attention loss (Tachibana et al., 2018). We utilize the HiFi-GAN (Kong et al., 2020) vocoder to convert the log Mel-filterbank to the raw waveform. We evaluate the Naturalness with the open-source NISQA-TTS (Mittag and Mller, 2020), the mean option score (MOS), and the comparison mean option score (CMOS) by native speakers on the randomly selected 200 sentences with various lengths (no overlapping with training data) generated by different models, in which case we keep the text content consistent. More details can be found in Appendix B.2. Table 3 shows the experimental results of TTS. The proposed SpeechT5 trained without L smlm is considered because the bidirectional masked prediction loss is proposed to help the encoder learn to encode the speech signal, and this variant achieves superior Naturalness, as shown in Table 13 (in Appendix D). The proposed SpeechT5 model behaves better than baseline and achieves a performance of 2.91 Naturalness and 3.65 MOS. Furthermore, our proposed SpeechT5 obtains a gain of +0.29 in CMOS with respect to the baseline model, which suggests the proposed pre-training method significantly improves the speech generation quality. We evaluate the ST task on the MUST-C dataset (Di Gangi et al., 2019), including English-German (EN-DE) and English-French (EN-FR) translation tasks. We use the default training setting of speech translation in Fairseq ST (Wang et al., 2020), and we also average the last 10 checkpoints and use a beam size of 5 for decoding. Translation results are evaluated with case-sensitive BLEU (Papineni et al., 2002). Details about the dataset and fine-tune setting are introduced in Appendix B.3. We list the BLEU scores of ST in Table 4. The result of SpeechT5 without initializing the decoder is also reported since we do not pre-train the decoder with German or French data, and it outperforms the strong baseline whose encoder is initialized by HuBERT encoder. The proposed SpeechT5 further beats the SpeechT5 without initializing the decoder, and achieves a significant improvement of 1.75 and 1.54 BLEU scores than baseline in EN-DE and EN-FR tasks, respectively, which demonstrates the effectiveness and superiority of our method. Besides, our SpeechT5 model outperforms existing models such as Fairseq ST (Wang et al., 2020), 5728 ESPnet ST (Inaguma et al., 2020), and Adapter Tuning (Le et al., 2021) that employs adapter modules to be further specialized in each language pair from different pre-trained models. VC aims to convert a speaker-dependent source speech waveform into a different one while preserving linguistic information of the source speech waveform. We follow the many-to-many setting and utilize speech recordings of four speakers in the CMU Arctic (Kominek and Black, 2004), including clb, bdl, slt, and rms. For the waveform synthesis, we use the Parallel WaveGAN (Yamamoto et al., 2020), a non-autoregressive variant of the WaveNet vocoder. We employ the average of MCD (Mel-Cepstral Distortion) and WER as the metrics for the VC task. More details about the dataset and fine-tune setting are given in Appendix B.4. We show the results of VC in Table 2, where we list the conversion from speaker bdl to slt and clb to slt as used in the voice Transformer network (VTN) (Huang et al., 2021). The experimental results demonstrate that the proposed SpeechT5 model achieves a significant gain than the strong baseline model. The proposed SpeechT5 model also outperforms the state-of-the-art VTN variants in terms of MCD, including VTN fine-tuned from ASR or TTS (Huang et al., 2021) and many-to-many VTN (Kameoka et al., 2021). SE is the task of removing background noise from a degraded speech signal and improving the intelligibility and the perceived quality of the signal. We use the WSJ0 Hipster Ambient Mixtures (WHAM!) dataset (Wichern et al., 2019) and conduct the 16 kHz max enhance-single task that recovers the signal from a mixture of only the first WSJ0 speaker and noise. We utilize HiFi-GAN to transform the log Mel-filterbank to the raw waveform. Since the input and output lengths are probably different in the encoder-decoder model, we can not evaluate it by PESQ (Rix et al., 2001) and ESTOI (Jensen and Taal, 2016), so we evaluate the negative impact on the ASR performance by WER. The implementation details of SE are in Appendix B.5. As shown in Table 5, our strong baseline model recovers contents from the noisy speech, achieving 10.9% WER from 76.1% WER. Moreover, the proposed SpeechT5 model gets a relative 9% WER reduction compared to the strong baseline model. The results suggest that although the noisy speech with WHAM! is challenging as summarized in Table 12 (in Appendix B.5), the proposed encoder-decoder framework can effectively suppress the noise and recover the content. We convert SID, a multi-class classification task of classifying each utterance for its speaker identity, to a speech to text task by sequence to sequence model. Compared to the ASR task, the text embedding table is replaced by a speaker embedding table, and the decoder predicts speaker identifies at the first step. We adopt the VoxCeleb1 dataset (Nagrani et al., 2017), which contains over 100,000 speech records uttered by 1,251 celebrities extracted from videos uploaded to YouTube. The top-1 speaker classification accuracy (ACC) is used as the evaluation metric of SID. Refer to Appendix B.6 for more details about the dataset and fine-tuning. As shown in Table 6, our baseline is superior to existing Transformer-based methods such as SpeechNet (Chen et al., 2021b) and pre-trained models from SUPERB (Yang et al., 2021). Moreover, it outperforms ResNet-based architectures such as Thin ResNet-34 (Chung et al., 2020), indicating the superiority of the encoder-decoder ar-5729 chitecture for the SID task. The SpeechT5 further improves the performance compared to baseline and achieves the state-of-the-art performance (i.e., 96.49% accuracy), which demonstrates the effectiveness of the proposed pre-training technique. To better understand why the proposed SpeechT5 model is effective, we investigate the influence of the pre-training methods by removing each of them independently. As shown in Table 7, we can draw the following conclusions: (1) The pre-training methods, including speech pre-training, text pre-training, and joint pre-training method, are important to SpeechT5 since without each of them, the performance of all tasks will degrade significantly; (2) Speech pretraining is more critical than text pre-training on these tasks that need to encode speech, and the ASR model fine-tuned from SpeechT5 without speech pre-training even can not converge; (3) Without the joint pre-training method, the performance of the ASR model decreases, which demonstrates that the learned alignment from joint pre-training brings benefits for cross-modality tasks; (4) The masked language model learning L smlm of speech data is mainly responsible for extracting acoustic features and learning better speech representation, which is beneficial to ASR and SID tasks. Large-scale pre-training models such as BERT (De-vlin et al., 2019), T5 (Raffel et al., 2020), wav2vec 2.0 (Baevski et al., 2020), and HuBERT (Hsu et al., 2021) have drawn much attention in the NLP and speech communities, due to its strong capability capability of generalization and efficient usage of large-scale data (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lewis et al., 2020; Chen et al., 2021c; Baevski et al., 2020; Lakhotia et al., 2021; Kharitonov et al., 2021; Chen et al., 2021a). However, the research mentioned above effects gear towards single-modal learning, hence they can only be used in either text or speech modeling. Although some speech-language pre-training work (Chung et al., 2021b; Kim et al., 2021; Qian et al., 2021) attempts to improve spoken language understanding tasks, these methods only focus on an encoder with task-specific layers for different tasks and do not pre-train a decoder for generation tasks such as speech synthesis or text generation. Besides, a series of research work begins to investigate joint text and speech training (Han et al., 2021; Ye et al., 2021; Tang et al., 2021a; Zheng et al., 2021; Tang et al., 2021b), but they are mainly designed for speech to text tasks. The proposed SpeechT5 method is most related to T5 (Raffel et al., 2020). The core idea of the T5 model, a unified framework for a variety of text-based language problems, is to treat every text processing problem as a text-to-text problem. SpeechT5 is also related to Speech Chain (Tjandra et al., 2020), which leverages the ASR model and TTS model to build a closed-loop machine speech chain to train models on the concatenation of both labeled and unlabeled data, and SpeechNet (Chen et al., 2021b), which designs a universal modularized model to perform multiple speech processing tasks with multi-task learning. The differences from the above models are that (1) SpeechT5 is a shared cross-modal encoder-decoder framework, whose input and output are speech or text through multiple pre/post-nets; (2) SpeechT5 attempts to pre-train and improve the universal model with large-scale unlabeled text and speech data. Another related work is SUPERB (Yang et al., 2021), a benchmark to examine the capability of pre-trained models such as HuBERT (Hsu et al., 2021). SUPERB focuses on investigating a simple framework to learn SUPERB tasks with a frozen and shared pre-trained encoder and lightweight prediction modules fine-tuned for each task. In contrast, the goal of SpeechT5 is to learn all spoken language processing tasks by fine-tuning a unified-modal encoder-decoder model, which is pre-trained on unlabeled speech and text corpus. In this paper, we have proposed SpeechT5 as a pre-trained encoder-decoder model for various spoken language processing tasks. We convert all spoken language processing tasks into a speech/text to speech/text format and propose a novel joint pretraining method to utilize cross-modal information by leveraging the unlabeled speech and text data. The proposed unified encoder-decoder model can support generation tasks such as speech translation and voice conversion. Massive experiments show that SpeechT5 significantly outperforms all baselines in several spoken language processing tasks. In the future, we are going to pre-train the SpeechT5 with a larger model and more unlabeled data. We are also interested in extending the proposed SpeechT5 framework to address multilingual spoken language processing tasks for future work. We thank Yanqing Liu and Sheng Zhao for their help in TTS human evaluation. We also want to thank the anonymous reviewers for insightful comments and suggestions.
abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain other other abstain result objective abstain abstain abstain abstain abstain abstain abstain objective objective objective objective objective abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain other abstain method method abstain method other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other other other other other other other other other other other other other objective objective abstain abstain method objective other other