We describe a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. The proposed method incorporates four self-supervised and supervised subtasks for cross modality learning. A self-supervised speech subtask leverages unlabelled speech data, and a (self-)supervised text to text subtask makes use of abundant text training data. Two auxiliary supervised speech tasks are included to unify speech and text modeling space. Our contribution lies in integrating linguistic information from the text corpus into the speech pre-training. Detailed analysis reveals learning interference among subtasks. Two pre-training configurations for speech translation and recognition, respectively, are presented to alleviate subtask interference. Our experiments show the proposed method can effectively fuse speech and text information into one model. It achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MUST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the LIBRISPEECH speech recognition task. 1 1 Introduction Pre-training can learn universal feature representations from a large training corpus and is benefi-cial for downstream tasks with limited amounts of training data (Peters et al., 2018; van den Oord et al., 2018; Chung et al., 2018; Zoph et al., 2020). With the advancement of computational power and self-supervised pre-training approaches, large volumes of unlabeled data may now be used in pre-training. Methods, such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020b) and wav2vec2.0 (Baevski et al., 2020b), have emerged as the backbone of many speech and natural language processing tasks. 1 https://github.com/pytorch/fairseq/tree/main/ examples/speech text joint to text. The aforementioned pre-training methods focus on learning feature representation either from text or speech. Many speech applications combine information learnt from both speech and text corpora to achieve state of the art results. In speech processing, transcribed speech training data is generally very scarce for many languages. It is difficult to build robust linguistic knowledge representation solely based on labeled speech training data. Jia et al. (2019); Chen et al. (2021) propose to generate synthetic data from text to augment speech training corpus. Li et al. (2021) demonstrate that models initialized with pre-trained wav2vec2.0 and mBART (Liu et al., 2020) modules are competitive for the multilingual speech to text translation task. Chuang et al. (2020) propose to concatenate the acoustic model and BERT model for speech Q&A. Chung et al. (2021b) align speech utterance representation to the corresponding text sentence representation, in which both representations are generated from unsupervised pre-trained models, for speech understanding. In this study, we are interested in pre-training for speech to text tasks using the Attention based Encoder-Decoder (AED) framework. In particular, we seek to answer the question whether the integration of data from different modalities is ben-eficial for representation learning. To answer this question, we propose Speech and Text joint PreTraining (STPT), a multi-task learning framework to combine different modalities, i.e., speech and text, in the pre-training stage. A self-supervised speech subtask and a (self-)supervised text to text subtask dominate the pre-training computation to leverage large amounts of unlabelled speech data and abundant text training corpus. Two auxiliary supervised speech subtasks are used to unify different modalities in the same modeling space. The proposed method fuses information from the text and speech training corpus into a single model, and it effectively improves the performance of down-1488 stream tasks, such as speech to text translation (ST) and automatic speech recognition (ASR). Our contributions are summarized as follows: 1. We propose a multi-task learning framework to learn four speech and text subtasks in one model and successfully integrate linguistic information from the text corpus into the speech pre-training. 2. We conduct detailed analyses on the proposed pre-training method, which reveal the interference among different subtasks. 3. Two joint pre-training configurations are proposed to alleviate learning interference for ASR and ST respectively. 4. State-of-the-art results are achieved on the downstream tasks. We obtain at least 1.7 BLEU improvement compared with the best MUST-C ST system reported and comparable WERs as wav2vec 2.0 in the LIBRISPEECHASR task. Pre-training : Self-supervised pre-training is usually optimized with two different criteria: contrastive loss (van den Oord et al., 2018; Chung and Glass, 2020; Baevski et al., 2020b) and masked prediction loss (Devlin et al., 2019). Contrastive loss focuses on distinguishing the positive samples from the negative ones given the reference sample and it has achieved great success for speech recognition (Baevski et al., 2020b). Masked prediction loss has been first studied for natural language processing tasks (Devlin et al., 2019; Lewis et al., 2020b) with subsequent application to speech processing (Baevski et al., 2020a; Hsu et al., 2021). Chung et al. (2021a) combine contrastive loss and masked prediction loss, which shows good performance for the downstream ASR task. The optimization of our self-supervised speech task is more related to the masked prediction loss. Instead of predicting the hard discretized label for the masked frames, which is error prone, we use KL divergence to minimize the distribution difference between the same feature frames with and without masking. Please refer to subsection 3.2 for more details. Self-training (or iterative pseudo labelling): self-training is another widely used approach to take advantage of unlabelled speech data to improve the ASR performance (Kahn et al., 2020; Xu et al., 2020; Pino et al., 2020; Zhang et al., 2020; Wang et al., 2021a; Xiao et al., 2021; Wang et al., 2021b). A seed model, which usually is trained with a small amount of supervised speech training data, is employed to generate pseudo labels for the unlabelled speech data. The speech data with pseudo labels is augmented into the training dataset to build another model, which is expected to outperform the seed model due to more training data exposure. Similar to self-training, we also use small amounts of supervised data to unify the speech and text modeling space. However, the self-supervised speech training in this work avoids making hard predictions and uses KL divergence to maximize the mutual information between masked span and observed feature frames. Multi-task learning : Due to data scarcity, multitask learning is widely adopted to leverage parallel text training data for ST (Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Tang et al., 2021b; Ye et al., 2021). Those methods primarily use supervised speech data sets during multi-task learning, whereas our method can leverage large amounts of unlabeled speech data during the pre-training stage, which has the potential to improve performance even further. A concurrent work from Ao et al. (2021) also proposes to jointly pre-train speech and text for ASR and text to speech application, which is fully unsupervised. Our method focuses on taking advantage of the supervised speech data, which could be the same data used for fine-tuning, to improve the joint speech text pre-training. Our results demonstrate the efficacy of supervised speech data in pretraining. Another concurrent work is from Bapna et al. (2021), which focuses on speech encoder pre-training using both speech and text data. Our method emphasizes the encoder-decoder framework and training both encoder and decoder in the pre-training stage. ASR and ST are the two main downstream tasks for the proposed pre-training method. Figure 1 depicts our joint pre-training framework, which consists of four subtasks: 1. (Self-)supervised T ext to T ext subtask (T2T) 2. S elf-supervised S peech L earning subtask (SSL) 1489 3. Supervised S peech to P honeme classification subtask (S2P) 4. Supervised AED based S peech to T ext subtask, which is the same as the downstream task, i.e., ST or ASR (S2T) The choice of the T2T subtask depends on the downstream task. For ASR, the T2T subtask is a denoising autoencoder task (BART) (Lewis et al., 2020a) while ST utilizes a text based neural machine translation task. The SSL subtask is a self-supervised speech learning task to leverage large amounts of unlabelled speech data optimized by the masked prediction loss. The last two supervised speech tasks (S2P and S2T) unify two modalities, i.e., speech and text, into one modeling space. In this study, we find that the subtasks for the ASR pre-training are complementary, while subtask interference is observed in the ST pre-training at some encoder layers. We propose two different configurations: fully shared encoder (FSE) (Fig-ure 1(a)) for the ASR pre-training, and partially shared encoder (PSE) (Figure 1(b)) for the ST pre-training. The FSE configuration aims to encourage information sharing between different subtasks while the PSE configuration tries to minimize the information sharing between encoder only subtasks, i.e., SSL and S2P, and sequence to sequence AED tasks, i.e., subtask T2T and S2T. More subtask interference analysis is presented in subsection 5.2. We describe the details of each subtask in the following subsections. In the sequence to sequence ASR and ST tasks, the decoder is a text generator conditioned on the encoder outputs. Large amounts of training samples are required to cover different linguistic aspects of the target language. Abundant text is an ideal supplement to the limited supervised speech data corpus. Assume the target text sequence is Y = ( y 1 , y 2 ,    , y N ) , its corresponding corrupted version, X = NOISE ( Y ) = ( x 1 , x 2 ,    , x M ) , can be created by masking or replacing token spans in Y (Lewis et al., 2020a) for the ASR pre-training. If the downstream task is ST, X is the corresponding source token sequence. The task is optimized by maximizing cross entropy L T2T =  N (cid:88) i log p ( y i | y 1: i  1 , X ) (1) In this subtask, we also convert the input text into the corresponding pronunciation form, i.e., phoneme sequence, as it would be easier to align the encoder outputs from speech and text (Tang et al., 2021b). The purple and black lines in Figure 1 describe the data flow in the T2T subtask. The SSL subtask aims to leverage vast amounts of unlabelled speech data and learn general speech representations. The model configuration follows wav2vec2.0 (Baevski et al., 2020b) where the speech model includes a feature extractor and a 1490 context encoder. The context encoder corresponds to the speech encoder in Figure 1(b) in the ST pre-training. If ASR is the downstream task, the context encoder includes one extra shared encoder as shown in Figure 1(a). We use different frameworks for the ST and ASR pre-training to reduce interference among subtasks. The detailed subtask interference is discussed in subsection 5.2. We propose a masked KL divergence loss to optimize the SSL subtask. It consists of two-pass computation. Given the speech input S = ( s 1 , s 2 ,    , s T ) , the feature extractor and context encoder outputs are Z = ( z 1 , z 2 ,    , z T (cid:48) ) and O = ( o 1 , o 2 ,    , o T (cid:48) ) respectively, where the speech input is down-sampled by the feature extractor and T > T (cid:48) . In the first pass, the output O is compared with the phoneme embedding E = ( e 1 , e 2 ,    , e I ) , which is from the T2T subtask described in subsection 3.1. I is the phoneme vocabulary size. The predicted phoneme distribution p ( o j | e i ) is defined as p ( o j | e i ) = exp( o j (cid:124)  e i ) (cid:80) i (cid:48) exp( o (cid:124) j  e i (cid:48) ) (2) In the second pass, speech feature spans  Z  Z are selected and corrupted as wav2vec2.0 (Baevski et al., 2020b). O is the corresponding context encoder output from  Z . We train the model to infer the corrupted p ( o j | e i ) being similar as p ( o j | e i ) by minimizing KL divergence. Compared with the masked prediction loss, instead of predicting the hard discretized label for the masked frames, we use the soft label prediction, i.e., predicted phoneme distribution from the first pass, to learn speech representation and avoid the hard prediction errors. The S2P subtask is employed to unify the self-supervised trained speech and text models. It shares the same model as in the SSL subtask. In this subtask, a transcribed ASR data set is used and the goal of this task is to predict the frame level phoneme labels. A HMM-GMM model is trained with the same transcribed dataset using Kaldi (Povey et al., 2011) to generate the frame-level labels with forced-alignment. where a ( j ) is the phoneme label associated with the context encoder output o j . The data flow in the S2P subtask is depicted with steelblue lines in Figure 1. 3.4 Supervised AED based speech to text subtask Besides the S2P subtask mentioned in the previous subsection, we include the potential downstream AED based task, i.e. ASR or ST, as another auxiliary subtask during the pre-training stage. In many speech translation datasets, such as MuST-C (Gangi et al., 2019) or CoVoST (Wang et al., 2020), we have both speech transcription and translation labels. The speech transcription is used in the S2P subtask while the S2T subtask can make use of the corresponding translation labels. We hope this auxiliary task would make the transition from pre-training to fine-tuning smooth and result in better performance in downstream tasks. The components involved during optimization are connected with blue lines in encoder and black lines in decoder as shown in Figure 1. They are trained with cross entropy criterion, L S2T =  (cid:88) t log p ( y i | y i  1 , O ) (5) where O is the input speech and Y = ( y 1 ,    , y N ) is the target labels. The overall pre-training loss is defined as the combination of four losses discussed above L = L T2T +  LSSL +  L S2P +  L S2T (6) where  ,  and  are task weights for the SSL, S2P and S2T subtasks respectively. During the pre-training, the shared encoder inputs come from two sources, either from speech encoder outputs in the S2T subtask or phoneme embeddings in the T2T subtask. The shared encoder inputs might be in different numerical scales. In order to stabilize the multi-task training, a Lay-erNorm (Ba et al., 2016) is applied to the shared encoder inputs and places those inputs in the same numerical scale as shown in Figure 1. 1491 4 Experimental setting In the pre-training, we first train modules with the T2T subtask until they are converged. It helps to stabilize the training and achieve a better result. Then the entire model is jointly optimized with all subtasks mentioned in section 3. Finally, the pre-trained model is fine-tuned on the downstream tasks. In the fine-tuning stage, we keep optimizing the model with the T2T and S2T subtasks. Two encoder-only subtasks (SSL and S2P) are dropped, since the model has learnt good speech representation from the unlabeled speech data in pre-training. Two downstream tasks, ASR and ST, are examined. The ASR system is evaluated on four LIBRISPEECH (Panayotov et al., 2015) evaluation sets: dev-clean, dev-other, test-clean and test-other. WER is reported in the experiments. ST models are evaluated on two translation directions: English-Spanish (EN-ES) and English-French (EN-FR). Case-sensitive detokenized SACREBLEU (Post, 2018) is reported on the tst-COMMON testset from MUST-C (Gangi et al., 2019). For both ASR and ST pre-training, 60k hours of unlabelled English speech data from Librilight (Kahn et al., 2020) is used to build the self-supervised speech task if not specifically mentioned. We employ the same labelled data for the supervised learning in pre-training and fine-tuning, i.e., LIBRISPEECH training data for ASR and MUST-C for ST. For ASR pre-training, the LIBRISPEECH language model (LM) training dataset is used to build the monolingual BART model. For ST pre-training, we take the parallel training corpus from WMT. More details about the training data could be found in Appendix A. 4.1 Model configuration The model takes raw speech audio as input. The feature encoder contains seven blocks and the temporal convolutions in each block have 512 channels with strides (5,2,2,2,2,2,2) and kernel widths (10,3,3,3,3,2,2). The speech encoder, shared encoder and shared decoder are all with 6 transformer layers, model dimension 768, inner dimension (FFN) 3,072 and 8 attention heads. We adopt Pre-LN in the transformer block as Xiong et al. (2020). The total number of parameters is 169 million. The task weight for each subtask is set by the number of mini-batches used during training. In the pre-training, the ratio of mini-batch numbers for each subtasks are 1.0, 7.0, 0.5 and 0.5 for the T2T, SSL, S2P and S2T subtasks respectively. We mask 30% tokens in the T2T BART subtask in ASR pre-training, and no masking is applied for the T2T NMT subtask in the ST pre-training. 7% of the feature frames in the SSL subtask and 3% of the feature frames in the two supervised speech subtasks are randomly selected as the mask span starting time-step. The mask span length is 10. The masking percentage is selected via grid search ( (20 , 30) for text masking, (6 , 6 . 5 , 7) and (2 , 3) for speech masking). Additional experimental details such as optimization hyper-parameters are included in Appendix B. 5 Experimental results 5.1 Main results We present the LIBRISPEECH recognition results in Table 1. Recognition results without/with an decoding LM are reported. The WERs obtained with LM are displayed within (). The second column shows the dataset used as unlabeled data in pre-training. LS-960 stands for LIBRISPEECH training dataset and LV-60k is the 60,000 hours Librilight dataset. The decoding LM is built with the LIBRISPEECH text training corpus , which is the text corpus used by the T2T subtask in the ASR pre-training and fine-tuning. The first part of the table shows results from the wav2vec 2.0 base model, which is a CTC based ASR system. Second part of the table presents results from two AED based ASR systems, and we mainly compare the proposed method with those two AED systems. LAS is a LSTM based system trained with the LIBRISPEECH data only. Transformer (Tang et al., 2021b) is based on multi-task learning and jointly trained with a text task. The results from STPT models are presented in the third part of the table. The fourth row shows results from a model that uses 960 hours LIBRISPEECH training data as the unlabelled pretraining data while the model in the fifth row is pre-trained with the 60k hours Librilight data. STPT outperforms all previous reported AED-based systems. On average, there is a 1.2 absolute WER reduction obtained compared to the jointly trained transformer model (Tang et al., 2021b). STPT also reduces 2.2 WER compared with the CTC based wav2vec model if no external LM is applied and achieves comparable WERs when it is decoded with a LM. One interesting observation is the decoding LM is not very helpful for the STPT model, 1492 Data set Unlabeled Dev Test ave. data clean other clean other wav2vec 2.0 (Baevski et al., 2020b) LS-960 3.2 (1.8) 8.9 (4.7) 3.4 (2.1) 8.5 (4.8) 6.0 (3.4) LAS (Park et al., 2019) --2.8 (2.5) 6.8 (5.8) Transformer (Tang et al., 2021b) -2.8 7.0 3.1 7.2 5.0 STPT LS-960 2.1 (1.9) 5.4 (5.2) 2.3 (2.2) 5.6 (5.3) 3.8 (3.6) STPT LV-60k 2.0 (2.1) 4.4 (4.2) 2.1 (2.1) 4.6 (4.5) 3.3 (3.2) Table 1: WER results on Librispeech. that only 0.2 WER reduction is observed when a decoding LM is applied. Other systems, on the other hand, show a considerable WER reduction when the LM is applied during decoding. It indicates that our multi-task learning in the pre-training and fine-tuning stages can effectively fuse linguistic information in the text data corpus into the ASR model. LM might not be required if it is trained on the same text corpus. We also report results from model pre-trained with 60k hours Librilight data at the fifth row. Compared with the LS-960 STPT model, Librilight data helps to reduce the WER in two difficult other datasets. In the following experiments, we will use Librilight as unlabelled data in pre-training. In Table 2, we present the speech translation results on the MuST-C datasets. Row one to four are the latest results from literature. Row one shows the results by training a speech to text translation task alone. Row two and three present results from two multi-task systems with speech and text jointly trained together. Row four is the best system reported, which is initialized with the pre-trained wav2vec 2.0 and machine translation model, then fine-tuned with the joint speech and text training. Our method achieves 2.3 and 1.7 more BLEU scores for EN-ES and EN-FR translation directions compared with the best system (Ye et al., 2021). Interference among subtasks may impede the progress of multi-task learning and lead to inferior results. In this study, we examine the task interference via comparing the gradient similarity between pair subtasks. We choose the pre-trained models using the FSE configuration discussed in section 3 and accumulate gradients from one of four jointly trained subtasks. We prepare 20 batches of training samples for each subtask, and retrieve the accumulated gradients by sending these batches to the models. Then we calculate the pairwise cosine similarity between gradients from any two subtasks. The pairwise subtask gradient similarity from the shared encoder are presented in Figure 2. The Figure 2(a) demonstrates the gradient similarity in ASR pre-training. In most layers, the gradient similarities are small. No serious gradient interference is observed. The Figure 2(b) depicts the gradient similarity from the ST pre-training. Compared with the ASR pre-training, the S2T and T2T subtasks are replaced by speech translation and text based neural machine translation subtasks in pre-training. The interference between different subtasks is significant as large positive and negative gradient similarities are observed in the third and fifth layers in Figure 2. Similarly, we compare task gradients in the speech encoder and no obvious task interference is observed within the speech encoder for both ASR and ST pre-training. Detailed analysis on the speech encoder is included in the Appendix C. In order to alleviate the task interference, the PSE configuration is proposed for the ST pretraining. Table 3 presents the performance comparison between two configurations on both ASR and ST pre-training. On the left part of the table, we list the ASR results using 100 hours labelled speech data (train-clean-100) in pre-training and fine-tuning. While the right part of the table shows the BLEU evaluated on the MUST-C dataset. As we expected, the FSE configuration encourages information sharing among tasks and it achieves lower WER for the ASR task. It indicates subtasks in the ASR pre-training are complementary to each other. On the other hand, the PSE configuration 1493 -0.4 -0.2 0 0.2 0.4 layer 0 layer 1 layer 2 layer 3 layer 4 layer 5 unsup_speech-sup_speech unsup_speech-sup_s2s unsup_speech-text sup_speech-sup_s2s sup_speech-text sup_s2s-text minimizes the information sharing between AED subtasks and encoder only subtasks, and it leads to higher BLEU for the ST task. The supervised speech data connects the text and speech modeling and unifies the representation from different modalities. An interesting question we want to investigate is how much supervised data is enough to learn a good cross modality representation. In this experiment, we choose different amounts of labelled data for ASR pre-training and fine-tuning, varied from 960 hours (the full dataset), 100 hours (train-clean-100) and 10 hours as (Kahn et al., 2020), to answer this question. In Table 4, the first column shows the amounts of supervised speech data available during the pretraining and the second column presents the amount of labelled data used in the fine-tuning stage. In pre-training, the same supervised speech data is used in the S2P and S2T subtasks. The first observation is that more supervised speech data in the pre-training stage is always helpful to get smaller WER. For example, if the models are fine-tuned with the full LIBRISPEECH training dataset, the average WER are 3.3 (row one), 3.6 (row two) and 4.0 (row four) for experiments with 960, 100 and 10 hours labelled data in the pre-training stage. The second observation is that PT (h) FT (h) Dev Test clean other clean other 960 960 2.0 4.4 2.1 4.6 100 960 2.3 4.9 2.2 5.1 100 3.2 6.8 3.5 7.2 10 960 2.7 5.3 2.8 5.3 100 3.8 7.8 4.0 7.7 10 19.9 27.5 22.0 28.8 Table 4: Impact of the amounts of supervised data. we are still able to obtain good speech representations even with small amounts of labelled data. In row four, the model is pre-trained with 10 hours labelled data, then fine-tuned with 960 hours supervised speech data. It can achieve an average 4.0 WER, which is better than the results of the AED systems in Table 1. However, we also notice the performance degrades quickly if only small amounts of labelled speech data are available. The average WER is increased to 24.6 (row six) when only 10 hours of supervised speech data is employed in both pre-training and fine-tuning. Another question we are interested is the generalizability of the pre-trained model. There are two data partitions in LIBRISPEECH : clean and other. The clean partition is supposed to be higher recording quality and with accents closer to US English while the other partition is difficult speakers with high WER (Panayotov et al., 2015). We create four data partitions for pre-training and fine-tuning to simulate the mismatched training conditions. train-clean-100 is used as the pretraining clean data set (PT C) and the first 30,000 utterance from train-clean-360 as the finetuning clean dataset (FT C). The first 30,000 ut-1494 FT C FT O clean other clean other PT C 3.0 6.7 3.2 5.9 PT O 3.0 5.9 3.2 5.8 Table 5: WER comparison under mismatched pretraining and fine-tuning conditions. terances and the following 30,000 utterances from train-other are used as the pre-training (PT O) and fine-tuning other (FT O) datasets. Each dataset includes approximately 100 hours speech data. In Table 5, models are trained under 4 different combinations with different supervised pretraining and fine-tuning data sets. We report average WER on the dev-clean and test-clean test sets as clean, and average WER on thedev-other and test-other as other to reduce the result variation. From Table 5, we have following observations. 1) a model achieves the best results on the matched condition. The model PT C + FT C achieves the lowest WER on the clean set while PT O + FT O achieves the best results on the other set. 2) training and test on totally different conditions could increase WER significantly. The model PT C + FT C increases 0.9 WER on the other set compared with the PT O + FT O model. 3) mismatched pre-training and fine-tuning might slightly increase the WER, 0.1 to 0.2 in this experiment. In the SSL subtask, we optimize the model to reduce the KL divergence loss between input without masking and with masking as described in subsection 3.2. It is a variant of the masked prediction loss (Baevski et al., 2020a) and no target labels are required in our implementation. Contrastive loss is another widely used method for the self-supervised speech learning (Baevski et al., 2020b). We compare the both criteria in Table 6. The number of distractors in the contrastive loss is 100 as (Baevski et al., 2020b). Both ASR and ST results are reported in Table 6, where the masked KL divergence loss achieves about 0 . 6 lower WER in the Librispeech dev sets and 0 . 7  1 . 4 more BLEU scores in the MuST-C tst-COMMON sets. It demonstrates the effectiveness of the proposed masked KL divergence loss for the SSL subtask. In Table 7, we present an ablation study by removing different steps/tasks in the pre-training stage. In order to make the pre-training more stable, the model training adopts a three-stage optimization strategy: 1) pre-training the T2T subtask to have a good initialization on the phoneme embeddings 2) joint pre-training with four subtasks to leverage large amounts of unlabelled speech data and abundant text data and 3) fine-tuning the model on the downstream task for best performance. In the second row, we skip the T2T pre-training step and initialize the model randomly for the joint pretraining. 0.5 WER increase is observed in average on two LIBRISPEECH dev sets. It also has more impact on the EN-ES translation direction where 1.2 BLEU score is lost without proper initialization. In the third row, we present the results without the S2T subtask. For both ASR and ST, significant performance degradation is observed, with an average 1.1 WER increase for two ASR tests and 1.8 BLEU decrease for two ST directions. We also try removing the S2P subtask while still keeping the S2T subtask. The training doesn't converge. The SSL subtask is with very small or zero cost since all predictions collapse into one or two target phonemes. Also, little progress has been made for the S2T subtask even though it is co-trained with the SSL and T2T subtasks. In the last row, the model is trained without pretraining, i.e., only the T2T and S2T subtasks are optimized. Compared with the STPT results, there is about 1.4 WER increase for two LIBRISPEECH test sets and 3.4 BLEU decrease for the two ST directions on average. In this work, we present a method to jointly pretrain speech and text in one model for speech translation and recognition under the AED framework. It includes four self-supervised and supervised subtasks from two different input modalities, hence the proposed method can leverage large amounts of unlabelled speech data and abundant text data in the pre-training stage. We conduct detailed analysis on the interference among different subtasks and propose two model configurations for the ASR and ST pre-training respectively to alleviate the subtask interference. Our experimental results show STPT can effectively fuse information within text and speech training data into one model. We achieves between 1 . 7 and 2 . 3 BLEU improvement over the state of the art on the MUST-C EN-FR and EN-ES speech translation tasks, and comparable WERs as wav2vec 2.0 in the LIBRISPEECHASR task. We highlight the potential that this work has positive impact in the society: augmenting speech processing tasks with text corpus, and improving speech related applications. At the same time, this work may have some negative consequences if the text data is not handled in a proper way. Before using the text data to train a speech system, one should evaluate fairness in the collected data, and make sure not to train on offensive or any type of inappropriate data.
abstain abstain abstain abstain objective abstain abstain objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain objective abstain abstain abstain objective objective objective objective result other other other other abstain method other other other other method abstain other abstain other abstain objective other objective abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain other abstain abstain other abstain method method abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain method abstain method method abstain abstain abstain abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain objective result result abstain abstain abstain abstain abstain