How to learn a better speech representation for end-to-end speech-to-text translation (ST) with limited labeled data? Existing techniques often attempt to transfer powerful machine translation (MT) capabilities to ST, but neglect the representation discrepancy across modalities. In this paper, we propose the S peechTE xt M anifold M ixup (STEMM) method to calibrate such discrepancy. Specifically, we mix up the representation sequences of different modalities, and take both unimodal speech sequences and multimodal mixed sequences as input to the translation model in parallel, and regularize their output predictions with a self-learning framework. Experiments on MuST-C speech translation benchmark and further analysis show that our method effectively alleviates the cross-modal representation discrepancy, and achieves significant improvements over a strong baseline on eight translation directions. Speech-to-text translation (ST) aims at translating acoustic speech signals into text in a foreign language, which has wide applications including voice assistants, translation for multinational video conferences, and so on. Traditional ST methods usually combine automatic speech recognition (ASR) and machine translation (MT) in a cascaded manner (Sperber et al., 2017; Cheng et al., 2018; Sper-ber et al., 2019; Dong et al., 2019b; Zhang et al., 2019a; Lam et al., 2021b), which might suffer from error propagation and high latency. To break this bottleneck, end-to-end ST systems attracted much * indicates corresponding authors. attention recently (Wang et al., 2020b,c; Dong et al., 2021a,b; Han et al., 2021; Inaguma et al., 2021a; Tang et al., 2021a), which learn a unified model to generate translations from speech directly. Some recent work has shown great potential for end-to-end speech translation, even surpassing traditional cascaded systems (Ye et al., 2021; Xu et al., 2021). As a cross-modal task, a major challenge in training an end-to-end ST model is the representation discrepancy across modalities, which means there is a modality gap between speech representations and text embeddings, as shown in the left sub-figure of Figure 1. Existing approaches often adopt a sophisticated MT model to help the training of ST, with some techniques like pretraining (Wang et al., 2020c; Ye et al., 2021; Xu et al., 2021), multitask learning (Ye et al., 2021; Han et al., 2021; Tang et al., 2021a) and knowledge distillation (Liu et al., 2019; Gaido et al., 2020; Inaguma et al., 2021b; Tang et al., 2021a). Although these methods have achieved impressive improvements in ST task, these methods are not necessarily the best way to leverage the MT knowledge. Considering that during training, the input of the translation module only include speech sequences or text sequences, the lack of multimodal contexts makes it difficult for the ST model to learn from the MT model. Inspired by recent studies on some cross-7050 lingual (Lample and Conneau, 2019; Liu et al., 2020a; Lin et al., 2020) and cross-modal (Li et al., 2021b; Zhou et al., 2020; Dong et al., 2019a) tasks, we suggest that building a shared semantic space between speech and text, as illustrated in the right sub-figure of Figure 1, has the potential to benefit the most from the MT model. In this paper, we propose the S peechTE xt M anifold M ixup (STEMM) method to bridge the modality gap between text and speech. In order to calibrate the cross-modal representation discrepancy, we mix up the speech and text representation as the input and keep the target sequence unchanged. Specifically, STEMM is a self-learning framework, which takes both the speech representation and the mixed representation as parallel inputs to the translation model, and regularizes their output predictions. Experimental results show that our method achieves promising performance on the benchmark dataset MuST-C (Di Gangi et al., 2019a), and even outperforms a strong cascaded baseline. Furthermore, we found that our STEMM could effectively alleviate the cross-modal representation discrepancy, and project two modalities into a shared space. In this section, we will begin with the basic problem formulation (Section 2.1) and introduce the model architecture (Section 2.2). Then, we introduce our proposed S peechTE xt M anifold M ixup (STEMM) in Section 2.3. Finally, we introduce our proposed self-learning framework with STEMM in Section 2.4 and present two mixup ratio strategies in Section 2.5. Figure 2 illustrates the overview of our proposed method. The speech translation corpus usually contains speech-transcription-translation triples, which can be denoted as D = { ( s , x , y ) } . Here s is the sequence of audio wave, x is the transcription in the source language, and y is the translation in the target language. End-to-end speech translation aims to generate translation y directly from the audio wave s , without generating intermediate transcription x . Inspired by recent works (Dong et al., 2021b; Xu et al., 2021) in end-to-end speech translation, we decompose the ST model into three modules: acoustic encoder , translation encoder , and translation decoder . The acoustic encoder first encodes the original audio wave into hidden states, fed into the translation encoder to learn further semantic information. Finally, the translation decoder generates the translation based on the output of the translation encoder . Acoustic Encoder As recent works (Ye et al., 2021; Han et al., 2021) show that Wav2vec2.0 (Baevski et al., 2020) can improve the performance of speech translation, we first use a pretrained Wav2vec2.0 to extract speech representations c from the audio wave s . We add two additional convolutional layers to further shrink the length of speech representations by a factor of 4, denoted as a = CNN ( c ) . Translation Encoder Our translation encoder is composed of N e transformer (Vaswani et al., 2017) encoder layers, which includes a self-attention layer, a feed-forward layer, normalization layers, and residual connections. For MT task, the input of the translation encoder is the embedding of transcription e = Emb ( x ) . For ST task, it is the output sequence of the acoustic encoder a . The input can also be the multimodal mixed sequence with our proposed STEMM (see details in Section 2.3). Generally, for the input sequence  , we obtain the contextual representations h (  ) after N e transformer (Vaswani et al., 2017) layers, which are fed into the translation decoder for predicting the translation. Translation Decoder Our translation decoder is composed of N t transformer decoder layers, which contain an additional cross-attention layer compared with transformer encoder layers. For the input sequence  , the cross entropy loss is defined as: Pretrain-finetune We follow the pretrain-finetune paradigm to train our model. First, we pretrain the translation encoder and translation decoder with parallel transcription-translation pairs, derived from both the speech translation corpus and the external MT dataset. Also, the acoustic encoder is pretrained on large amounts of unlabeled audio data in a self-supervised manner. We combine those pretrained modules and finetune the whole model for ST. As we mentioned in Section 1, to alleviate the representation discrepancy due to the lack of multimodal contexts, we present the S peechTE xt M anifold M ixup (STEMM) method to mix up the sequence of speech representations and word embeddings. We first introduce STEMM in this section and later show how to use it to help the training of ST. Note the sequence of sub-word embeddings as e = [ e 1 , e 2 , ..., e | e | ] and the sequence of speech representations as a = [ a 1 , a 2 , ..., a | a | ] , where the sequence lengths usually follow | a |  | e | . We first perform a word-level forced alignment between speech and text transcriptions to determine when particular words appear in the speech segment. Formally, the aligner recognizes a sequence of word units w = [ w 1 , w 2 , ..., w T ] , and for each word w i , it returns the start position l i and end position r i in the sequence of speech representation a . Meanwhile, we denote the corresponding sub-word span for word w i as [ x m i : x n i ] , with its embeddings matrix [ e m i : e n i ] , where m i and n i are the start position and end position in the sequence of sub-words. To mix up both sequences, for each word unit w i , we choose either the segment of speech representations [ a l i : a r i ] or sub-word embeddings [ e m i : e n i ] with a certain probability p  , referred to mixup ratio in this paper. Note that in terms of the mixup representation sequence length, we have | e |  | m |  | a | . Considering the positions of tokens have changed after mixup, we add positional encodings to the token embeddings. We further perform layer normalization to normalize the embeddings: where Pos (  ) is the sinusoid positional embedding (Vaswani et al., 2017). Mixup (( s , x ) , p  ) indicates the mixup sequence of speech s and text x with probability p  , which is fed into the translation encoder for predicting the translation. With the help of our proposed STEMM, we are now able to access multimodal mixed sequences, in addition to the unimodal speech sequences. We integrate them into a self-learning framework. Specifically, we input both unimodal speech sequences and multimodal mixed sequences into the translation module ( translation encoder and translation decoder ). In this way, translation of unimodal speech sequences focuses on the ST task itself, while the translation of multimodal mixed sequences is devoted to capture the connections between representations in different modalities. Besides, we try to regularize above two output predictions by minimizing the Jensen-Shannon Divergence (JSD) between two output distributions, LJSD ( s , x , y ,p  ) = | y | (cid:88) i =1 JSD { p  ( y i | y <i , h ( s ))  p  ( y i | y <i , h ( Mixup (( s , x ) , p  ))) } , (5) where h (  ) is the contextual representation outputted by the translation encoder . p  ( y i | y <i , h ( s )) is the predicted probability distribution of the i -th target token given the speech sequence s as input, and p  ( y i | y <i , h ( Mixup (( s , x ) , p  ))) is that given the multimodal mixed sequence as input. When using our proposed STEMM, an important question is how to determine the mixup ratio p  . Here we try two strategies: static mixup ratio and uncertainty-aware mixup ratio . Static Mixup Ratio We use the same mixup ratio p  for all instances throughout the whole training process. We will show how we determined this important hyper-parameter in Section 4.3. Uncertainty-aware Mixup Ratio With this strategy, we determine the mixup ratio for each instance according to the prediction uncertainty of the ST task, defined as the average entropy of predicted distributions of all target tokens: where U is a normalization factor which re-scales u to [0 , 1] ,  (  ) is a sigmoid function to prevent p  from dropping too quickly. MuST-C We conduct experiments on MuST-C (Di Gangi et al., 2019a) dataset. MuST-C is a multilingual speech translation dataset, which contains translations from English (En) to 8 languages: German (De), French (Fr), Russian (Ru), Spanish (Es), Italian (It), Romanian (Ro), Portuguese (Pt), and Dutch (Nl). It is one of the largest speech translation datasets currently, which contains at least 385 hours of audio recordings from TED Talks, with their manual transcriptions and translations at the sentence level. We use dev set for validation and tst-COMMON set for test. MT Datasets Our model architecture allows us to utilize external parallel sentence pairs in large-scale machine translation datasets. Therefore, we incorporate data from WMT for En-De, En-Fr, En-Ru, En-Es, En-Ro, and OPUS100 1 for En-Pt, En-It, En-Nl, as pretraining corpora. The detailed statistics of all datasets included are shown in Table 1. Pre-processing For speech input, we use the raw 16-bit 16kHz mono-channel audio wave. To perform word-level force alignment, we use Montreal Forced Aligner 2 toolkit, whose acoustic model is trained with LibriSpeech (Panayotov et al., 2015). For text input, we remove the punctuation from the source texts for the ST dataset. Both source and target texts are case-sensitive. For each translation direction, we use a unigram SentencePiece 3 model to learn a vocabulary on the text data from ST dataset, and use it to segment text from both ST and MT corpora into subword units. The vocabulary is shared for source and target with a size of 10k. Model Configuration Our model consists of three modules. For the acoustic encoder , we use Wav2vec2.0 (Baevski et al., 2020) following the base configuration, which is pretrained on audio data from LibriSpeech (Panayotov et al., 2015) without finetuning 4 . We add two additional 1-dimensional convolutional layers to further shrink the audio, with kernel size 5, stride size 2, padding 2, and hidden dimension 1024. For the translation encoder , we use N e = 6 transformer encoder layers. For the translation decoder , we use N d = 6 transformer decoder layers. Each of these transformer layers comprises 512 hidden units, 8 attention heads, and 2048 feed-forward hidden units. Training and Inference We train our model in a pretrain-finetune manner. During pretraining, we train the MT model i.e., translation encoder and translation decoder , with transcription-translation pairs. The learning rate is 7e-4. We train the model with at most 33k input tokens per batch. During finetuning, the learning rate is set to 1e-4. We finetune the whole model up to 25 epochs to avoid overfitting, with at most 16M source audio frames per batch. The training will early-stop if the loss on dev set did not decrease for ten epochs. During both pretraining and finetuning, we use an Adam optimizer (Kingma and Ba, 2015) with  1 = 0 . 9 ,  2 = 0 . 98 , and 4k warm-up updates. The learning rate will decrease proportionally to the inverse square root of the step number after warm-up. The dropout is set to 0.1, and the value of label smoothing is set to 0.1. We use the uncertainty-aware mixup ratio strategy by default, and the mixup ratio p  is set to 0.4 when using static strategy. The weight  of JSD loss is set to 1.0. During inference, We average the checkpoints of the last 10 epochs for evaluation. We use beam search with a beam size of 5. We use sacreBLEU 5 (Post, 2018) to compute case-sensitive detokenized BLEU (Papineni et al., 2002) scores and the statistical significance of translation results with paired bootstrap resampling (Koehn, 2004) for a fair comparison 6 . All models are trained on 8 Nvidia Tesla-V100 GPUs. We implement our models based on fairseq 7 (Ott et al., 2019). Baseline Systems We compare our method with several strong end-to-end ST systems including: 5 https://github.com/mjpost/sacrebleu 6 sacreBLEU signature: nrefs:1 | bs:1000 | seed:12345 | case:mixed | eff:no | tok:13a | smooth:exp | version:2.0.0 7 https://github.com/pytorch/fairseq 7054 Fairseq ST (Wang et al., 2020a), AFS (Zhang et al., 2020), DDT (Le et al., 2020), MTL (Tang et al., 2021b), Self-training (Pino et al., 2020), BiKD (In-aguma et al., 2021a), FAT-ST (Zheng et al., 2021a), JT-S-MT (Tang et al., 2021a), SATE (Xu et al., 2021), Chimera (Han et al., 2021) and XSTNet (Ye et al., 2021). Besides, we implement a strong baseline W2V2-Transformer based on Wav2vec2.0. It has the same model architecture as our proposed STEMM and is pretrained in the same way. The only difference is that it is only finetuned on the ST task, while we adopt a self-learning framework during finetuning. Comparison with End-to-end Baselines As shown in Table 2, our implemented W2V2-Transformer is a relatively strong baseline, which proves the effectiveness of Wav2vec2.0 module and MT pretraining. Without external MT data, our method achieves an improvement of 1.0 BLEU (average over 8 directions) over the strong baseline, which proves our proposed self-learning framework could effectively improve the performance of the ST task. It even outperforms baselines with external MT data on En-Es, En-It, En-Ro, En-Pt, and En-Nl. When we introduce additional MT data, our method also yields a 0.8 BLEU improvement compared with baseline. Note that our performance is slightly worse than XSTNet (Ye et al., 2021). However, our method is orthogonal with theirs, which focuses on the training procedure of end-to-end ST model. We will investigate how to combine them together in the future. Comparison with Cascaded Baseline We also implement a strong cascaded system, whose ASR part is composed of a pretrained Wav2vec2.0 module and 6 transformer decoder layers, and the MT part is the same as our pretrained MT module. Both cascaded systems and end-to-end models are trained with the same data ( D and DMT ). As shown in Table 3, the end-to-end baseline W2V2-Transformer is inferior to the cascaded system, but our method significantly outperforms it, which shows the potential of our STEMM method. Is Each Learning Objective Effective? As shown in Equation 6, our training objective contains three terms. Besides the cross-entropy objec-Mixup Ratio STEMM Trans. tive LCE ( s , y ) for speech translation, we investigate the effects of the other two auxiliary training objectives. As shown in Table 4, when we input the additional multimodal mixed sequence into the model and optimize the cross-entropy loss (Line 3), it can already outperform the baseline (Line 4) significantly. When we regularize two output predictions with JSD loss (Line 2), the performance can be further boosted. The uncertainty-aware strategy reduces the cost for searching mixup ratio and has better performance. We present two different mixup ratio strategies in Section 2.5. To evaluate their impacts, we conduct another ablation study on MuST-C En-De. We observe that the BLEU scores on tst-COMMON set are 28.5 and 28.7 for static strategy and uncertainty-aware strategy , respectively. The uncertainty-aware strategy can slightly improve the performance, and more importantly, it lowers the manual cost for searching an optimal mixup ratio to get the best performance. When using static mixup ratio strategy , it is important to choose the mixup ratio p  . We constrain p  in [0 . 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8] for experiments on MuST-C En-De tst-COMMON set, as shown in Figure 3. When p  = 0 . 0 , the translation task with the mixed sequence as input degrades to the MT task. We interestingly find that self-learning with MT tasks performed the worst (i.e. lowest BLEU) than self-learning with STEMM at other mixup ratios. This confirms what we mentioned in Section 1, that the representation discrepancy between speech and text makes the MT task an inferior boost to ST. Our method achieves the best performance at p  = 0 . 4 . To find a reasonable explanation, we do a more in-depth study of the representation of the speech, text, and their mixup sequence (STEMM). We plot the bivariate kernel density estimation based on the reduced 2-dim representation. We find that when p  = 0 . 4 , the mixup representation just lies between the representation of speech and text sequences. That is why it calibrates the cross-modal representation discrepancy more easily and gets the best ST performance. To examine whether our method alleviates the cross-modal representation discrepancy, we conduct some analysis of cross-modal word representations. As described in Section 2.3, for each word unit w i , we identify the corresponding segment of speech representation [ a l i : a r i ] and text embedding [ e m i : e n i ] . We define the word representation in each modality as follows:  i = AvgPool ([ a l i : a r i ]) , (9)  i = AvgPool ([ e m i : e n i ]) , (10) where AvgPool() denotes average-pooling operation across the sequence dimension,  i and  i denote the representation of word unit w i in speech and text modalities, respectively. We calculate the average cosine similarity between  i and  i over all word units w i in MuST-C 60 40 20 0 20 40 60 80 x 100 50 0 50 100 y Speech Mixup 0.4 Text Figure 4: The bivariate kernel density estimation visualization of the averaged sentence representation of the speech, text and STEMM sequences after pretraining. En-De tst-COMMON set. As shown in Table 5, our method could significantly improve the similarity of word representations across modalities over baseline. We believe it is because when training with our proposed STEMM, the speech segment and text segment of a word will appear in a similar multimodal context, which leads to similar representations. We also show the visualization of an example in Figure 5, we can observe that our method brings word representations within different modalities closer compared with baseline. One important contributor to our excellent performance is the usage of external MT data. Therefore, how the amount of MT data affects the final performance is an important question. We vary the amount of available external MT data during pretraining on En-De direction. As shown in Figure 6, we observe a continuous improvement of BLEU scores with the increase of MT data, which shows that external MT data is helpful to improve ST. 4.6 Can the Final Model still Perform MT Task? Our model is first pretrained on the MT task and then finetune for ST. An important question is whether there is a catastrophic forgetting problem during finetuning. We evaluate the model on the MT task and show the result in Table 6. We observe that when we only finetune the model on the ST task (W2V2-Transformer), the ability of text trans-Models BLEU Pretrained MT 31.7 W2V2-Transformer 19.5 STEMM 31.5 Table 6: BLEU scores of MT task on MuST-C EnDe tst-COMMON set. lation will be forgotten a lot. In contrast, when we use our self-learning framework during finetuning, even though there is no MT task, the MT capability can still be preserved. 5 Related Works End-to-end ST To overcome the error propagation and high latency in the cascaded ST systems, Brard et al. (2016); Duong et al. (2016) proved the potential of end-to-end ST without intermediate transcription, which has attracted much attention in recent years (Vila et al., 2018; Salesky et al., 2018, 2019; Di Gangi et al., 2019b,c; Bahar et al., 2019a; Inaguma et al., 2020). Since it is difficult to train an end-to-end ST model directly, some training techniques like pretraining (Weiss et al., 2017; Berard et al., 2018; Bansal et al., 2019; Stoian et al., 2020; Wang et al., 2020b; Pino et al., 2020; Dong et al., 2021a; Alinejad and Sarkar, 2020; Zheng et al., 2021b; Xu et al., 2021), multi-task learning (Le et al., 2020; Vydana et al., 2021; Tang et al., 2021b; Ye et al., 2021; Tang et al., 2021a), curriculum learning (Kano et al., 2017; Wang et al., 2020c), and meta-learning (Indurthi et al., 2020) have been applied. To overcome the scarcity of ST data, Jia et al. (2019); Pino et al. (2019); Bahar et al. (2019b) proposed to generate synthesized data based on ASR and MT corpora. To overcome the modality gap, Han et al. (2021); Huang et al. (2021); Xu et al. (2021) further encode acoustic states which are more adaptive to the decoder. Previous works have mentioned that the modality gap between speech and text is one of the obstacles in the speech translation task, and to overcome such gap, one branch of the works (Liu et al., 2020b; Dong et al., 2021b; Xu et al., 2021) introduced a second encoder based on the conventional encoder-decoder model, to extract semantic information of speech and text. Recently, Han et al. (2021) built a shared semantic projection module that simulates the human brain, while in this work, we 7057 explored how to construct an intermediate state of the two modalities via the recent mixup method ( i.e. S peechTE xt M anifold M ixup) to narrow such gap. Note that our work is orthogonal with Ye et al. (2021)'s study in training procedure of end-to-end ST model. Mixup Our work is inspired by the mixup strategy. Zhang et al. (2018) first proposed mixup as a data augmentation method to improve the robustness and the generalization of the model, where additional data are constructed as the linear interpolation of two random examples and their labels at the surface level. Verma et al. (2019) extended the surface-level mixup to the hidden representation by constructing manifold mixup interpolations. Recent work has introduced mixup on machine translation (Zhang et al., 2019b; Li et al., 2021a; Guo et al., 2022; Fang and Feng, 2022), sentence classification (Chen et al., 2020; Jindal et al., 2020; Sun et al., 2020), multilingual understanding (Yang et al., 2022), and speech recognition (Medennikov et al., 2018; Sun et al., 2021; Lam et al., 2021a; Meng et al., 2021), and obtained enhancements. Our approach is the first to introduce the idea of manifold mixup to the speech translation task with two modalities, speech, and text. In this paper, we propose a S peechTE xt M anifold M ixup (STEMM) method to mix up the speech representation sequences and word embedding sequences. Based on STEMM, we adopt a self-learning framework, which learns the translation of unimodal speech sequences and multimodal mixed sequences in parallel, and regularizes their output predictions. Experiments and analysis demonstrate the effectiveness of our proposed method, which can alleviate the cross-modal representation discrepancy to some extent and improve the performance of ST. In the future, we will explore how to further eliminate this discrepancy and fill the cross-modal transfer gap for ST. Acknowledgements We thank all the anonymous reviewers for their insightful and valuable comments. This work was supported by National Key R&D Program of China (NO. 2017YFE0192900).
abstain abstain objective method result abstain abstain abstain abstain abstain abstain other abstain abstain method objective abstain abstain result result method method method method abstain abstain abstain abstain method method method method method method method method method method abstain method abstain other method method method abstain method method other method method method method method method abstain method method abstain method abstain method abstain method abstain abstain abstain abstain method abstain method method method abstain method other method method method result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain result abstain abstain result result abstain result abstain result abstain abstain abstain abstain other abstain abstain result abstain result abstain abstain abstain result abstain result result abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain other abstain objective method objective abstain