End-to-end simultaneous speech-to-text translation aims to directly perform translation from streaming source speech to target text with high translation quality and low latency. A typical simultaneous translation (ST) system consists of a speech translation model and a policy module, which determines when to wait and when to translate. Thus the policy is crucial to balance translation quality and latency. Conventional methods usually adopt fixed policies, e.g. segmenting the source speech with a fixed length and generating translation. However, this method ignores contextual information and suffers from low translation quality. This paper proposes an adaptive segmentation policy for end-to-end ST. Inspired by human interpreters, the policy learns to segment the source streaming speech into meaningful units by considering both acoustic features and translation history, maintaining consistency between the segmentation and translation. Experimental results on English-German and Chinese-English show that our method achieves a good accuracy-latency trade-off over recently proposed state-of-the-art methods. Recent years have witnessed extensive studies and rapid progress of Simultaneous translation (ST). It aims to perform translation from source speech into the target language with high quality and low latency and is widely used in many scenarios, such as international conferences, press releases, etc. Generally, the research of ST falls into two categories: the cascaded method, and the end-to-end  Corresponding author. 1 In German, each singular noun is assigned a gender, either masculine , feminine , or neuter , which determines whether the definite article (like The in English) preceding the noun is  Der ,  Die  or  Das . Therefore, translating The hastily without receiving the following noun may cause mistranslation. method. The cascaded method consists of an automatic speech recognition (ASR) model which transcribes the source speech into source streaming text (Moritz et al., 2020; Wang et al., 2020b; Li et al., 2020a), and a followed-by machine translation (MT) model that generates translation based on the ASR output. Since there are no sentence or segment boundaries in the streaming source text output by ASR, a segmentation policy is required to link the ASR and the MT to determine when to read more source tokens and when to start translation (Oda et al., 2014; Dalvi et al., 2018; Ma et al., 2019; Arivazhagan et al., 2019; Zhang et al., 2020; Wilken et al., 2020). However, cascaded methods 7862 face two main challenges. One is the error propagation that the ASR errors may hurt the translation quality. The other is the increase of latency because the translation model has to wait for the output of the ASR model. To overcome these limitations, the end-to-end method attempts to directly translate from source speech to target text, without explicitly transcribing the source speech (Bansal et al., 2018; Di Gangi et al., 2019b; Jia et al., 2019). To balance the translation quality and latency, the key challenge lies in the segmentation policy that determines the translation boundaries of the speech frames. Most of the previous work used fixed policies. Some of them take fixed-length policy (Nguyen et al., 2021; Ma et al., 2020b, 2021) that splits speech at a fixed frequency, for example, to generate one target word every T s ms (Figure 1 (a)). Other work adopts word-based policy that splits the speech into words and generates one target word whenever a new source word is detected, which calls for an auxiliary source word detector (Ren et al., 2020; Elbayad et al., 2020; Ma et al., 2020b; Zeng et al., 2021; Chen et al., 2021), see Figure 1 (b). However, both the above methods are hard policies, which do not consider the contextual information and result in low translation quality (Ari-vazhagan et al., 2019; Zhang et al., 2020). In this paper, we propose an adaptive segmentation policy for end-to-end simultaneous translation based on Meaningful Unit (MU). The idea is borrowed from human interpreters, who do interpretation based on a unit with clear meaning rather than fixed frame length or word. We model the speech segmentation policy as a binary classification that determines whether a speech segment is an MU. Once an MU is detected, it is fed into an end-to-end speech translation model, as illustrated in Figure 1 (c). We propose a supervised training method, using both acoustic features and translation features to train the policy. Besides, we propose an incremental decoding method to construct training data from speech and translation pairs. Concretely, we first train a full speech translation model MST , and then gradually expand speech frames to simulate simultaneous translation. When the translation of the current speech segment is a prefix of the full speech translation, the segment is extracted as an MU. At inference time, we employ the same MST to maintain the consistency between segmentation and translation. Our method is more flexible than fixed policies, as it dynamically detects meaningful units according to contextual information. Experiments on two language pairs show that the proposed approach outperforms the strong baselines in balancing translation quality and latency. Cascade Simultaneous Translation. To eliminate the impact of ASR errors, most previous work of cascade ST use golden transcript, rather than ASR result, to explore different read/write policies in ST. Existing policies can be classified into two categories: 1) The fixed policy segments the source text based on fixed lengths (Ma et al., 2019; Dalvi et al., 2018). For example, wait-k (Ma et al., 2019) is a typical fixed policy that first read k source words, then generates one target word immediately after receiving one subsequent source word. 2) The adaptive policy learns to segment the source text according to its context (Oda et al., 2014; Cho and Esipova, 2016; Gu et al., 2017; Arivazhagan et al., 2019; Ma et al., 2020a; Zhang et al., 2020). It has been proven that the adaptive policy is more effective than the fixed policy in balancing translation quality and latency (Zhang et al., 2020). End-to-End Simultaneous Translation. The method has shown great potential over the cascaded method (Brard et al., 2016; Weiss et al., 2017; Bansal et al., 2018; Jia et al., 2019; Wang et al., 2020a; Li et al., 2020b; Ansari et al., 2020). End-to-end ST contains a speech translation model, along with a policy to decide when to translate. However, most previous studies are based on fixed-length policy that translate every T s ms (Nguyen et al., 2021; Ma et al., 2020b), or decide to translate whenever a fixed number of words are detected (Ren et al., 2020; Elbayad et al., 2020; Zeng et al., 2021; Ma et al., 2021; Chen et al., 2021), following the fixed policy of cascade ST systems. This paper presents an adaptive policy for end-to-end simultaneous translation. We are motivated by an adaptive policy proposed for cascade ST (Zhang et al., 2020), which proposed to perform translation when a source text segment is detected to be a unit with clear meaning. However, there are three main differences. First, our method is proposed for end-to-end ST, while Zhang et al. (2020) is for cascade ST. Second, our method directly detects MU on speech rather than on the streaming text of the output of ASR. Third, we propose a multi-modal MU detection model using both acoustic features 7863 and translation history. The overall framework of our adaptive speech segmentation policy is illustrated in Figure 2. Given a streaming speech s , we incrementally detect whether a speech clip s  t ( t = 1 , 2 , ... ) is an MU, where s  t denotes the head t F frames of s , and F is the detection interval. Once an MU is detected, the speech translation model produces its translation y (cid:48) with the translation history y p force decoded as a translation prefix. Meanwhile, y (cid:48) is displayed to users and added to the translation history y p to improve MU detection. In the following, we first introduce our MU detection model (Section 3.1), then propose a method to construct MU training data (Section 3.2). Finally, we describe the training details in Section 3.3. We model the MU detection as a classification problem. Given a source speech s , the detector incrementally reads speech clips at each time t , to make a decision whether s  t is an MU. We propose a multi-modal detector that uses both acoustic features and translation history. See the bottom green block of Figure 2 for illustration. For the acoustic feature extractor E fa , we use stacked temporal convolutional layers performed on raw speech features (80-channel log-mel filter-banks). Each of the convolutional layers is followed by layer normalization and a GELU activation function (Hendrycks and Gimpel, 2016), following Baevski et al. (2020). For the context feature extractor E ft , we use a trainable word embedding layer. The outputs of the feature extractors are fed to the acoustic encoder E e a and context encoder E e t to generate latent representation, respectively. We add a position embedding to the textual embedding, as in BERT (Devlin et al., 2019), and add a convolutional layer as the relative positional embedding to the acoustic embedding, similar to Mohamed et al. (2019); Baevski et al. (2019). Both E et and E ea follow the Transformer architecture (Sperber et al., 2018; Devlin et al., 2019). Next, we add different type embeddings e Type ( e [ TXT ] and e [ AUD ] ) to both text and audio embeddings to indicate their source type. These two sequences are then concatenated and fed to a 6-layer Transformer for cross-modal fusion. Special tokens [ CLS ] and [ SEP ] are added in this process following BERT (Devlin et al., 2019). The final hidden state corresponding to [ CLS ] is used as the aggregated sequence representation to predict the classification result l (cid:48) using a fully-connected layer, followed by a softmax. where E mm denotes the multi-modal fusion Transformer and fc performs a fully-connection layer. Since there are no standard MU segmentation training corpora, we propose a simple method to automatically extract meaningful speech units to construct MU training samples. We expect that MUs can be translated properly without waiting for future speech. Therefore, we define MU as the minimum speech segment whose translation will not be changed by subsequent speech . This requires MUs to contain enough information to generate stable translation. Accordingly, we propose to extract meaningful speech units by comparing the translation of every speech prefix segment and the full-speech translation with a pre-trained speech translation model MST . For a speech segment s  t , if its translation y (cid:48) = MST ( s  t ) is a prefix of the full-speech translation (cid:101) y = MST ( s ) , we identify that s  t is sufficient to provide a stable translation and annotate it to an MU. We propose an incremental-translation paradigm. We incrementally translate s  t , t = 1 , 2 , ... , to judge whether its translation y (cid:48) is a prefix of (cid:101) y . If so, we extract s  t as an MU, and force-decode its translation y (cid:48) as a prefix in detecting subsequent speech segments. This is to keep consistent with the force-decoding strategy at the inference stage. Moreover, while comparing y (cid:48) with (cid:101) y , as illustrated in Figure 3, we propose a tail-truncation strategy that discards the last k words from the partial decoding results y (cid:48) . This is to avoid translation 2 Note that the length of acoustic encoding T is not equal to the number of source frames t F for the temporal sampling of convolutional feature extractor layers. errors caused by ambiguous speech fragments. For example, s  3 pronounces  The dog has  is translated to Der Hund hat, which is not a prefix of the full-speech translation (cid:101) y due to the current received speech is ambiguous 3 . However, after truncated the tail word, y (cid:48) turn to be Der Hund and becomes a prefix of (cid:101) y . Therefore, discarding tail words from y (cid:48) enables the model to discover translation that partially matches (cid:101) y in advance, thus shortening the granularity of extracted MUs and reducing latency. At the inference stage, we also remove tail k words from the translation of detected MUs. Our pre-trained speech translation model MST includes an acoustic feature extractor E fa , an acoustic encoder E e a , and a textual translation decoder, as shown in Figure 2(c). E fa and E ea are shared with the multi-modal MU detection model so that the acoustic forward computation can be shared at inference time. The translation decoder is based on Transformer, which links the acoustic encoder through cross-attention (Vaswani et al., 2017). Note that to keep MU detection consistency in both training and decoding, we initialize the acoustic feature extractor and context encoder with the weights from MST , and keep them frozen in training MU detection, instead of joint training with 3 The English word "has" can be translated to either "ist" or "hat" in most cases, depending on what follows. Formally, the MST model optimizes the two acoustic encoders and the translation decoder first with auto-regressive loss LST : LST =  (cid:88) ( s,y ) D STN (cid:88) i =1 log p ( y i | s, y <i ;  ae ,  td ) (3) Then the MU detection model is optimized without gradients back-propagated to the acoustic encoders: 7865 LMU =  (cid:88) ( s,y p ,l ) D MU log p ( l | s, y p ;  te ,  mm ) (4) where  ae denotes weights of the two acoustic encoders,  td is the translation decoder of MST ,  te is the textual encoders for y p and  mm is the weights of the multi-modal fusion. DST is the speech translation dataset and DMU contains training triplets generated by the pre-trained MST model. We carry out experiments on English-German (En-De) and Chinese-English (Zh-En) simultaneous translation. We use sacreBLEU (Post, 2018) to evaluate the translation performance and the acoustic average lagging (AL) (Ma et al., 2019, 2020b) as the latency metric. The AL measures the system lagging behind an ideal policy, which produces translation at the same speed as the audio received. We evaluate our method on MuST-C (Di Gangi et al., 2019a) En-De dataset and BSTC (Zhang et al., 2021a) Zh-En dataset. To compare with the previous methods, we carried out experiments under two settings: the Limited-training-corpora setting that constrains the training data to a limited set of corpora, and the Open-training-corpora setting uses more data. For En-De, we set the training data of Limited-training-corpora setting as the training set of MuST-C, a dataset consisting of 408 hours of speech with transcription and translation, while the experiments with the Open-training-corpora setting use unlimited datasets of up to 1,302 hours of speech. See appendix A.1 for detail. We compare our method with previous strong ST approaches. Methods listed with  *  are carried out under the Open-training-corpora setting, while others use the Limited-training-corpora setting. Wait-k (Chen et al., 2021) integrates the wait-k (Ma et al., 2019) policy into end-to-end speech translation with an additional ASR module to detect the number of source words within the streaming speech. SimulST (Ma et al., 2020b) takes the fixed-length policy that translates out one token every T s ms. We set T s to 280 following their best experimental settings. StreamMemory (Ma et al., 2021) proposes an end-to-end speech translation model with augmented memory, which stores previous states of streaming speech to reduce the computation cost. They use the same fixed-length policy as in SimulST . RealTranS (Zeng et al., 2021) proposes a fixed policy (Wait-k-Stride-N) for an end-to-end ST that triggers translation based on the number of words within the streaming speech, which is detected based on a CTC module built on top of the speech translation encoder. Wait-K-Stride-S-Write-N * (Nguyen et al., 2021) proposes a fixed-length policy for end-to-end ST that first wait for K frames, then alternatively decoding N target words and read-ing S frames. ON-TRAC * (Elbayad et al., 2020): A cascade system that achieved the first-place of the IWSLT2020 En-De simultaneous translation shared task (Federico et al., 2020). It takes a fixed policy ( wait-k ) to link the ASR output and the MT module. MU-ST : Our proposed method that triggers the speech translator with an MU-based adaptive policy. The MST model is trained from scratch. MU-ST ( +pretrain )*: To compare with methods of the Open-training-corpora setting, we take pre-training techniques in training the speech translator MST . to enhance the acoustic representations, as performed in SimulST (Ma et al., 2020b), StreamMemory (Ma et al., 2021) and RealTranS (Zeng et al., 2021). For MU-ST ( +pretrain )*, we follow the recently proposed speech translation pre-training method (Li et al., 2020b) to initialize the encoder with wav2vec2.0 4 (Baevski et al., 2020) and initialize the decoder with mBART50 5 (Tang et al., 2020), then fine-tune with speech translation corpora. As listed in Table 1, MU-ST takes the Base model and MU-ST ( +pretrain )* takes the Big version 6 . We set the number of truncated words k=2 in tail-truncation and the length of speech clips T s = 250 ms as default. Figure 4 shows the results on MuST-C dev set and tst-COMMON set. We calculate the computation-aware latency on the MuST-C dev set and computation-unaware latency on the MuST-C tst-COMMON set, to be consistent with previous work. The difference between them is whether the model inference time is taken into account. MU-ST achieves higher translation quality under the same latency on both datasets. denotes the probability threshold of MU detector, i.e.,  = [0 . 3 , 0 . 4 , ..., 0 . 9] corresponds to the results of taking p ( l = 1 | s, y p ) >  as the criterion of determining s to be an MU. Small  produces fine-grained speech segments and small delay, but if some ambiguous speech segments are incorrectly recognized as MUs, it will result in poor translation quality. On the dev set, we compare MU-ST with Wait-k (Chen et al., 2021), SimulST (Ma et al., 2020b) and StreamMemory (Ma et al., 2021). SimulST and StreamMemory takes fixed-length speech policy (Figure 1 (a)) while Wait-k performs wait-k based on the number of words detected from an ASR module (Figure 1 (b)). We also plot the result of a Cascaded system based on textual Wait-K (Chen et al., 2021) . We observed that:  Our adaptive policy outperforms the Wait-k methods, and the Wait-k approaches are superior to the fixed-length methods. We report the result of translating the whole speech without segmentation in the full-speech translation. MU-ST approaches the BLEU of full-speech translation as early as  = 0 . 6 , indicating that our method can achieve comparable BLEU with full-speech translation with a very small latency (about 2100ms), while other methods still have a large gap with the full-speech translation under corresponding delay. On the tst-COMMON set, we compare MU-ST with three fixed-policy methods. ON-TRAC * and RealTranS follow the wait-k policy, while Wait-K-Stride-S-Write-N * takes the fixed-length policy. We observed that:  Our proposed MU-ST trained with MuST-C achieves higher BLEU at all latency regimes than other approaches. In particular, it even superior to the cascade method ON-TRAC *, 7867 15 16 17 18 19 20 21 22 23 0 1000 2000 3000 4000 5000 6000 BLEU AL(ms) tail-truncation-0tail-truncation-2tail-truncation-4tail-truncation-6Full-Speech Figure 5: Impact of the number of truncated words in tail-truncation . which utilized large-scale ASR and MT corpora for training. MU-ST ( +pretrain )* outperforms MU-ST in BLEU by taking advantage of pre-trained models and more training data for MST . The full-speech translation of MU-ST ( +pretrain )* has 3.32 BLEU points improvement (22.58  25.90) over MU-ST . Meanwhile, the latency of MUST ( +pretrain )* is longer than MU-ST under the same  . This may be because the pre-trained large speech translation model of MUST ( +pretrain )* enables the high translation diversity of its speech translation model MST , which is not conducive in constructing fine-grained meaningful units. Strong translation diversity will reduce the probability of prefix matching between partial translation y (cid:48) i and full-speech translation (cid:101) y in MU construction, thus bringing in longer MUs and higher latency. We conduct experiments concerning various aspects of our MU-based policy in this section. All ablation results are trained on the Limited-training-corpora setting and evaluated on MuST-C tst-COMMON set. When constructing the data for MU detection, we proposed a tail-truncation strategy, which removes the last k words from the translation of each speech segment to avoid translation errors caused by ambiguous speech segments. Now we verify its significance. We compare models with different numbers of truncated words k in tail-truncation , with results shown in Figure 5. It is observed that without tail-truncation ( k = 0), the translation quality is worse and the latency is longer, compared with k = 2. This corroborates our motivation specified in Section 3.2 that tail-truncation enables the model to discover fine-grained meaningful units. Moreover, it also facilitates producing context-aware translation by taking longer context in translation. Therefore, tail-truncation strategy plays an important role in extracting meaningful units. Increasing k from 2 to 4 and 6 generally brings higher latency, along with a tiny improvement in translation quality. According to the I-MOS ranking mechanism (Zhang et al., 2021b) for ST systems, k = 2 and k = 6 ranks tied, both better than k = 4. k = 2 is superior at low-latency regime and k = 6 performs better at high-latency regime. This is because according to our MU extraction algorithm, we can always guarantee the consistency between the MU translation and the full-speech translation, regardless of the value of k . Larger k makes it easier to match partial translation and full-speech translation, thus producing more fine-grained MUs. But at the same time, truncating more translated words can avoid displaying problematic translations at the tail. So the translation quality of large k will not degrade. On the contrary, using a larger k improves the translation accuracy because it receives more source speech when performing translation. To further study the effect of the translation history for MU detection, we remove the previously generated translation y p from the multi-modal MU detection model. Without y p , the segmentation model detects MUs only based on the input speech clip s  t , and only optimizes the top 6-layer Transformer. We build golden segmentation on tst-COMMON based on the meaningful speech units construction algorithm (Section 3.2), then evaluate different models on MU segmentation, translation quality, and latency. The results are shown in Table 2. It is observed that for both limited and open training corpora settings, the multi-modal method which combines speech features and translation history outperforms the single-modal method on MU segmentation in terms of F1 score (absolute improvements of 1.6-2 percentage points). However, there are only slight improvements in translation quality and a slight delay in latency. This is because incorrect segmentation does not necessarily lead to the decline of BLEU, which also depends on 7868 Model setting F1 (%) BLEU AL (ms) MU-ST Single-Modal 72.6 20.92 1642.2 Multi-Modal 74.6 21.07 1684.8 MU-ST ( +pretrain )* Single-Modal 72.5 22.73 1925.7 Multi-Modal 74.1 22.78 1952.5 Table 2: The performance of single-modal and multimodal MU detection models evaluated on MuST-C tst-COMMON at  = 0 . the robustness of the speech translation model. For some small errors brought by wrong segmentation, a robust speech translation model may ignore them and generate correct translation when translating subsequent MUs. In such cases, the overall BLEU will not be largely affected. We also evaluate our method on Zh-En ST using BSTC (Zhang et al., 2021a) dataset. BSTC is the largest Zh-En public speech translation corpus, but contains only 66 hours of speech, corresponding to 37k sentences. To alleviate the data scarcity, we first construct pseudo speech translation data by translating the transcript of ASR corpora (AISHELL-1 (Bu et al., 2017), AISHELL-3 (Shi et al., 2020), and aidatatang_200zh 8 ) with a Zh-En machine translation model trained on a translation corpus, CCMT2019 (Yang et al., 2019). Then the pseudo speech translation data, together with the BSTC, are assigned as the training corpus for the Zh-En end-to-end speech translation model. The combined training set contains a total of 529 hours of speech, corresponding to 478k sentence pairs of transcript and translation. We implement three methods for comparison:  Cascade : we use an adaptive policy (Zhang et al., 2020) to connect an ASR model and an MT model. The ASR model is trained on 529 hours of speech, and the MT model based on Transformer big is pre-trained on CCMT2019 and fine-tuned on BSTC. The adaptive policy based on textual MU (Zhang et al., 2020) is trained on BSTC. Cascade * : Similar to Cascade , the only difference is that it adopts a public real-time ASR API 9 that uses more than 9400 hours of ASR training data (Amodei et al., 2016). 8 a free Chinese Mandarin speech corpus by Beijing DataTang Technology Co., Ltd (www.datatang.com) 9 https://ai.baidu.com/tech/speech/realtime_asr 20 22 24 26 28 30 32 34 1000 1200 1400 1600 1800 2000 2200 2400 2600 BLEU Computation-unaware AL(ms) Cascade Cascade* MU-ST Full-SpeechTranslation ... 4373 4526  = 0.5  = 0.9  = 0. MU-ST : Our proposed adaptive speech segmentation policy for end-to-end ST. The acoustic encoders and target decoders of our speech translation model are initialized by the ASR and MT model of Cascade method, respectively. Then we fine-tune with speech translation data following Li et al. (2020b). The results in Figure 6 show that: 1) Cascade * has a significant advantage over the other two methods. This is because the word error rate of the ASR model is 10.32% and 21.58% for Cascade * and Cascade , respectively, leading to 5.9 BLEU points of gap between their full-speech translation results (27.2 vs. 33.1) 10 . 2) Cascade and our end-to-end method MU-ST are optimized with identical training data, but MU-ST outperforms Cascade . We attribute this improvement to two reasons. First, the end-to-end method avoids ASR error propagation, with the full-speech translation of MU-ST surpassing Cascade by 0.7 BLEU points (27.9 vs. 27.2). Second and more important, MU-ST detects MUs directly from speech, thus avoiding loss of information. The average gap between Cascade and MU-ST at five ST results is 2.9 BLEU points, much larger than that of full-speech translation (0.7). This represents that segmentation from the source speech is superior to segmentation from noisy ASR results. Accordingly, we expect our MU-ST to have greater potential based on large-scale training data. We present an adaptive speech segmentation policy for end-to-end simultaneous translation, which triggers translation with a meaningful speech unit de-10 de-10 Note that, our proposed MU-ST surpassed the cascade method ON-TRAC * in En-De experiments, but it failed to surpass Cascade * in Zh-En because the ASR training data of ON-TRAC * in En-De is only three times that of MU-ST (in hours), while the training data of Cascade * is thousands of times that of MU-ST in Zh-En experiments. tector. Experiments across two language pairs show that our method outperforms state-of-the-art methods with constrained training corpus, suggesting the effectiveness of our adaptive policy. Ablation studies reveal key factors that lead to its success, including tail-truncation, multi-modal segmentation, and speech-text pre-training.
abstain abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain objective objective objective abstain method abstain abstain other other other other other other other other other method abstain other abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain method other abstain result abstain