Simultaneous machine translation (SiMT) starts translating while receiving the streaming source inputs, and hence the source sentence is always incomplete during translating. Different from the full-sentence MT using the conventional seq-to-seq architecture, SiMT often applies prefix-to-prefix architecture, which forces each target word to only align with a partial source prefix to adapt to the incomplete source in streaming inputs. However, the source words in the front positions are always illusoryly considered more important since they appear in more prefixes, resulting in position bias , which makes the model pay more attention on the front source positions in testing. In this paper, we first analyze the phenomenon of position bias in SiMT, and develop a Length-Aware Framework to reduce the position bias by bridging the structural gap between SiMT and full-sentence MT. Specifically, given the streaming inputs, we first predict the full-sentence length and then fill the future source position with positional encoding, thereby turning the streaming inputs into a pseudo full-sentence. The proposed framework can be integrated into most existing SiMT methods to further improve performance. Experiments on two representative SiMT methods, including the state-of-the-art adaptive policy, show that our method successfully reduces the position bias and thereby achieves better SiMT performance. Simultaneous machine translation (SiMT) (Cho and Esipova, 2016; Gu et al., 2017; Ma et al., 2019; Arivazhagan et al., 2019) starts translating while receiving the streaming source inputs, which is crucial to many live scenarios, such as simultaneous interpretation, live broadcast and synchronized subtitles. Compared with full-sentence machine translation (MT) waiting for the complete source senCorresponding author: Yang Feng. tence, SiMT is more challenging since the source sentence is always incomplete during translating. To process the incomplete source, SiMT has a different architecture from full-sentence MT, as shown in Figure 1. Full-sentence MT applies the seq-to-seq architecture (Sutskever et al., 2014), where each target word can be translated based on a complete source sentence. SiMT always applies prefix-to-prefix architecture (Ma et al., 2019) to force each target word to only align with a source prefix rather than the complete source sentence, where the source prefix consists of partial source words in the front position and is monotonically non-decreasing at each step. Although the prefix-to-prefix architecture effectively adapts to the streaming inputs by removing the subsequent source words, it intensifies the structural gap between SiMT and full-sentence MT, resulting in the following issues. First, since each target word is forced to align with a monotonically non-decreasing source prefix, the source words in different positions become no longer fair. Specifically, the source words in the front position participate in more target words' translation due to earlier appearance, and hence are always illusoryly 6775 considered more important, resulting in position bias (Ko et al., 2020; Yan et al., 2021). Due to the position bias, SiMT model prefers to pay more attention to the source words in front position during testing, which not only robs the attention of the words that are supposed to be aligned (increase mis-translation error) (Zhang and Feng, 2021b), but also results in great overlap on attention distribution (aggravate the duplication translation error) (Elbayad et al., 2020). We will analyze the detailed causes and disadvantages of position bias in Sec.3. Second, prefix-to-prefix architecture directly removes the subsequent source words, resulting in the lost of some potential full-sentence information (Zhang et al., 2021). Most importantly, the prefix-to-prefix training makes the model insensitive to the full-sentence length, which can provide a global planning for translation (Feng et al., 2020, 2021). Under these grounds, we propose a Length-Aware Framework ( LAF ) for SiMT to turn the incomplete source into a pseudo full-sentence, thereby reducing the position bias. We aim to extend the incomplete source sentence in SiMT to the full-sentence length and meanwhile guarantee that future source words would not be leaked to fulfill the streaming inputs during testing. To this end, LAF first predicts the full-sentence length based on the current incomplete source sentence. Then, LAF fills the future source positions (between the current source length and predicted full-sentence length) with the positional encoding (Vaswani et al., 2017) to construct the pseudo full-sentence. Accordingly, each target word is translated based on the pseudo full-sentence and no longer forced to align with the source prefix. LAF can be integrated into most of the existing SiMT methods to further improve performance by bridging the structural gap between SiMT and full-sentence MT. We apply LAF on two representative and strong SiMT methods, and experiments on IWSLT15 En  Vi and WMT15 De  En tasks show that our method achieves better performance in both cases. target length I . Transformer (Vaswani et al., 2017) is the currently most widely used model for full-sentence MT, which consists of encoder and decoder. The encoder maps x into the source hidden states h = { h 1 ,    , h J } , and the decoder generates the i th target word y i based on source hidden states h and previous target words y <i . Overall, the decoding probability of full-sentence MT is: p full ( y | x ) = I (cid:89) i =1 p ( y i | x , y <i ) (1) Attention Transformer calculates the attention weights with dot-product attention, and the encoder-decoder cross-attention  ij is calculated based on target hidden state s i and source hidden state h j :  ij = softmax (cid:32) s i WQ (cid:0) h j WK (cid:1)   d k (cid:33) (2) where WQ and WK are input matrices, and d k is the input dimension. Positional encoding Transformer (Vaswani et al., 2017) adds positional encoding (PE) to the input embedding to capture the position information, which is fixed and only related to the absolute position. The d th dimension of the positional encoding in position pos is calculated as: PE ( pos, 2 d ) = sin (cid:16) pos/ 10000 2 d/d model (cid:17) (3) PE ( pos, 2 d +1) = cos (cid:16) pos/ 10000 2 d/d model (cid:17) (4) where d model is the dimension of input embedding. Different from full-sentence MT waiting for the complete sentence, SiMT translates concurrently with the streaming inputs and hence prefix-to-prefix architecture (Ma et al., 2019) is proposed to adapt to the incomplete source, where the target word y i is generated based on a partial source prefix. Prefix-to-prefix architecture Let g ( i ) be a monotonically non-decreasing function of i that denotes the length of received source sentence (i.e., source prefix) when translating the target word y i . Given g ( i ) , the probability of generating the target word y i is p (cid:0) y i | x  g ( i ) , y <i (cid:1) , where x  g ( i ) is first g ( i ) source words and y <i is previous target words. Overall, the decoding probability of SiMT is: p sim ( y | x ) = I (cid:89) i =1 p (cid:0) y i | x  g ( i ) , y <i (cid:1) (5) 6776 To determine g ( i ) during translating process, SiMT requires a policy to determine translating' a target word or waiting' for the next source word, falling into fixed policy and adaptive policy. Fixed policy performs waiting' or translating' according to pre-defined rules. Wait-k policy (Ma et al., 2019) is the most widely used fixed policy, which first waits for k source words and then translates one target word and waits for one source word alternately. Besides, Ma et al. (2019) also proposed a test-time wait-k policy , using a full-sentence model to perform wait-k policy in testing. Adaptive policy can dynamically adjust waiting' or translating' according to the current state. Monotonic multi-head attention ( MMA ) (Ma et al., 2020) is the current state-of-the-art adaptive policy, which predicts a Bernoulli action READ/WRITE to decide to wait for the next source word (READ) or translate a target word (WRITE). To train the Bernoulli actions, MMA predicts the writing probability of y i when receiving x j , denoted as  ij , and uses it to approximate the READ/WRITE actions during training (Arivazhagan et al., 2019). In this section, we analyze the phenomenon and cause of position bias in SiMT. In full-sentence MT, the source sentence is complete, so that each source word participates in the translation of all target words. While in prefix-to-prefix architecture for SiMT, each target word is forced to align with an increasing source prefix, which directly causes that the source words in the front position participate in the translation of more target words during training and hence are always illusoryly considered more important, resulting in position bias . A theoretical analysis of position bias refers to Appendix A. During testing, position bias is reflected in the preference of paying more attention to the source words in front positions. To explore the specific impact of position bias, we select the samples with the same source length (77 sentences) in WMT15 De  En test set as a bucket, and then calculated the average attention weight obtained by each source position in the bucket. Since the times of each source position being paid attention to may be different in SiMT, the average attention weight is averaged on the times of being attended, so the evaluation is fair for each source position. Specifically, give the attention weight  ij between target word y i and source word x j , the average attention weight 2 4 6 8 10 12 14 16 18 20 Source Position 0.00 0.05 0.10 0.15 A v e r a g e A tt e n ti on Full-sentence Wait-k MMA (b) Wait-k v.s. Test-time Wait-k Figure 2: Average attention A obtained by different source positions on the De  En task, showing wait-5, test-time wait-k, MMA and full-sentence MT. A j at source position j is calculated as: A j = (cid:80) Ii =1  ij (cid:80) Ii =1 1 j  g ( i ) (6) where (cid:80) Ii =1  ij is the sum of attention on the j th source position, and (cid:80) Ii =1 1 j  g ( i ) counts the times of the j th source position being paid attention to. What is position bias? Figure 3(a) shows the average attention obtained by different source positions 1 in two representative SiMT methods, compared with full-sentence MT. SiMT has a significant difference from the full-sentence MT on the average attention to the source position. In full-sentence MT, the average attention on each position is similar and the back position gets slightly more attention (Voita et al., 2021). However, in both the fix and adaptive policy in SiMT, the front source positions obviously get more attention due 1 Note that we do not add  bos  in front of the source sentence, and the word in the first source position is x 1 . to position bias, especially the first source word. Compared with wait-k, MMA alleviates the position bias by dynamically adjusting waiting' or translating', but the first source position still abnormally gets more attention. Note that the average attention on the back positions in SiMT is higher since the times they are attended are less (the denominator in Eq.(6) is smaller). Specific attention characteristics Furthermore, we compare the characteristics of attention distribution in full-sentence MT and SiMT, shown in Figure 3. In SiMT, more attention weights are concentrated on the front source positions (Arivazha-gan et al., 2019; Zhang and Feng, 2022a), which is not conducive to translation. First, the biased attention on front positions robs the attention of the aligned source word, resulting in mis-translation error. Second, much overlapping on attention distribution aggravates the duplication translation error, where a human evaluation proposed by Elbayad et al. (2020) shows that duplication error in SiMT is 500% of full-sentence MT. Besides, in some cases, even if the aligned source words have not been received, the prefix-to-prefix architecture still forces the target word to align with the irrelevant source prefix, resulting in the confusion on attention (Chen et al., 2021). Does position bias affect SiMT performance? To analyze whether the position bias in SiMT results in poor translation quality, we use the ratio of the average attention on the first source position to all positions ( A 1 / (cid:80) j A j ) to reflect the degree of position bias, and accordingly divide WMT15 De  En test set into 5 parts evenly. We report the translation quality of these 5 parts in Figure 4, Bottom Bottom-Mid Mid Top-Mid Top Degree of Position Bias (weak heavy) 20 25 30 35 BLEU 32.32 32.65 32.99 31.48 29.84 27.94 28.71 28.81 26.96 19.99 4.38 3.94 4.18 4.52 9.85 Full-sentence Wait-k (a) Divided based on position bias degree in wait-k. (b) Divided based on position bias degree in MMA. where the position bias is heavier from Bottom' to Top'. The translation quality of both wait-k and MMA significantly decrease as the position bias becomes heavy, while full-sentence MT remained high-quality translation on these parts. More importantly, as the position bias intensifies, the performance gap between SiMT and full-sentence MT is amplified, where wait-k and MMA are 9.85 BLEU and 7.03 BLEU lower than full-sentence MT respectively on the Top' set. Therefore, the position bias is an important cause of the performance gap between SiMT and full-sentence MT. What is the position bias caused by? To verify that the preference for front source positions is caused by the structural gap between SiMT and full-sentence MT rather than streaming inputs during testing, we compare the average attention of wait-k and test-time wait-k' in Figure 3(b), where test-time wait-k' is trained with full-sentence structure and tested with wait-k policy. After replacing the prefix-to-prefix architecture with the seq-to-seq architecture during training, the position bias in the test-time wait-k' is significantly weakened, which shows that prefix-to-prefix training is the main cause of position bias. However, directly training with full-sentence structure leaks many future source words, where the obvious training-testing mismatch results in inferior translation quality of test-time wait-k' (Ma et al., 2019). In practice, prefix-to-prefix architecture forces the target word to assign attention to the prefix even if its corresponding source word has not been read in, which will undoubtedly cause the attention to become chaotic and tend to be distributed to the front position. This also explains why the position bias is more serious in the fixed policy, since the read/write cannot be adjusted, in more cases the prefix does not contain the corresponding source word but is forced to pay attention to. Besides, prefix-to-prefix architecture increases the frequency of front source positions during training, and previous works (Zhou and Liu, 2006; Luong et al., 2015; Gu et al., 2020) show that NMT models have a tendency towards over-fitting on high-frequency words, resulting in the position bias. Based on the preliminary analyses on position bias, we hope that in SiMT, target words can also align with the reasonable source positions as them in full-sentence MT, including the future positions even though the words on these positions have not yet been received. Along this line, we develop a Length-Aware Framework ( LAF ) to turn the streaming inputs into pseudo full-sentence and thereby allow the target words to align with the full-sentence positions rather than a prefix, as shown in Figure 5. The details are introduced following. Length prediction To turn the incomplete source into pseudo full-sentence, full-sentence length is an essential factor. Therefore, at step i , LAF predicts the full-sentence length L i based on the received source sentence x  g ( i ) , through a classification task. Note that the predicted length dynamically updates with the increase of received source words. Formally, the probability of full-sentence length L i is predicted through a multi-layer perceptron (MLP) based on the received source words: p l (cid:0) L i | x  g ( i ) (cid:1) =softmax (cid:0) W tanh (cid:0) V h  g ( i ) (cid:1)(cid:1) (7) where h  g ( i ) = 1 g ( i ) (cid:80) g ( i ) j =1 h j is the the mean of hidden states of the currently received source words. V  R d model  d model and W  RN max  d model are the parameters of MLP, where N max is the max length of the source sentence in the corpus. Note that softmax(  ) is normalized on all possible length values. In testing, the value with the highest probability is selected as the full-sentence length. If source sentence is already complete (receiving  eos  ) or the predicted length L i is not larger than the received source length ( L i  g ( i ) ), we use the current length g ( i ) as the full-sentence length. Pseudo full-sentence Given the predicted full-sentence length, we fill the future source position ( g ( i ) , L i ] with positional encoding to construct the pseudo full-sentence. Formally, given the hidden states of received source word h  g ( i ) and the predicted full-sentence length L i , the pseudo full-sentence hidden states (cid:101) h ( i ) at step i is: (cid:101) h ( i ) = (cid:0) h 1 ,    , h g ( i ) , PE g ( i )+1 ,    , PEL i (cid:1) (8) Note that pseudo full-sentence is constructed at the hidden states level, so there is no need to recompute the source hidden states. Then, the target word y i is generated based on the pseudo full-sentence hidden states (cid:101) h ( i ) , and hence cross-attention  ij in Eq.(2) can be assigned to future positions, rewritten as:  ij = softmax   s i WQ (cid:16)(cid:101) h ( i ) j WK (cid:17)   d k   (9) Overall, the decoding probability of the length-aware framework is: p laf ( y | x ) = I (cid:89) i =1 p l (cid:0) L i | x  g ( i ) (cid:1)  p (cid:0) y i | x  g ( i ) , y <i , L i (cid:1) (10) 4.2 Training Objective The length-aware framework consists of a length prediction module and a translation module. For the length prediction module, we take the complete source length J as the ground-truth length label and train the model with cross-entropy loss: L len =  I (cid:88) i =1 log p l (cid:0) J | x  g ( i ) (cid:1) (11) 6779 For the translation module, we complement the source prefix to the ground-truth source length J with positional encoding and train the translation module by minimizing the cross-entropy loss: L ce =  I (cid:88) i =1 log p (cid:0) y i | x  g ( i ) , y <i , J (cid:1) (12) where y  is the ground-truth target sentence. During testing, we apply the predicted full-sentence length to complement the source prefix. We will compare the performance of training with ground-truth or predicted full-sentence length in Sec.7.1. Finally, the total loss of LAF is calculated as: L laf = L ce + L len (13) 4.3 Integrated into SiMT Policy The length-aware framework can be integrated into most existing SiMT methods. We take wait-k and MMA as representatives to introduce the slight difference when integrated to fix and adaptive policy respectively. LAF predicts the full-sentence length based on the currently received source words x  g ( i ) , so the key is to calculate g ( i ) , which may be different in fix and adaptive policy. Fixed policy Since wait-k is a pre-defined fixed policy, g wait  k ( i ) in wait-k during both training and testing is invariably calculated as: g wait  k ( i ) = min { k + i  1 , J } (14) Adaptive policy Since MMA can dynamically predict READ/WRITE actions, the calculation of g ( i ) during training and testing is different. During testing, we take the number of source words received by the model when starting to translate y i as g ( i ) . During training, MMA does not have explicit READ/WRITE actions, but predicts the writing probability  ij , where  ij represents the probability of translating y i after receiving source word x j . Therefore, we select the position of x j with the highest writing probability as g mma ( i ) : g mma ( i ) = argmax j  ij (15) 5 Related Work The main architectures of SiMT model are divided into two categories: seq-to-seq architecture and prefix-to-prefix architecture. The early SiMT methods always used a full-sentence MT model trained by seq-to-seq architecture to translate each segment divided by the SiMT policy (Bangalore et al., 2012; Cho and Es-ipova, 2016; Siahbani et al., 2018). Gu et al. (2017) used reinforcement learning to train an agent to decide whether to start translating. Alinejad et al. (2018) added a predict operation based on Gu et al. (2017). Zhang et al. (2020b) proposed an adaptive segmentation policy based on meaning units. However, the mismatch between training and testing usually leads to inferior translation quality. The recent SiMT methods, including fix and adaptive policies, mainly used prefix-to-prefix architecture. For the fixed policy, Ma et al. (2019) proposed a wait-k policy, which always translates k words behind the source words. Zhang and Feng (2021a) proposed a char-level wait-k policy. Zhang and Feng (2021c) proposed a universal SiMT with the mixture-of-experts wait-k policy. For the adaptive policy, Zheng et al. (2019a) trained an agent with the golden read/write action sequence. Zheng et al. (2019b) added a delay token and introduced limited dynamic prediction. Arivazhagan et al. (2019) proposed MILk, using a Bernoulli variable to determine whether to write. Ma et al. (2020) proposed MMA to implement MILK on the Transformer. Wilken et al. (2020) and Zhang and Feng (2022b) proposed alignment-based SiMT policy. Liu et al. (2021a) proposed cross-attention augmented transducer for SiMT. Zhang et al. (2021) and Alinejad et al. (2021) introduced a full-sentence model to guide SiMT policy. Miao et al. (2021) proposed a generative SiMT policy. Although the prefix-to-prefix architecture simulates the streaming inputs, it brings the position bias described in Sec.3. Therefore, we proposed a length-aware framework to reduce the position bias and meanwhile fulfill the streaming inputs. IWSLT15 2 English  Vietnamese (En  Vi) (133K pairs) (Cettolo et al., 2015) We use TED tst2012 as validation set (1553 pairs) and TED tst2013 as test set (1268 pairs). Following the previous setting (Raffel et al., 2017; Ma et al., 2020), we replace words that the frequency less than 5 by  unk  , and the vocabulary sizes are 17K and 7.7K for English and Vietnamese respectively. WMT15 3 German  English (De  En) (4.5M 2 nlp.stanford.edu/projects/nmt/ 3 www.statmt.org/wmt15/translation-task 6780 2 4 6 8 10 Average Lagging(AL) 26 27 28 29 BLEU Full-sentence Wait-k + LAF Wait-k MMA + LAF MMA pairs) Following Ma et al. (2019), Arivazhagan et al. (2019) and Ma et al. (2020), we use new-stest2013 as validation set (3000 pairs) and new-stest2015 as test set (2169 pairs). BPE (Sennrich et al., 2016) was applied with 32K merge operations and the vocabulary is shared across languages. Transformer (Vaswani et al., 2017). Wait-k Wait-k policy proposed by Ma et al. (2019), the most widely used fixed policy, which first waits for k source words and then translates a target word and waits for a source word alternately. MMA 4 Monotonic multi-head attention (MMA) proposed by (Ma et al., 2020), the SOTA adaptive policy. At each step, MMA predicts a Bernoulli variable to decide whether to start translating. * + LAF Applying proposed length-aware framework on Wait-k or MMA. The implementation of all systems are adapted from Fairseq Library (Ott et al., 2019) based on Transformer (Vaswani et al., 2017) with the same setting in Ma et al. (2020). For En  Vi, we apply Transformer-small (4 heads). For De  En, we apply Transformer-Base (8 heads) and Transformer-Big (16 heads). We evaluate these systems with BLEU (Papineni et al., 2002) for translation quality and Average Lagging (AL) (Ma et al., 2019) for latency. AL is calculated based on g ( i ) : AL = 1   (cid:88) i =1 g ( i )  i  1 I/J (16) 4 github.com/pytorch/fairseq/tree/ master/examples/simultaneous_translation Train Test AL BLEU LAF GT Pred 4.11 28.34 Pred LAF Pred Pred 4.07 28.21 Oracle LAF GT GT 3.93 28.37 Table 1: An ablation study of using predicted full-sentence length (Pred) or ground-truth source length (GT) in training and testing respectively, where the results are based on the wait-5 policy. Figure 6 shows the performance improvement that LAF brings to Wait-k and MMA, where our method achieves higher translation quality under all latency. LAF has a more significant improvement on the fixed policy Wait-k, improving about 0.28 BLEU on En  Vi, 1.94 BLEU on De  En(Base), 1.50 BLEU on De  En(Big), which is because the position bias in original wait-k is more serious. Compared with the SOTA adaptive policy MMA, our method also performs better and is much closer to full-sentence MT performance. We conduct extensive analyses to understand the specific improvements of our method. Unless otherwise specified, all the results are reported on De  En(Base) and tested with wait-5 (AL=4.10) and MMA (AL=4.57) under similar latency. We use ground-truth full-sentence length to train the translation module, and use the predicted full-sentence length in testing. We conduct the ablation 6781 0 2 4 6 8 10 12 Average Lagging(AL) 15 20 25 30 35 40 A cc u r ac y ( % ) Wait-k + LAF MMA + LAF study of using predicted full-sentence length (Pred) or ground-truth length (GT) for translation in training and testing respectively, reported in Table 1. LAF has a better performance than Pred LAF', indicating that using ground-truth length during training is more helpful for learning translation. Compared with Oracle LAF' that uses ground-truth full-sentence length in testing, LAF achieves comparable performance, which shows that the length prediction module in LAF performs well. Figure 7(a) shows the prediction accuracy of the full-sentence length in LAF, indicating that our method achieves good prediction performance. As the latency increases, the prediction accuracy of both Wait-k+LAF' and MMA+LAF' gradually increases. Specifically, Wait-k+LAF' predicts more accurately at low latency, which shows that the regular form of fixed policy is more conducive for LAF to learn the full-sentence length. Besides, in Figure 7(b), with the continuous increase of received source words, the predicted full-sentence length is updated in real time and the prediction accuracy gradually improves, which is in line with our expectations. We show the change of average attention 5 after applying LAF in Figure 8. With LAF, the position bias in SiMT is significantly reduced, where the front positions are no longer illusoryly considered more important. By constructing the pseudo full-sentence, LAF bridges the structural gap between SiMT and full-sentence MT, so that the importance of source positions are more similar to that in full-sentence MT, thereby reducing the position bias. Position bias makes the target word tend to focus on the front source word, which leads to much overlap in the attention distribution, resulting in duplicate translation errors (Elbayad et al., 2020). Following See et al. (2017), we count the n-grams duplication proportion in translation in Figure 10. There are few duplicate n-grams in reference and full-sentence MT, especially when n> 2 . However, position bias in SiMT makes the model always focus on some particular source words in the front position, thereby exacerbating duplicate translation errors, especially in the fixed policy. In 3-grams, the duplicate translation of Wait-k is about 6 times that of full-sentence MT, which is in line with the previous conclusion (Elbayad et al., 2020). After applying LAF, the duplicate translation in SiMT is significantly reduced, similar to full-sentence MT. 7.5 Improvement on Various Difficulty Levels The word order difference is a major challenge of SiMT, where many word order inversions may force the model to start translating before reading the aligned source words (Chen et al., 2021). Following Zhang and Feng (2021c), We evenly divide the test set into three sets: Easy, Mid and Hard based on the number of reversed word orders 6782 e i n e ( _ a ) n e u e ( _ n e w ) R un d e ( _ r o un d ) i n d i @@ r e k @@ t e r ( _ i n d i r e c t ) G e s p r  c h e ( _ t a l k s ) w i r d ( _ w ill ) v o r a u ss i c h t li c h ( _ li k e l y ) n o c h ( _ s t ill ) i n d i e s e m ( _ t h i s ) M o n a t ( _ m o n t h ) i n  g y p t e n ( _ E g y p t ) b e g i nn e n ( _ b e g i n ) . in alignments using fast-align 6 (Dyer et al., 2013), and report the results on each set in Table 2. For full-sentence MT, word order reversal will not cause too much challenge, so that the performance gap between different sets is small. In SiMT, word order reversal often causes the model to translate before reading the aligned source words, which forces the target word to focus on some unrelated source words, resulting in poor performance in Hard set. LAF complements the incomplete source to the full-sentence length, which allows the target word to focus on the subsequent position instead of must focusing on the current irrelevant source word when the aligned word is not received, thereby obviously improving the performance on Hard set. LAF constructs the pseudo full-sentence by predicting the full-sentence length and filling the future position with positional encoding. To verify the importance of the future position, we count the attention weights on the future position (i.e., filled with positional encoding) at each decoding step in Figure 11. In the beginning, the future posi-6 https://github.com/clab/fast_align 5 10 15 20 25 30 35 40 Decoding Step 0.0 0.1 0.2 0.3 0.4 0.5 A tt e n ti on on F u t u r e S ou r ce P o s iti on Wait-k + LAF MMA + LAF Figure 11: The attention on future source position (filled with positional encoding) in different decoding steps. tion gets much attention weight, especially getting about 30% attention in the first decoding step. As the received source words increase, the attention received by future positions gradually decreases. Furthermore, we visualize the attention distribution of an example in Figure 9. In Wait-k and MMA, attention is more concentrated on the front position, especially Wait-k extremely focuses on the first source word, which leads to duplicate translation  expected to to hold . With LAF, when the aligned source word has not been received, the future positions tend to get more attention, e.g. when Wait-k+LAF' translating  take place  before receiving  beginnen . Besides, the predicted length in LAF changes dynamically and gradually approaches the full-sentence length. Overall, LAF reduces the position bias and thus the attention in SiMT is more similar to the attention in full-sentence MT, resulting in better translation quality. In this paper, we develop a length-aware framework for SiMT to reduce the position bias brought by incomplete source. Experiments show that our method achieves promising results by bridging the structural gap between SiMT and full-sentence MT. 6783 Acknowledgements We thank all the anonymous reviewers for their insightful and valuable comments. This work was supported by National Key R&D Program of China (NO. 2017YFE0192900).
abstain abstain abstain objective abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain objective objective abstain abstain abstain result other other other other other other other other other other other other other other other other method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain other abstain abstain abstain method method other abstain other method method method abstain method abstain other method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective result abstain