Document-level neural machine translation (NMT) has proven to be of profound value for its effectiveness on capturing contextual information. Nevertheless, existing approaches 1) simply introduce the representations of context sentences without explicitly characterizing the inter-sentence reasoning process; and 2) feed ground-truth target contexts as extra inputs at the training time, thus facing the problem of exposure bias. We approach these problems with an inspiration from human behavior  human translators ordinarily emerge a translation draft in their mind and progressively revise it according to the reasoning in discourse. To this end, we propose a novel Multi-Hop Transformer (MHT) which offers NMT abilities to explicitly model the human-like draft-editing and reasoning process. Specifically, our model serves the sentence-level translation as a draft and properly refines its representations by attending to multiple antecedent sentences iteratively. Experiments on four widely used document translation tasks demonstrate that our method can significantly improve document-level translation performance and can tackle discourse phenomena, such as coreference error and the problem of polysemy. Neural machine translation (NMT) employs an end-to-end framework (Sutskever et al., 2014) and has advanced promising results on various sentence-level translation tasks (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017; Wan et al., 2020). However, most of NMT models handle sentences independently, regardless of the linguistic context that may appear outside the current sentence (Tiedemann and Scherrer, 2017a). This makes NMT insufficient to fully resolve the typical context-dependent phenomena problematic,  These authors contributed equally to this work. e.g. coreference (Guillou, 2016), lexical cohesion (Carpuat, 2009), as well as lexical disambiguation (Gonzales et al., 2017). Recent studies (Tu et al., 2018; Maruf and Haffari, 2018; Maruf et al., 2019; Tan et al., 2019; Kim et al., 2019; Zheng et al., 2020; Chen et al., 2020; Sun et al., 2020; Ma et al., 2020) have proven to be effective on tackling discourse phenomena via feeding NMT with contextual information, e.g. source-side (Wang et al., 2017; Voita et al., 2018; Zhang et al., 2018) or target-side context sentences (Baw-den et al., 2018; Miculicich et al., 2018). Despite their successes, these methods simply merge the representations of context sentences together, lacking a mechanism to explicitly characterize the inter-sentence reasoning upon the context. Another shortage in existing document-level NMT is the problem of exposure bias. Most of methods utilized the ground-truth target context for training but the generated translations for inference, leading to inconsistent inputs at training and testing time (Ranzato et al., 2015; Koehn and Knowles, 2017). Intuitively, human translators tend to acquire useful context information from the reasoning process among sentences, thus figuring out the correct meaning when they encounter ambiguity during translation. Sukhbaatar et al. (2015) and Shen et al. (2017) empirically verified that modeling multi-hop reasoning among sentences benefits to the language understanding task, e.g text comprehension. Voita et al. (2019) showed that document-level NMT model can profit from relative positions with respect to context sentences, which to some extent confirms the importance of the relationship among sentences. Meanwhile, Xia et al. (2017) demonstrated that sentence-level NMT could be improved by a two-pass draft-editing process, of which the second-pass decoder refines the target sentence generated by a first-pass standard decoder. document-level NMT using a novel framework  Multi-Hop Transformer, which imitates draft-editing and reasoning process of human translators. Specifically, we implement an explicit reasoning process by exploiting source and target antecedent sentences with concurrently stacked attention layers, thus performing the progressive refinement on the representations of the current sentence and its translation. Besides, we leverage the draft to present context information on the target side during both training and testing, alleviating the problem of exposure bias. We conduct experiments on four widely used document translation tasks: English-German and Chinese-English TED, English-Russian Opensubtitles, as well as English-German Europarl-7 datasets. Experimental results demonstrate that our method significantly outperforms both context-agnostic and context-aware methods. The qualitative analysis confirms the effectiveness of the proposed multihop reasoning mechanism on resolving many linguistic phenomena, such as word sense disambiguation and coreference resolution. Our contributions are mainly in:  We propose the Multi-Hop Transformer. To the best of our knowledge, this is the first pi-oneer investigation that introduces multi-hop reasoning into document-level NMT. The proposed model takes target context drafts into account at the training time, which devotes to avoid the training-generation discrepancy. Our approach significantly improves document-level translation performance on four document-level translation tasks in terms of BLEU scores and solves some context-dependent phenomena, such as coreference error and polysemy. Transformer NMT is an end-to-end framework to build translation models. Vaswani et al. (2017) propose a new architecture called Transformer which adopts self-attention network for both encoding and decoding. Both its encoder and decoder consist of multiple layers, each of which includes a multi-head self-attention and a feed-forward sublayer. Additionally, each layer of the decoder ap-plys a multi-head cross attention to capture information from the encoder. Transformer has shown superiority in a variety of NLP tasks. Therefore, we construct our models upon this advanced architecture. Document-level NMT In order to correctly translate the sentence with discourse phenomena, NMT models need to look beyond the current sentence and integrate contextual sentences as auxiliary inputs. Formally, let X = ( x 1 , x 2 , ..., x I ) be a source-language document composed of I sentences, where x i = ( x i 1 , x i 2 , ..., x iN ) denotes the i th sentence containing N words. Correspondingly, the target-language document also consists of I sentences, Y = ( y 1 , y 2 , ..., y I ) , where y i = ( y i 1 , y i 2 , ..., y iM ) denotes the i th sentence involving M words. Document-level NMT incorporates contextual information from both source side and target side to autoregressively generate the best translation result that has highest probability: P  ( y i | x i ) = M (cid:89) m =1 P  ( y im | y i<m , x i , X  i , Y  i ) (1) where y i<m is the sequence of proceeding tokens before position m . Related Work Several studies have explored multi-input models to leverage the contextual information from source-side (Jean et al., 2017; Kuang and Xiong, 2018) or target-side sentences (Kuang et al., 2018; Miculicich et al., 2018). For the former, Zhang et al. (2018) propose a new encoder to represent document-level context from previous source-side sentences . Tiedemann and Scherrer (2017b) and Junczys-Dowmunt (2019) utilize the concatenation of previous source-side sentences as input, while Voita et al. (2018) make use of gate mechanism to balance the weight between current source sentence and its context. For the latter, Miculicich et al. (2018) propose a hierarchical attention (HAN) framework to capture the target contextual information in the decoder. Bawden et al. (2018), Maruf and Haffari (2018) and Maruf et al. (2019) take both source-side and target-side context into account. Motivation As seen, both of the existing methods simply introduce the context sentences without explicitly characterizing the inter-sentence reasoning. Intuitively, when humans have difficulty in translation like encountering ambiguity phenomenon, they could acquire more information Source-side Context Encoder Source Embedding K-th Hop Attention Layer Second Hop Attention Layer First Hop Attention Layer Self-AttentionLayer  Source-side Context Encoder Source-side Context Encoder Target Embedding K-th Hop Attention Layer Second Hop Attention Layer First Hop Attention Layer Draft-Attention Layer  1  Encoder-Decoder Attention Layer + 1  + Target-side Context Encoder Self-AttentionLayer Target-side Draft Encoder Softmax    +       +   6  6  Translation Source Sentence i Target Sentence i Draft Sentence Context Draft Sentence i-1 Context Draft Sentence i-k+1 Context Draft Sentence i-k Source Context Sentence i-1 Source Context Sentence i-k+1 Source Context Sentence i-k Source-side Sentence Encoder Multi-Hop Encoder Multi-Hop Decoder Target-side Sentence Encoder Target-side Context Encoder Target-side Context Encoder       Figure 1: Illustration of Multi-Hop Transformer. from the contexts sentence by sentence and then perform reasoning to figure out the exact meaning. We attribute that such reasoning process is also beneficial to machine translation task. Recent successes in text comprehension communities have to some extent supported our hypothesis (Hill et al., 2015; Kumar et al., 2016). For example, Sukhbaatar et al. (2015) propose a multi-hop end-to-end memory network, which can renew the query representation with multiple computational steps (which they term hops). Dhingra et al. (2016) extend an attention-sum reader to multi-turn reasoning with a gating mechanism. In addition, Shen et al. (2017) introduce multi-hop attention, which used multiple turns to effectively exploit and reason over the relation among queries and documents. In this paper, we propose to bring the idea of multi-hop into document translation and aim at mimicking the multi-step comprehension and revising process of human translators. Contrast with those models for text comprehension which scan the query and document for multiple passes, our model iteratively focuses on different context sentences, which captures the inter-sentence reasoning semantics of contextual sentences to incrementally refine the representation of current sentence. With this mind, we propose a novel method called Multi-Hop Transformer, which models the reasoning process among multiple contextual sentences in both source side and target side. The source-side contexts are directly acquired from the document. The target-side contexts, called target-side drafts in this paper, are generated by a sentence-level NMT model. These contexts are fed into the Multi-Hop Transformer with pre-trained encoders. The overall architecture of our proposed model is illustrated in Figure 1, which consists of three components:  Sentence Encoder: This component contains two pre-trained encoders, one of which is called source-side sentence encoder and the other is called target-side sentence encoder. These encoders generate representations for source-side contexts and target-side drafts respectively. Multi-Hop Encoder: We extend the original Transformer encoder with a novel multi-hop encoder to efficiently perform sentence-by-sentence reasoning on source-side contexts and generate the representation for the current sentence. Multi-Hop Decoder: Similarly, a multi-hop decoder is proposed to acquire information from the target-side drafts and models the translation probability distribution. We use multi-layer and multi-head self-attention architecture (Vaswani et al., 2017) to obtain the representations for source-side contexts and target-side drafts. Similar to the encoder of Transformer, sentence encoder contains a stack of six identical layers, each of which consists of two sub-layers. The first sub-layer is a multi-head attention( Q , K , V ), which takes a query Q , a key K and a value V as inputs. The second sub-layer is a fully connected feed-forward network (FFN). source-side contexts, as shown in Figure 1. For the current sentence s = x i to be translated, we use the previous sentences X  i = ( x i  k , x i  k +1 , ..., x i  1 ) in the same document as the source-side context, specially denoted as c i  k s , c i  k +1 s , ..., c i  1 s for clarity. k is the context window size. For the j th context, we obtain the A ( n ) c i  j s which denotes the n th hidden layer representation of c i  j s as follows: A ( n ) c i  j s = MHA( H ( n  1) c i  j s , H ( n  1) c i  j s , H ( n  1) c i  j s ) (2) where n = 1 , 2 , ..., 6 . MHA represents the standard Multi-Head Attention function (Vaswani et al., 2017). j denotes the distance between the context sentence and current sentence. Target-Side Sentence Encoder. Most existing works use ground-truth target-side contexts as the input of decoder during training (Voita et al., 2019). However, the target contexts at training and testing are drawn from different distributions, leading to the inconsistency between training and testing. To alleviate this problem, we instead make use of target-side context drafts generated from a pre-trained sentence-level translation model. Similar to source-side sentence encoder, this target-side context draft encoder is used to obtain the context representation A ( n ) c i  j t of the j th target-side draft c i  j t . Besides, we obtain a draft translation d of the current sentence from the pre-trained sentence-level translation model and use a target-side draft encoder to obtain the representation A ( n ) d . The multi-hop encoder contains a stack of 6 identical layers, each of which contains the following sub-layers: Self-Attention Layer. The first sub-layer makes use of multi-head self-attention to encode the information of current source sentence s and obtains the representation A ( n ) s . Multi-Hop Attention Layer. The second sublayer uses a multi-hop attention to perform sentence-by-sentence reasoning on c s in sentence order as shown in Figure 1. Each reasoning step, also called a hop , is implemented by a multi-head attention layer. The first hop takes representation A ( n ) s as the query and the representation A ( n ) c i  k s of the previous k th sentence as the key and value. B ( n ) s i  k = MHA( A ( n ) s , A ( n ) c i  k s , A ( n ) c i  k s ) (3) The other hops are implemented: B ( n ) s i  j = MHA( B ( n ) s i  j  1 , A ( n ) c i  j s , A ( n ) c i  j s ) (4) where j = k  1 , k  2 , ..., 1 . j denotes the distance between the context sentence and current sentence. Context Gating. The information of current source sentence is crucial in translation while the contextual information is auxiliary. In order to avoid excessive utilization of contextual information, a context gating mechanism (Tu et al., 2017; Yang et al., 2017, 2019) is introduced to dynamically control the weight between context sentences and current sentence:  =  ( W a A ( n ) s + W b B ( n ) s i  1 ) , (5) where  is the logistic sigmoid function and  is the context gate. Similarly, the multi-hop decoder involves a stack of 6 identical layers. Each of them contains five sub-layers. Self-Attention Layer. The first sub-layer utilizes multi-head self-attention to encode the information of current target sentence t and obtains the representation A ( n ) t . Draft-Attention Layer. Inspired by Xia et al. (2017), we introduce the complete draft d translated from current source sentence by a sentence-level NMT. Then this draft representation A ( n ) d is encoded by the target-side draft encoder in Section 3.1. The draft attention is achieved by multi-head attention: F ( n ) t = MHA( A ( n ) t , A ( n ) d , A ( n ) d ) . Multi-Hop Attention Layer. Similar to the encoder, a multi-hop reasoning process is performed on the target-side contexts. The target-side drafts are generated from corresponding source sentences by a pre-trained sentence-level NMT model. The first hop takes representation F ( n ) t as the query and the representation A ( n ) c i  k t of the previous k th draft as the key and value. The other hops are achieved: where  is used to regulate the weight of target-side contextual information. Encoder-Decoder Attention Layer. Finally, we use an encoder-decoder attention layer to integrate the output of multi-hop encoder Enc s with the current target representation G ( n ) t . To evaluate the effectiveness of the proposed MHT, we conduct experiments on four widely used document translation tasks, including the TED Talk (Cettolo et al., 2012) with two language pairs, Opensubtitles (Maruf et al., 2018) and Europarl7 (Maruf et al., 2018). All datasets are tokenized and truecased with the Moses toolkit (Koehn et al., 2007), and splited into sub-word units with a joint BPE model (Sennrich et al., 2016) with 30K merge operations. The datasets are described as follows:  TED Talk (English-German) : We use the dataset of IWSLT 2017 MT English-German track for training, which contains transcripts of TED talks aligned at sentence level. dev2010 is used for development and tst2016-2017 for evaluation. Statistically, there are 0.21M sentences in the training set, 9K sentences in the development set, and 2.3K sentences in the test set. TED Talk (Chinese-English) : We use the corpus consisting of 0.2M sentence pairs extracted from IWSLT 2014 and 2015 Chinese-English track for training. dev2010 involves 0.8K sentences for development and tst2010-2013 contains 5.5K sentences for test. Opensubtitles (English-Russian) : We make use of the parallel corpus from Maruf et al. (2018). The training set includes 0.3M sentence pairs. There are 6K sentence pairs in development set, and 9K in test set. Europarl7 (English-German) : The raw Eu-roparl v7 corpus (Koehn, 2005) contains SPEAKER and LANGUAGE tags where the latter indicates the language the speaker was actually using. We process the raw data and extract the parallel corpus as same as Maruf Method TED Opensubtitles Europarl7 Params AVG En  De Zh  En En  Ru En  De Transformer (cid:63) 24.55 18.36 19.46 30.18 50M 23.14 CA-Transformer  25.04 18.77 20.21 30.67 72M 23.67 (Maruf et al., 2018)  -19.13 (cid:5) 26.49 (cid:5) -CA-HAN  25.70 18.79 20.08 26.61 70M 22.79 (Maruf et al., 2019)  24.62 (cid:5) -54M (cid:5) CADec  26.08 19.01 19.46 30.36 91M 23.98 MHT (Ours)  26.22 19.52 20.46 31.25 80M 24.36 Table 1: BLEU scores on TED Talk, Opensubtitles and Europarl7 tasks. et al. (2018). 0.1M sentence pairs are used for training, 3K sentence pairs for development, and 5K sentence pairs for evaluation. context-agnostic NMT model (Vaswani et al., 2017). CA-Transformer : A context-aware transformer model (CA-Transformer) with an additional context encoder to incorporate document contextual information into model (Zhang et al., 2018). CA-HAN : A context-aware hierarchical attention networks (CA-HAN) which integrate document contextual information from both source side and target side (Miculicich et al., 2018). CADec : A two-pass machine translation model (Context-Aware Decoder, CADec) which first produces a draft translation of the current sentence, then corrects it using context (Voita et al., 2019). Our model is implemented on the open-source toolkit Thumt (Zhang et al., 2017). Adam optimizer (Kingma and Ba, 2014) is applied with an initial learning rate 0.1. The size of hidden dimension and feed-forward layer are set to 512 and 2048 respectively. Encoder and decoder have 6 layers with 8 heads multi-head attention. Dropout is 0.1 and batch size is set to 4096. Beam size is 4 for inference. Translation quality is evaluated by the traditional metric BLEU (Papineni et al., 2002) on tokenized text. Context window size is set to 3, consistent with the experiments in Section 5.2. To initialize the source-side sentence encoder in Section 3.1, a sentence-level NMT model is trained from source language to target language using the corresponding datasets without additional corpus. The encoder of this trained model is used to initialize the source-side context encoder. Also, we utilize the trained model to translate the source-side sentences and obtain the target-side drafts. Similarly, we train a sentence-level model from target language to source language to initialize the target-side encoders in Section 3.1. In order to reduce the computational overhead, we share the parameters among the sentence encoders on the same side. The settings of these two sentence-level NMT models are consistent with our baseline Transformer model. Table 1 summarizes the BLEU scores of different systems on four tasks. As seen, our baseline and re-implemented existing methods outperform the reported results on the same data, which we believe makes the evaluation convincing. Clearly, our model MHT significantly improves translation quality in terms of BLEU on these tasks, and obtains the best average results that gain 0.38, 0.69 and 1.57 BLEU points over CADec, CA-Transformer and CA-HAN respectively. These results demonstrate the universality and effectiveness of the proposed approach. Moreover, without in-0 1 2 3 4 window size 25.4 25.6 25.8 26.0 26.2 BLEUTED (en-de) 0 1 2 3 4 window size 18.8 18.9 19.0 19.1 19.2 19.3 19.4 19.5 BLEUTED (zh-en) Figure 2: The performance of the MHT model on TED (En-De) and TED (Zh-En) translation task using different context window sizes. troducing large-scale pre-trained language models, our translation systems achieve new state-of-the-art translation qualities across three examined translation tasks, which are TED (En-De), Opensubtitles (En-Ru) and Europarl7 (En-De). Overall, our experiments indicate the following two points: 1) explicitly modeling underlying reasoning semantics by a multi-hop mechanism indeed benefits neural machine translation, and 2) the improvements of our model are not from enlarging the network. In this section, to gain further insight, we explore the effectiveness of several factors of our model, including 1) multi-hop attention; 2) context window size; 3) reasoning direction; 4) sides for introducing context; and 5) target contexts. Moreover, we show qualitative analysis on discourse phenomena to better understand the advantage of our model. To further investigate the effect of multi-hop reasoning, we compare our multi-hop attention with two baseline context modeling methods, including Concat and Hierarchical Attention. Table 2 shows the results of three different context modeling modules on TED, which use same inputs containing original training data and drafts. Concat denotes the MHT model simply using the concatenation of the three context sentences representations to get the final context representation. Hierarchical Attention denotes the MHT model with a hierarchical attention to model context, which consists of a sentence-level attention and a token-level attention to capture information from the appropriate context sentences and tokens, as in Miculicich et al. (2018). As depicted in Table 2, we replace multi-hop attention with these two baseline modules for experiments. Hierarchical Attention slightly outperforms Concat, while multi-hop attention leads both of them by a much larger margin. The results demonstrate that multi-hop attention is capable of providing a more fine-grained representation of reasoning state over context and consequently capturing context semantic information more accurately. methods. As shown in Figure 2, we conduct experiments with different context window sizes to explore its effect. When the window size is less than 4, the model obtains more information from contexts and achieves better performance as the window size gets larger. However, when window size is increased to 4, we find that the performance doesn't improve further, but decreases slightly. This phenomenon shows that contexts far from the target sentence may be less relevant and cause noise (Kim et al., 2019). Therefore, we choose the window size 3 for our model MHT. In Table 3, we conduct an ablation study to investigate the effect of reasoning direction on MHT model. L2R denotes the MHT model with natural reasoning direction, which encodes context sentences from left to right by multi-hop layers, while Direction TED (En-De) TED (Zh-En) L2R 26.22 19.52 R2L 25.80 19.18 Table 3: The performance of the MHT model on TED (En-de) and TED (Zh-En) using different reasoning direction. R2L indicates the MHT model encoding context sentences with an opposite direction. We observe that integrating reasoning processes by multi-hop attention with both direction can improve the effect of Transformer due to the incorporation of extra context information. Besides, MHT model reasoning with natural sentence order outperforms the MHT model with an opposite reasoning direction. This is within our expectation since the L2R reasoning is consistent with the reading and reasoning direction of human being. As shown in Table 4, we conduct an ablation study to explore how MTH model benefits from contexts on source side and target side of MTH model. None indicates the MTH model without multihop attention module on any side of MHT model, but only the draft of the current sentence. Source, Target and Source & Target indicate the MHT models with multi-hop attention module to introducing context on only source side, only target side and both sides respectively. We find that integrating source-side context or target-side context into the model brings improvements over None that ignores context on both side. Besides, MHT with context on both sides achieves the best performance, indicating that the beneficial context information captured by multi-hop attention on the source side and the target side are divergent and complementary. In training, the context draft sentences can be the drafts from a pre-trained MT system or the context references, while only the generated drafts are accessible during inference. Table 5 shows the BLEU scores of the MHT models using generated drafts and context references during training. We can see that the MHT model using drafts as contexts outperforms the MHT model directly using target-side context references, possibly because using context references faces the problem of exposure bias and the drafts generated from pre-trained translation system can bridge the gap between training and testing data. We present the translated results from baselines and our model in Table 6 to explore how multihop reasoning mitigate the impact of common discourse phenomena in translation process. According to Case 1 in Table 6, the noun hum in source sentence is translated to der Summen by Transformer and CA-Transformer, which fail to understand the correct coreference. In German, der is a masculine article. The correct article is neutral article das because the hum is from a machine. MHT can perform a reasoning process to leverage the context information effectively and figure out the hum is from an engine according to Context 2. Case 2 indicates that MHT can understand the exact meaning of a polysemous word, benefiting from the reasoning process among the contexts. In this case, Transformer, CA-Transformer and CA-HAN all translates the noun show into zeigt, which means display. The translation is clearly wrong in this context. The correct meaning of show is TV shows like Breaking Bad according to the Context 1. In contrast, our model can take previous contexts in consideration and reason out the exact meaning of the polysemous word. In this paper, we propose a novel document-level translation model called Multi-Hop Transformer with an inspiration from human reasoning behavior to explicitly model the human-like draft-editing and reasoning process. Experimental results on four widely used tasks show that our model can achieve better performance than both context-agnostic and context-aware strong baseline. Furthermore, the qualitative analysis shows that the multi-hop reasoning mechanism is capable of solving some discourse phenomena by capturing context semantics more accurately. This work was supported by National Key R&D Program of China (2018YFB1403202). We thank the anonymous reviewers for their insightful comments.
abstain abstain abstain objective abstain objective abstain abstain objective abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method objective abstain objective objective objective objective abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain objective abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain result abstain other other