Multi-encoder models are a broad family of context-aware neural machine translation systems that aim to improve translation quality by encoding document-level contextual information alongside the current sentence. The context encoding is undertaken by contextual parameters , trained on document-level data. In this work, we discuss the difficulty of training these parameters effectively, due to the sparsity of the words in need of context (i.e., the training signal), and their relevant context. We propose to pre-train the contextual parameters over split sentence pairs, which makes an efficient use of the available data for two reasons. Firstly, it increases the contextual training signal by breaking intra-sentential syntactic relations, and thus pushing the model to search the context for disambiguating clues more frequently. Secondly, it eases the retrieval of relevant context, since context segments become shorter. We propose four different splitting methods, and evaluate our approach with BLEU and contrastive test sets. Results show that it consistently improves learning of contextual parameters, both in low and high resource settings. Neural machine translation (NMT) has seen substantial improvements in recent years, fostered by the advent of the Transformer model (Vaswani et al., 2017). A remaining challenge for modern machine translation (MT) is the ability to contextualize translation of the current sentence with other sentences in the document (Lubli et al., 2018). For this reason, contextual NMT has recently triggered a lot of attention and many approaches have been proposed in the literature. A common taxonomy (Kim et al., 2019; Li et al., 2020) divides them in two broad categories: single-encoder (con-catenation) approaches (Tiedemann and Scherrer, 2017; Agrawal et al., 2018; Ma et al., 2020; Zhang et al., 2020) and multi-encoder approaches (Jean et al., 2017; Tu et al., 2017; Bawden et al., 2018; Miculicich et al., 2018; Voita et al., 2018; Maruf et al., 2019; Zheng et al., 2020). Multi-encoder models are more flexible and can be more efficient than concatenation approaches, but they have been criticized as being mere regularization methods (Kim et al., 2019; Li et al., 2020). In some cases, they have even been shown to perform worse than sentence-level systems on discourse-aware targeted test suites (Lopes et al., 2020). In this work, we address this criticism by showing that training multi-encoder models is difficult because of two reasons: (i) the sparsity of contextual training signal , i.e. the signal that pushes systems to translate in a context-aware fashion, which comes from the words that need context to be correctly translated; (ii) the sparsity of relevant context words, the ones needed to disambiguate translation. A trivial way to improve context-aware learning is by increasing the amount of document-level training data. Large document-level parallel corpora are not always available, but some works have proposed data augmentation techniques to remedy scarcity (Sugiyama and Yoshinaga, 2019; Stojanovski et al., 2020; Huo et al., 2020). However, as we will show in our experimental section, this solution is not efficient and often sub-optimal. We therefore introduce a novel pre-training strategy, divide and rule (d&r) , that is based on a simple and yet powerful technique to augment the contextual training signal and to ease learning efficiently: splitting parallel sentences in segments (see Figure 1). Simply put, feeding a context-aware model with a sequence of incomplete, shorter, consecutive segments, forces it to look for context (i.e., surrounding segments) more frequently, and makes it easier to retrieve relevant context because segments are shorter. This results in faster and improved learning. We pre-train multi-encoder models on split datasets and evaluate them in two ways: BLEU score, and 4557 S i, 1 He said that it was a project of peace S i, 2 and unity and that it brought people together . T i, 1 Il disait que c' tait un projet de paix T i, 2 et d' unit et qu' il runissait les gens . S j, 1 I think single-cell organisms are S j, 2 possible within two years . T j, 1 Je pense que les organismes unicellulaires T j, 2 sont possibles dans 2 ans . Our main contributions are the following: (i) we show that context-aware multi-encoder models need to be trained carefully, because the contextual training signal is sparse, as well as the context elements useful for contextualization; (ii) we propose the d&r pre-training strategy, which fosters training of contextual parameters by splitting sentences into segments, with four splitting variants; (iii) we support this strategy with an analysis of the impact of splitting on the distribution of discourse phenomena; (iv) we demonstrate that this strategy is both effective and efficient, as it allows multi-encoder models to learn better and faster than by simply increasing the training data. The most straightforward approach to context-aware NMT consists in concatenating the context to the current sentence before feeding it to the standard encoder-decoder architecture (Tiede-mann and Scherrer, 2017; Agrawal et al., 2018; Junczys-Dowmunt, 2019; Ma et al., 2020; Zhang et al., 2020). A special token is introduced to mark the boundaries between sentences. Generation can then follow two strategies: the many-to-many strategy consists in translating all the source sentences, and then discarding contextual sentences; the many-to-one strategy consists in translating the current sentence only. The modeling capacity of concatenation methods is limited to few sentences because the complexity of attention scales quadratically with sentence length, although some recent works try to solve this constraint (Tay et al., 2020). Multi-encoder models couple a self-standing sentence-level NMT system, with parameters  S , with additional parameters for modeling the context either on source side, target side, or both. We refer to these parameters as the contextual parameters  C . The full context-aware architecture has parameters  = [  S ;  C ] . Most of the multi-encoder models can be described as instances of two architectural families (Kim et al., 2019), that only differ in the way the encoded representations of the context and the current sentence are integrated. Outside integration. In this approach, depicted in Figure 2, the encoded representations are merged outside the decoder (Maruf et al., 2018; Voita et al., 2018; Zhang et al., 2018; Miculicich et al., 2018; Maruf et al., 2019; Zheng et al., 2020). This can happen in different ways, such as by simple concatenation of the encodings, or with a gated sum. Inside integration. Here the decoder attends to the context representations directly, using its internal representation of the decoded history as query (Tu et al., 2018; Kuang et al., 2018; Bawden et al., 2018; Voita et al., 2019b; Tan et al., 2019). Many of these works found it useful to share parameters of current-sentence and context encoders (Voita et al., 2018; Li et al., 2020). In this way, the amount of contextual parameters to learn, |  C | , and the computational cost are drastically reduced. Shared representation can also be cached to be re-used and further processed by contextual parameters without the need of re-encoding sentences from scratch, which represents an advantage with respect to single-encoder approaches. Most of the approaches proposed in the literature focus on a few previous sentences, where most of the relevant context is concentrated. Two-step training. Multi-encoder models are commonly trained following a two-step strategy (Tu et al., 2018; Zhang et al., 2018; Miculicich et al., 2018; Maruf and Haffari, 2018; Li et al., 2020). The first step consists in training  S independently on a sentence-level parallel corpus CS . Secondarily, contextual parameters  C are trained on a document-level parallel corpus CD , while fine-tuning or freezing  S . Note that CS can also include sentences from CD . Novel MT systems are usually evaluated by computing BLEU (Papineni et al., 2002) on the test data. However, BLEU is ill-equipped to capture the improvements achieved by context-aware MT (Hard-meier, 2012), because contextualization can improve the translation of only a small fraction of the words in a document, while most of the words can be correctly translated without knowing the context. For instance, only a fraction of the anaphoric pronouns in a document has its nominal antecedent outside its own sentence. However, despite being sparse, these few cases strongly impact the quality of translation (Lubli et al., 2018; Popescu-Belis, 2019). Consequently, a number of discourse-targeted test sets and automatic metrics have been proposed to measure improvements in context-aware MT (Maruf et al., 2021), the most widely adopted ones being contrastive test sets. Contrastive test sets (Bawden et al., 2018; Mller et al., 2018; Voita et al., 2019a) consist of a number of source sentences, each paired with a correct translation and some incorrect ones. Models are assessed on their ability to rank the correct translation first. In many cases, this can be identified only by looking at context, which is provided for both source and target sides. Therefore, the ranking accuracy reflects the context-modeling ability of the evaluated translation system. Some works criticized multi-encoder methods (Kim et al., 2019; Li et al., 2020), arguing that they do not improve sentence-level baselines in terms of BLEU when the baseline is well regularized. When there are improvements, it is argued that the context-encoder simply works as a noise-generator that makes training more robust, and the improvements are not due to better context-modeling. Along this path, Lopes et al. (2020) showed that multi-encoder architectures struggle to model contextual information, and even deteriorate the performance of a sentence-level baseline on contrastive test sets. In fact, many proponents of multi-encoder models only show BLEU improvements, without providing any kind of targeted evaluation. This doesn't allow a direct evaluation of their context-modeling capability. We posit that training the contextual parameters of multi-encoder models is non-trivial because of two challenges: (i) the sparsity of the training signal, which comes from the words that need context to be correctly translated (most of the words of a sentence can be translated without context); (ii) the sparsity of context words that are useful for contextualization (most of the context is useless). As such, missing the right experimental setting can lead to unsuccessful training and unconvincing results. More data? A trivial way to offset sparsity is to increase the volume of training data. In fact, existing works that report strong results with targeted evaluation train their contextual parameters with millions of document-level sentence pairs (Baw-den et al., 2018; Mller et al., 2018; Voita et al., 2019b; Zheng et al., 2020; Wong et al., 2020; Kang et al., 2020). In contrast, many works in the literature train models with the TED talks' subtitles released by the IWSLT shared tasks (Cettolo et al., 2012), which only consist of a couple of hundred thousand parallel sentences. In the experimental section, we will show that IWSLT's subtitles are not sufficient to effectively train multi-encoder models. It follows that one can not make fair comparisons between alternative architectures in such experimental settings. On the other hand, we will give an empirical confirmation to the intuition that increasing the volume of training data helps learning contextual parameters. However, increasing the amount of training data is an inefficient solution, and one that is not always feasible: large document-level training sets may not be available in many languages. In the following section, we propose a pretraining solution that makes an efficient use of the available data for learning contextual-parameters effectively. One way to simulate document-level data is to split sentences in two or more segments (Luong et al., 2016). In this way intra-sentential syntactic relations are broken, and a word previously disam-4559 Algorithm 1: Split parallel corpus 1: input: Parallel corpus C , minimum source length l min , function wheresplit() 2: for i = 1 , . . . , |C| do 3: if len ( S i )  l min then 4: m S , m T = wheresplit( S i , T i , ... ) 5: S i, 1 = S i<m S and S i, 2 = S i  m S 6: T i, 1 = T i<m T and T i, 2 = T i  m T 7: end if 8: end for 9: return Split corpus CD biguated by looking at its neighbours in the sentence, now requires contextual information from the other segment in order to be correctly translated. Moreover, splitting sentences increases the concentration of relevant words within the context segment, as we will show in Section 4.2. Within the framework of MT, if we split the source sentence, its corresponding reference has to be split too. The proposed approach, divide and rule ( d&r ), consists in pre-training the model on a dataset CD that results from splitting all the sentences of a parallel corpus C that have at least l min tokens, as described by Algorithm 1. Each source-side sentence S i , with index i = 1 , ..., |C| , is split into S i, 1 and S i, 2 . Its corresponding reference T i is split into T i, 1 and T i, 2 . The resulting corpus is a document-level parallel corpus CD , such that, if the original corpus C was itself document-level, then CD keeps the same document boundaries as C . Figure 1 illustrates two examples of parallel sentences that are split in the middle. In both examples, a context-aware system needs to look at S i, 1 for translating S i, 2 correctly, i.e. to look at past context. In the first one, the English neuter pronoun it" could be translated into il" or elle", according to the gender of its antecedent (there is no singular neuter 3rd-person in French). The antecedent a project", which is in the previous segment, allows to disambiguate it into il". In the second example, the adjective possible can be correctly translated into its plural version possibles by looking back at the noun it refers to: organisms. In Algoritm 1, the wheresplit function returns the token indices m S and m T of S i and T i , where the sentence is split. In this work, we propose and experiment with four variants of this function. Middle-split. The simplest strategy is to split both the source and the target in the middle. In this case, wheresplit = middlesplit( S i , T i ) returns m S = (cid:98) len ( S i ) / 2 (cid:99) and m T = (cid:98) len ( T i ) / 2 (cid:99) . Following this method, it can happen that S i,j and T i,j , with j = 1 , 2 , are not parallel, as illustrated in the second example of Figure 1. The verb are belongs to S i, 1 , but its translation sont does not belong to its corresponding reference segment T i, 1 . In other words, sometimes the splitting can separate a set of words from their reference, which end up in the other segment. Clearly, this method requires the two languages not to excessively diverge in terms of word order, to avoid too large mismatches between S i,j and T i,j , with j = 1 , 2 . Aligned-split. As a solution to the misalignment problem between source and target segments, we can calculate word alignments A i , and use them to inform our splitting strategy by setting wheresplit = alignedsplit( S i , T i , A i ) , where alignedsplit splits each sentence close to the middle, while avoiding to separate aligned words in different segments. Synt-split . The objective of splitting being to break intra-sentential syntactic and semantic relations in order to force the model to exploit the context more frequently, we can run an NLP toolkit over the training set to retrieve relations L (e.g. syntactic dependencies or coreferences), and then by defining wheresplit = syntsplit( S i , T i , L i ) so that it splits sentences close to the middle, while ensuring that at least a relation is broken whenever possible. Since not all relations raise translation ambiguities when broken, one can choose which of them must be prioritized; in this work we chose pronominal coreferences. Multi-split. The aforementioned methods can be extended to splitting sentences in more than two segments. The more we split sentences the more likely it is that context is needed for each segment, hence increasing training signal for contextual parameters. In Section 6.3, we present an empirical comparison between the four splitting methods. We refer to Appendix A and to our implementation 1 for further details. To give an explicit picture of how and why sentence splitting helps learning contextual parameters, we processed the source training data of IWSLT17 with CoreNLP (Manning et al., 2014) and we computed some statistics on coreference chains and dependency parse trees, before and after applying the middle-split method. Statistics show how splitting the sentences of a document helps in two ways: More cases. Splitting generates new cases that require context for disambiguation, making training signal more abundant. When syntactic dependencies are split in two segments, the model needs to access the context for reconstructing the syntactic structure of the source sentence and correctly translate it, as shown in Figure 1. In order to have an idea of the magnitude of this effect, we calculated the percentage of the sentences where the splitting method breaks at least one syntactic dependency between the main verb of the sentence (the root) and : (i) the subject or object (18.1% of the sentences); (ii) any complement (9.5%); (iii) any modifier (9.3%). If we consider all the dependencies with the root, except punctuations, we find that in 84.8% of the sentences at least a syntactic dependency is broken. Given such high proportion, the middle-split variant is in fact a good approximation of a syntactically supported splitting approach. These cases add up to the many other cases of broken relations, such as coreferences, which make the overall contextual training signal more abundant. Denser cases. The splitting also has the effect of shortening the average length of text sequences, which eases the job of context-aware systems because they have to attend to fewer words while looking for context. In Figure 3, we show how many antecedents of an anaphoric pronoun are present in the data at a given distance d , expressed as number of sentences from the current one for original data, and number of segments for split data. d = 0 means that both the pronoun and its antecedent are in the same sentence (or segment); d = 1 means that the antecedent is in previous sentence (or seg-ment), and so on. We show statistics up to d = 3 , which is the maximum context distance that we experiment with. The absolute number of antecedents is divided by the average length of a sentence or segment. The resulting bar plot shows that splitting sentences into segments makes pronominal antecedents more dense in the set of context tokens that the model is attending, which fosters the learning of contextual parameters. The same effect applies to the other discourse phenomena that require contextual disambiguation. 2 5 Experimental setup 5.1 Data We conduct experiments for three language pairs: English  Russian/German/French, on different domains. Following Kim et al. (2019), we pre-train sentence-level baselines on large sentence-level parallel data to make them as robust as possible. In particular, we employ data released by Voita et al. (2019b) for En  Ru (6.0M sentences from OpenSubtitles2018 (Lison et al., 2018)), data from the WMT17 3 news translation shared task for En  De (  5.2M sentences), and data from WMT14 4 for En  Fr (  35.8M sentences). We train the contextual parameters of context-aware models in two settings, while freezing the rest of the parameters: High resource data. For En  Ru, it consists of the document-level data released by Voita et al. (2019b). For the other two language pairs, we build the training set by assembling (i) News-Commentary-v12 for En  De and News-Commentary-v9 for En  Fr; (ii) Europarl-v7 for En  De/Fr; (iii) TED talks subtitles released by IWSLT17 (Cettolo et al., 2012) for En  De/Fr. Low resource data. For En  Ru, it consists of a random subset of the high resource documents, amounting to 1/10th of its total. For En  De/Fr, we use IWSLT17's TED talks alone. The resulting size of the two training settings after pre-processing is reported in Table 1. In the 2 More details are available in Appendix B, along with the same statistics for Opensubtitles2018. case of En  De/Fr, baselines and context-aware models that were trained on high resources are also fine-tuned on IWSLT17, so that both high and low resource settings can be bench-marked on the IWSLT17's test set 2015. Test-sets 2011-2014 are used as development set. For En  Ru, we use the dev and test sets provided by Voita et al. (2019b). 5 5.2 Evaluation Besides evaluating average translation quality with BLEU (Papineni et al., 2002), 6 we employ three contrastive test suites for the evaluation of the translation of discourse phenomena. 7 En  Ru EllipsisVP (Voita et al., 2019b). Consisting of 500 examples from OpenSubtitles2018, each containing multiple contrastive hypotheses to evaluate the translation of verb phrase ellipses. Source sentences contain an auxiliary verb (e.g. "do") and an omitted main verb, which can be imputed thanks to one of the three context sentences. Voita et al. (2019b) proposed test sets for the evaluation of other discourse phenomena, but we do not use them because they are conceived for systems using target-side context too. En  De ContraPro (Mller et al., 2018). A large-scale test set from OpenSubtitles2018 (Li-son et al., 2018), that measures translation accuracy of the English anaphoric pronoun it into the corresponding German translations er , sie or es . Examples are balanced across the three pronoun classes (4,000 examples each). Each example requires identification of the pronominal antecedent, either in the source or target side, that can be found in the current sentence or any of the previous ones. En  Fr ContraPro (Lopes et al., 2020). A large-scale test set from OpenSubtitles2018, completely analogous to the previous one but focused on the translation of two English pronouns: it and 5 We report in Appendix C a re-cap of the datasets used and details about pre-processing. Transformer-base by Vaswani et al. (2017). K1 . A context aware multi-encoder architecture with outside integration (see Section 2.2), that encodes a single past source sentence as context. K3 . A context aware multi-encoder architecture with outside integration , that encodes three past source sentences as context. 8 For both K1 and K3 , sentence-level parameters  S follow the Transformer-base configuration (hid-den size of 512, feed forward size of 2048, 6 layers, 8 attention heads, total of 60.7M parameters), while contextual parameters  C follow hierarchical architecture with source-side encoder proposed by Miculicich et al. (2018) (hidden size of 512, feed forward size of 2048, 8 attention heads, total of 4.7M parameters). 9 Context-aware models are trained following the two-step strategy described in Section 2.2. Sentence-level parameters  S of both K1 and K3 are initialized with K0 and freezed. This has the advantage of saving time and computation, since only a small fraction of parameters (  C ) is trained (4.7M over a total of 65.2M). In this section we provide evidence about the difficulty of training contextual parameters on document-level data. In the first block of lines of Table 2, after the results of the sentence-level baseline K0 , we report performance of context-aware models trained on original document-level data, comparing low and high resource settings. When trained on low resources , models display good BLEU on the test set, generally without relevant degradation with respect to K0 , or even with some improvements. However, such marginal fluctuations in BLEU are difficult to interpret, as they do not necessarily correspond to better or worse translation (Freitag et al., 2020). Accuracy on the contrastive test sets also increases marginally over baseline, if at all, for En  De/Fr. 8 Although the splitting does not increase the number of inter-segment phenomena for d > 1 , it strengthens the signal by making it more dense (see Section 4.2). Thus, K3 and any wider-context model can profit from the proposed approach. 9 Details can be found in Appendix C 4562 En  De En  Fr En  Ru Avg. K1 even shows a slight degradation of performance over the sentence-level baseline for En  Fr. These results highlight the struggle of contextual parameters to learn an appropriate use of context, other than acting as mere regularizers, as it was suggested by Kim et al. (2019) and Li et al. (2020). On Russian instead, models display some improvements w.r.t. K0 . This aligns with our expectations, since En  Ru Low Res has a volume of inter-sentential discourse phenomena (such as coreferences) that is comparable with En  De/Fr Low Res, but sentences are 2.5x shorter. 10 In other words, the double challenge of sparsity is mitigated on this corpus. When trained on high resources , systems show substantial improvements in their context-modeling capabilities, on all language pairs. Instead, BLEU improves of a few decimal points only, showing its limits to measure improvements in context-aware translation. These results confirm the intuition discussed in Section 3: increasing the volume of data is a trivial solution to mitigate sparsity. In this section, we show that the proposed pretraining strategy is a more efficient answer to the double challenge of sparsity than simply adding more data, and one that allows improvements when resources are abundant too. The second block of Table 2 displays performance of models that have undergone d&r pre-training on the same document-level data as the models in the previous block, but where sentences were split in two segments following the middle-split method with l min = 7 (see 4.1). During d&r pre-training, K1 and K3 encode one and three past segments (instead of 10 See Table 1; more details can be found in Appendix B sentences), respectively. After d&r pre-training, models have been tuned and evaluated on original, non-split data. The pre-training proves to be very effective, as all models show strong improvements in terms of accuracy on the test suites, with the sole exception of K1-d&r on En  Fr High Res. The average improvement is of +10.79 accuracy points on Low Res, +8.49 on High Res, showing that d&r brings strong improvements even when data are abundant. Interestingly, improvements are not uniformly distributed across language pairs and domains: +17.20 on average for En  Ru, +8.67 for En  De, +3.09 for En  Fr. In terms of BLEU instead, we keep seeing minor fluctuations. This confirms that, while context-aware translation improves dramatically, the average translation quality measured with BLEU stays more or less constant. 11 It is now evident that a proper assessment of multi-encoder approaches can not be undergone without careful training of contextual parameters that targets the problem of sparsity. Efficiency . A comparison between d&r models trained on Low Res against models trained on High Res without d&r shows another quality of the d&r pre-training strategy: efficiency. The same context-aware models achieve superior performances with 1/10th of the document-level data and a much shorter training time (last column). Following Section 4.1, we study the impact of using a different splitting method other than middlesplit . All the variants are applied to the En  De/Fr 11 To verify that the improvements on test suites after d&r pre-training really come from a better use of context, we present in Appendix D an analysis of pronoun translation by antecedent distance, and an ablation study in which we test models on ContraPro with inconsistent context. low resource setting (IWSLT), with l min = 7 , and the d&r pre-trained models are evaluated on ContraPro. The aligned-split method is based on alignments learned with fast_align (Dyer et al., 2013), while for the synt-split method we retrieve intra-sentential pronominal coreferences with CoreNLP (Manning et al., 2014), and we try to split them wherever present in a sentence-pair. We split sentences as close to the middle as possible, while attempting to break the maximum number of coreferences. 12 Finally, for the multi-split method, we split sentence-pairs in a half for len ( S i )  7 , and also in three segments of identical size for len ( S i )  15 . The performance differences between models pre-trained with middle-split and the other variants are reported in Table 3. As we can see, splitting variants allow small improvements in 7 cases out of 10, although variations are marginal: the simple middle-split method seems to be already close to optimal. This observation can be explained by multiple elements. Firstly, middle-split produces segment pairs that are already well aligned: most of the source and target segments are aligned with the exception of one or two words, and the fact of having only a few misplaced words might act as a regularization factor. Secondly, middle-split breaks a syntactic relation for the vast majority of sentences already, as explained in Section 4.2, which means that improvements achieved with syntactically driven splitting can only be marginal. Thirdly, splitting in more than one segment can be beneficial in some cases, because it allows to break more syntactic relations and increase density of signal, but it also increases the risk of misalignment between source and tar-12 More sophisticated synt-split methods could be devised, targeting other discourse phenomena, or several of them at the same time, with different degrees of priority. get, and might make the task too hard. Finally, tools like fast_align and CoreNLP are characterized by a non-negligible language-dependent error rate, which affects the performance of the methods. In conclusion, d&r pre-training with middle-split seems to be the most convenient alternative for most use-cases because of its efficacy, its simplicity and its language-independence. Even though middle-split relies on word order similarity between source and target languages, we argue that the required degree of similarity is met by a large number of language pairs, in the order of millions. In fact, there are around 4,000 written languages in the world (Eberhard et al., 2021), and most of them can be grouped in a few types with similar word orders, as shown by the ample literature on word order typologies (Tomlin, 2014; Dryer and Haspelmath, 2013). The primary order of interest is the constituent order , concerning the relative order of subject (S), object (O) and verb (V) in a clause. There are seven possible language types with respect to the constituent order (Dryer, 2013c): SOV, SVO, VSO, VOS, OVS, OSV, NDO (non-dominant or-der). Tomlin (2014) estimates that more than 40% of the world languages belong to the SOV type (languages adopting the SOV order), another 40% belong to the SVO type, while almost 10% of languages adopt VSO order. The other types are rarer. As we have shown above, the middle-split method is beneficial both in the case of language pairs of the same type, that deploy the same constituent order, like En-Fr/Ru, which all adopt SVO order, as well as for language pairs that belong to different types, as for En-De, where English is SVO and German is NDO, deploying both SOV and SVO according to the use cases (Dryer, 2013c). Similar observations also apply when we look at other word order categories. For instance, when looking at the order of modifiers or adverbials, languages can be clustered in a few types too, where the wide majority of languages belong to the biggest or second biggest type (Dryer, 2013b,a). Therefore, we believe that our method can be beneficial for millions of language pairs, including many low resource languages belonging not only to same word order types, but also slightly different ones (as in the case of SOV and SVO). For a wider contextualization of our results, we report in the first block of Table 4 some experimental results by Lopes et al. (2020), who trained and evaluated various context-aware approaches on the same low-resource setting (IWSLT17), adopting the very same experimental setup as ours. Specifi-cally, they trained and evaluated: K0 : a baseline like ours; Zhang2018 : a multi-encoder model that encodes three past source sentences, with both inside integration (see 2.2) and outside integration (Zhang et al., 2018); Tu2018 : a multi-encoder model that encodes all the past context (at any distance) with a caching system with inside integration in the decoder, both on the source and the target side (Tu et al., 2018); Concat21 : a many-to-one (2-to-1) single-encoder approach (see 2.1) that exploits contextual information from one past source sentence; Concat22 : a many-to-many (2-to-2) single-encoder approach that exploits contextual information from one past sentence, both on the source and the target side; Multi-encoder models ( Zhang2018 and Tu2018 ) perform poorly or even lag behind K0 , confirming the difficulty of multi-encoder models to learn contextualization on low resources and without any help against the problem of sparsity. Instead, concatenation approaches are stronger, likely because they do not have extra contextual parameters to train and simply finetune the same sentence-level Transformer architecture on the context-aware task. This makes them less affected by the problem of sparsity. Our splitting strategy proves to be very effective, since both d&r pre-trained models outperform Concat21 using the same amount of training data. Moreover, K1/3 d&r beat the strong Concat22 on En  Fr, which has the non-negligible advantage of using also the target-side context along the source-side. This benchmarking show that multi-encoder models are a viable solution for context-aware NMT, but that they need to be carefully (pre)-trained to harness their capabilities. We leave to future works a more detailed comparison between single-encoder and multi-encoder approaches, as well as between d&r and other recently proposed pre-training strategies for context-aware models (e.g., Fernandes et al. (2021)). Multi-encoder models are a broad family of context-aware NMT models. In this work we have discussed the difficulty of training contextual parameters due to the sparsity of the words in need of context, and the sparsity of their relevant context. We have proposed a pre-training approach called divide and rule , based on splitting the training sentences, with four variants. After having analysed the implications of splitting on the distribution of discourse phenomena within the training data, we have shown that d&r allows to learn contextual parameters better and faster than by simply increasing training data. We have also shown that the simplest and language independent splitting variant, middlesplit , is a strong baseline that can be easily applied for pre-training any multi-encoder NMT model. We thank the anonymous reviewers for their insightful comments. This work has been partially supported by the Multidisciplinary Institute in Artificial Intelligence MIAI@Grenoble Alpes (ANR-19-P3IA-0003), and it was granted access to the HPC resources of IDRIS under the allocation 2020-101501 made by GENCI.
abstain abstain method objective abstain abstain objective abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain result objective abstain abstain method objective objective objective abstain abstain abstain abstain abstain other other other other other method other other other other other other other other other other other other other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method method abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method method method method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain method method method method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method result abstain method method method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain method objective result result other other