Document-level neural machine translation (DocNMT) achieves coherent translations by incorporating cross-sentence context. However, for most language pairs there's a shortage of parallel documents, although parallel sentences are readily available. In this paper, we study whether and how contextual modeling in DocNMT is transferable via multilingual modeling. We focus on the scenario of zero-shot transfer from teacher languages with document level data to student languages with no documents but sentence level data, and for the first time treat document-level translation as a transfer learning problem. Using simple concatenation-based DocNMT, we explore the effect of 3 factors on the transfer: the number of teacher languages with document level data, the balance between document and sentence level data at training, and the data condition of parallel documents (genuine vs. back-translated). Our experiments on Europarl-7 and IWSLT-10 show the feasibility of multilingual transfer for DocNMT, particularly on document-specific metrics. We observe that more teacher languages and adequate data balance both contribute to better transfer quality. Surprisingly, the transfer is less sensitive to the data condition, where multilingual DocNMT delivers decent performance with either back-translated or genuine document pairs. Recent years have witnessed a trend moving from sentence-level neural machine translation (Sen-NMT) to its document-level counterpart (Doc-NMT). SenNMT inevitably suffers from translation errors related with document phenomena (Maruf et al., 2021) and delivers obviously inferior performance when compared against human translations and evaluated at a document level (Lubli et al.,  Work done while Biao Zhang was interning at Google Research. 2018). Most efforts on DocNMT aim at improving contextual modeling via dedicated model architectures and/or decoding algorithms (Bawden et al., 2018; Voita et al., 2019; Chen et al., 2020) and heavily rely on large-scale parallel document resources. Nevertheless, document resources are unevenly distributed across language pairs, with most pairs having little to no such resources. 1 One promising way to accommodate languages with varied training data is multilingual modeling, as demonstrated in multilingual SenNMT (Firat et al., 2016; Johnson et al., 2017). By sharing parameters across languages, multilingual modeling encourages cross-lingual knowledge transfer, enabling performance improvement and even zero-shot transfer (Aharoni et al., 2019; Arivazhagan et al., 2019b; Zhang et al., 2020, 2021). In the context of translation, however, most studies on multilingual transfer center around SenNMT, seldom going beyond sentence-level translation. So far, the question of whether and how document-level contextual modeling can be learned cross-lingually in multilingual DocNMT is still unanswered. In this paper, we study zero-shot generalization for DocNMT  the ability to attain plausible Doc-1 Note that we use language and language pair interchangeably since one side of our parallel data is always English. NMT quality for some focused ( student ) language pair(s), with only parallel sentences for the student but parallel documents for other ( teacher ) languages in the multilingual mix. The high-level research question we seek to answer is illustrated in Figure 1. We resort to transfer learning via multilinguality to leverage document resources in teacher languages to help the student languages. We perform our analysis using a simple concatenation based DocNMT, where consecutive sentences are chained into one sequence for translation. We investigate three dimensions extensively to understand the transfer in multilingual DocNMT: 1) the number of languages with document level data (teacher languages), where we simplify our transfer setup to contain either only one teacher language (with N students) or N teachers (with one student); 2) the data balance for parallel documents, i.e. manipulating the ratio of document-level data to sentence-level data during training; and 3) the data condition of parallel documents, where we adopt back-translated parallel documents when only monolingual documents are given in teacher languages or use genuine parallel documents crawled natively. We conduct experiments on two publicly available datasets, namely Europarl-7 and IWSLT-10, covering 6 and 9 languages from/to English respectively. We analyze one-to-many (En  Xx) and many-to-one (Xx  En) translation scenarios separately. Following recent work (Ma et al., 2021), we adopt document-specific metrics for evaluation apart from BLEU and support our findings with human evaluations. We also propose a pronoun F1 metric (targeted at gendered pronouns: he/she) for Xx  En translation, and employ accuracy on contrastive test sets (Bawden et al., 2018; Mller et al., 2018) for En  Xx translation. Our main findings are summarized below:  Zero-shot transfer from sentences to documents is feasible through multilingual DocNMT modeling, particularly when evaluated with document-specific metrics. This is partially supported by human evaluation. Transfer quality is strongly affected by the number of teacher languages that use document level data and the data balance for documents. Higher quality is achieved with more teacher languages and adequate document schedule, where the optimal balance varies across scenarios. Surprisingly, transfer via back-translated documents performs comparable to transfer via genuine parallel documents. Zero-shot transfer from high-resource document level languages and to low-resource sentence level ones is relatively easier, resulting in better transfer results. Document-level MT Integrating document-level information meaningfully into NMT is a challenging task, which has inspired research not only on exploring advanced context-aware neural architectures, including simple concatenation-based models (Tiedemann and Scherrer, 2017; Junczys-Dowmunt, 2019; Lopes et al., 2020), multi-source models (Jean et al., 2017; Bawden et al., 2018; Zhang et al., 2018), hierarchical models (Miculi-cich et al., 2018; Zheng et al., 2020; Chen et al., 2020), multi-pass models (Voita et al., 2019; Yu et al., 2020; Mansimov et al., 2021) and dynamic context models (Kang et al., 2020), to name a few. But it has also motivated the field to revisit the common protocols resorted for evaluation (Freitag et al., 2021). Despite the hard to measure success, all the above mentioned methods implicitly assume an abundance of document resources and overlook the data scarcity problem. In this study, we adopt the simple concatenation model as our experimental protocol, and leave the exploration of various input formatting options and modelling to future work. Considering the fast changing landscape of the (contextual) MT evaluation, we also provide multiple evaluation metrics including human evaluations, to give a full picture of the phenomena under investigation, while acknowledging the current imperfections of and disagreements on the right way of evaluating MT systems (Kocmi et al., 2021). Zero-Shot Transfer via Multilinguality Multilingual modeling often clusters sentences of similar meaning from different languages within a shared semantic space (Kudugunta et al., 2019; Siddhant et al., 2020). Such representation space is hypothesized to enable zero-shot transfer, delivering improved performance in many cross-lingual tasks (Eriguchi et al., 2018; Hu et al., 2020; Chi et al., 2021; Ruder et al., 2021), especially based on large-scale pretrained multilingual Transformers (Devlin et al., 2019; Conneau and Lample, 2019; Xue et al., 2021). When it comes to transla-4177 tion, multilingual SenNMT successfully achieves zero-shot translation, transferring sentence-level generation knowledge to language pairs unseen during training (Firat et al., 2016; Johnson et al., 2017; Gu et al., 2019; Arivazhagan et al., 2019a) even in massively multilingual settings (Aharoni et al., 2019; Arivazhagan et al., 2019b; Zhang et al., 2020, 2021). Our study extends multilingual SenNMT to multilingual DocNMT and aims at document-level knowledge transfer from languages that have document level data to languages that only have sentence level data. To the best of our knowledge, our study is the first demonstrating the emergence of document-level zero-shot transfer across languages for multilingual machine translation. We first formulate the zero-shot generalization framework explored in this paper. Given N+1 language pairs, we assume that all of them have parallel sentences for training, but only some of them have parallel documents (teachers). Through multilingual training, we study to what degree contextual modeling in document-supervised DocNMT can be transferred to those document-poor (student) languages as in Figure 1. Any form of parallel document for student languages is disallowed at training, ensuring that the transfer is measured zero-shot. We employ the concatenation-based method with a D 2 D structure for DocNMT, where D consecutive sentences in a document are concatenated into one sequence for translation (Junczys-Dowmunt, 2019; Sun et al., 2020). Sentence boundary is indicated by a special symbol [SEN]. We adopt the language token method (Johnson et al., 2017) for multilingual DocNMT, using source and target language token for Xx  En and En  Xx translation respectively. Instead of appending this token to the source sequence, we add its embedding to each source word embedding to strengthen the language signal in a document translation setting. For training , we adopt a two-stage method: we first pretrain a multilingual SenNMT on sentence level data for all languages; then, we finetune it to obtain multilingual DocNMT on a mix of document level data from teacher languages and sentence level data from student languages. Our analysis requires training a large number of DocNMT models, and the two-stage method saves substantial amounts of computation by sharing the pretrained SenNMT. For evaluation , we distinguish sentence-level inference (SenInfer) from its document-level counterpart (DocInfer). SenInfer translates sentences separately (out of context), while DocInfer translates D consecutive and non-overlapping sentences in context with each other. 2 3.2 Zero-Shot Setup We explore three factors for the zero-shot transfer:  The number of teacher languages The source of the transfer comes from teacher languages. Intuitively, both the number of teacher languages and their relevance to student language(s) affect the transfer result. However, exhaustively exploring all possible teacher-student configurations in a multilingual setting will lead to a large search space that expands exponentially with respect to the total number of languages involved. Instead, we simplify our study by exploring two extreme transfer settings, namely N21 and 12N transfer. The first setting uses N teachers that incorporate document level data with 1 student having sentence level data only, while the second setting has 1 teacher and N students. Note that in either N21 or 12N transfer, there exist N teacher-student configurations, and we report average results over them. 3  The data balance for parallel documents When varying the number of teacher languages, the proportion of document data at training also changes. Such imbalance could deeply affect transfer (Arivazhagan et al., 2019b). To offset this effect, we include the data balance for analysis by controlling the sampling ratio p of documents from 0.1 to 0.9 with a step size of 0.1. Note p is for documents in all teacher languages , and the relative proportion among teachers is always retained. The data condition of parallel documents We also study when teacher languages have no parallel documents but only monolingual ones. Methods utilizing monolingual documents for DocNMT vary greatly. Following recent work (Sugiyama and Yoshinaga, 2 At decoding phase, the last chunk in a source document can have < D sentences for DocInfer. 2019; Huo et al., 2020; Ul Haq et al., 2020), we adopt back-translation (BT) to construct pseudo parallel documents. Note that, for teacher languages, we replace all sentence level training data with pseudo documents rather than mixing them according to our empirical results in Appendix C. 4 Experimental Settings Datasets We conduct experiments on two public datasets: Europarl-7 and IWSLT-10. Europarl-7 is extracted from European Parliament (v10) and has translations between English and N = 6 different languages, including Czech, German, Finnish, French, Lithuanian and Polish (Koehn, 2005). This dataset offers sentence-aligned parallel documents (0.9K  3.7K documents, 190K  1.9M sentences) and also monolingual documents (9.7K  11K documents, 0.65M  2.28M sentences) for training. For evaluation, we use the WMT dev and test sets (Bar-rault et al., 2020) available for each language pair (from 2013 to 2020). In contrast, IWSLT-10 is collected from TED talks and covers translations between English and N = 9 different languages, including Arabic, German, French, Italian, Japanese, Korean, Dutch, Romanian and Chinese (Cettolo et al., 2017). Unlike Europarl-7, the distribution of training data over languages in IWSLT-10 is much smoother (uniform). There are  1.9K sentence-aligned parallel documents with  240K sentences for each language pair. We further collected about 1K TED talks for each language pair (crawled from Feb 2018 to Jan 2021) as monolingual documents. We use IWSLT17 dev and test sets for evaluation. Detailed statistics are given in Appendix A. We preprocess all texts with the byte pair encoding (BPE) algorithm (Sennrich et al., 2016) implemented in the sentencepiece toolkit (Kudo and Richardson, 2018), and set the vocabulary size to 32K and 64K for IWSLT-10 and Europarl-7, respectively. Model Details We use the Transformer-base model (Vaswani et al., 2017) for experiments with 6 encoder/decoder layers, 8 attention heads and a model dimension of 512/2048. We set D = 5 for DocNMT. We use Adam (Kingma and Ba, 2015) (  1 = 0 . 9 ,  2 = 0 . 98 ) for parameter update with a learning rate warmup step of 4K and label smoothing rate of 0.1. We apply dropout to residual connections and attention weights with a rate of 0.5 and 0.2, respectively. Other training and decoding details are given in Appendix B. Back-Translation Some of our models are trained using back-translated monolingual documents. Back-translations are obtained using bilingual SenNMT (independently for Europarl-7 and IWSLT-10). To train these models, we halve the BPE vocabulary size as well as the training steps. All other settings are kept as mentioned above. Evaluation Following previous work, we use BLEU (Post, 2018) 4 to measure the general translation quality. Document-level BLEU is calculated by counting n-gram at the document level instead of at the individual sentence level (Sun et al., 2020). Measuring improvements to document phenomena in translation automatically remains challenging and oftentimes simple surface-based metrics such as BLEU (Lubli et al., 2018) are not sensitive enough. Therefore, we evaluate our model on test sets that focus on such document phenomena. We use the contrastive test sets for En-De (Mller et al., 2018) and En-Fr (Bawden et al., 2018) which measure a model's ability to distinguish correct from incorrect anaphoric pronoun translations. We include 4 and 1 additional context sentences for EnDe and En-Fr contrastive evaluation, respectively. Gender bias in translation models has attracted much attention recently (Kuczmarski and Johnson, 2018; Saunders and Byrne, 2020). We expect that contextual information can help to alleviate it. To this end, we introduce gendered pronoun F1 based on the following precision and recall scores to evaluate English translations: Precision = (cid:80) i,g G min( C g r i , C g h i ) (cid:80) i,g G C g h i Recall = (cid:80) i,g G min( C g r i , C g h i ) (cid:80) i,g G C g r i , (1) where r i and h i denotes the i -th gold reference and hypothesis sentence respectively, comprising the gendered pronouns of interest G 5 . C g x denotes the count of pronoun g in sentence x . Finally, we conduct human evaluation to verify the performance delivered by zero-shot transfer. We work on En-De, Europarl-7, where we sample 50 source documents from the test set, and translate them into the target language using the corresponding models and decoding techniques. The translated documents are presented to bilingual human raters who are native in the non-English locale. The raters are asked to evaluate translation qualities while taking the full source document context into account. The raters assign a score in a 0-6 scale to every sentence-translation pair in the document, where 0 and 6 mean nonsense and perfect translations, respectively. For each model, the scores are aggregated across the entire test corpus and the average scores are reported. To ensure a fair diversity of ratings, each rater rates no more than 6 documents per model; an average of 18 raters evaluated each model independently. Does SenNMT have the capability of leveraging context? Not really! We put our major analysis on Europarl-7 (N=6, all European languages). Before diving deep into the transfer, we start with analyzing whether SenNMT models trained on sentences alone could generalize to contextual translation. If multilingual SenNMT can be directly used for DocInfer, studying zero-shot transfer would be Model Xx  En En  Xx SenNMT w/ SenInfer 22.40 18.82 SenNMT w/ DocInfer ( D = 2 ) -2.98 -4.05 SenNMT w/ DocInfer ( D = 5 ) -11.7 -13.0 Table 1: Average BLEU on Europarl-7 for multilingual SenNMT with SenInfer and DocInfer. meaningless. Results in Table 1 challenge this possibility: SenNMT results in large quality reduction with DocInfer. We observe that SenNMT produces significantly shorter translations under DocInfer, preferring to translate the first few input sentences. We ascribe such failures to the poor generalization to documents from sentence-level training. Impact of the data balance and the number of teacher languages on zero-shot transfer Figure 2 and 3 summarize the results for En  Xx and Xx  En translation, respectively, where we report the average performance paired with the stan-4180 0.1 0.3 0.5 0.7 0.9 Sen proportion p Doc 6 4 2 0 BLEU v s . dard deviation over N configurations. 6 Overall, the document-level zero-shot transfer is achievable via multilingual modeling. Transfer-based DocNMT could successfully identify and translate the correct number of input sentences for student languages. With a proper sampling ratio for document-level data, student DocNMT yields better performance than its SenNMT counterpart, especially shown by document-specific evaluations (F1 and ACC). Increasing teacher languages improves transfer. In En  Xx and Xx  En translation, we find that N21 transfer performs consistently better than 12N transfer on all metrics. This is reasonable since N21 transfer has N teacher languages, offering richer and more informative sources for transfer. Balancing between document and sentence data matters for transfer. We also observe that performance changes over the document proportion on 6 Note the average results are for transfer directions, not the supervised ones. Each experiment in N21 transfer has only one transfer direction, so we directly report the average over N configurations; by contrast, in 12N transfer, we have N transfer directions, where we first perform average over these N transfer results followed by another average over N configurations. Also note, the average results contains transfer from high/low and similar/distant languages. all metrics in both 12N and N21 transfer. Applying more or fewer documents during training often hurts zero-shot transfer, indicating a trade-off. Roughly, setting p to 30%  50% delivers good performance (Figure 2 and 3), although the optimal proportion depends. SenInfer underperforms DocInfer on document-specific metrics. DocNMT w/ SenInfer performs similarly to SenNMT, and better than DocInfer on BLEU. When evaluating document phenomena, however, SenInfer shows clear insufficiency. This resonates with the findings of Ma et al. (2021). monolingual documents? Yes. We next repeat our experiments with BT document pairs. Figure 4 and 5 show that BT performs surprisingly well on document-level zero-shot transfer. We observe almost the same performance pattern compared to training with genuine documents in all settings (En  Xx and Xx  En, N21 and 12N transfer and different metrics), although BLEU scores become worse and the optimal proportion also changes. We argue that the target-side genuine context information in BT documents helps contextual model-4181 Xx  En BLEU F1 High  Low   High  Low High  Low   High  Low DocNMT + 12N transfer -1.07 -1.79 -1.71 -1.15 +1.73 +0.95 +0.73 +1.95 w/ BT -1.19 -1.37 -1.59 -0.97 +1.61 +2.38 +1.36 +2.64 En  Xx BLEU ACC En-De ACC En-Fr High  Low   High  Low High  Low  High  Low  DocNMT + 12N transfer -1.85 -2.05 -2.03 -1.87 +8.67 +6.27 +10.25 +7.50 w/ BT -3.29 -4.03 -4.39 -2.93 +8.55 +6.57 +6.61 +6.19 Table 2: Relative performance to multilingual SenNMT baseline when transferring from and into high-resource (High) and low-resource (Low) languages for En  Xx and Xx  En translation on Europarl-7. ing (Ma et al., 2021). These results are promising, encouraging further research on exploring monolingual documents for multilingual DocNMT. Impact of high/low-resource languages on zero-shot transfer. The data distribution of Europarl-7 is highly skewed over languages, with Cs, Lt, Pl being relatively low-resource languages while De, Fi, Fr being high-resource ones. Studies on multilingual SenNMT have witnessed the transfer from high-resource to low-resource languages (Aharoni et al., 2019; Zhang et al., 2020). We next analyze how this data scale difference affects document-level zero-shot transfer. We mainly explore 12N transfer because of the single transfer source, avoiding interference from other teacher languages. transferring from high-resource teacher languages often outperforms that from low-resource ones. Besides, transferring into low-resource student languages delivers better transfer than into high-resource ones. These suggest that increasing the document data for teacher languages benefits zero-shot transfer. Note we also provide transfer results from individual languages to De and Fr in Appendix D. Performance on Europarl-7 and IWSLT-10 We summarize the main results on both datasets in Table 3. Although IWSLT-10 (N=9) includes more (distant) languages and distributes quite differently over languages, the results on IWSLT-10 resemble those on Europarl-7. On both datasets, we observe that transfer, both 12N and N21, yields very positive results, particularly with document-specific metrics. Unlike Europarl-7, BT-based transfer per-4182 Models Human Rating (  ) Reference 4.96 SenNMT (Baseline) 3.31 DocNMT  w/ SenInfer 3.60 DocNMT  w/ DocInfer 3.84 N21 Transfer w/ DocInfer 3.46 12N Transfer w/ DocInfer 2.78 N21 Transfer + BT w/ DocInfer 3.18 12N Transfer + BT w/ DocInfer 2.72 Table 4: Document-level human ratings (  ) for En-De on Europarl-7. forms much worse than models trained on genuine document pairs on IWSLT-10. We ascribe this to the data scarcity, where only very small-scale monolingual documents are used for BT in IWSLT-10. This also reinforces our observation that more document resources benefits zero-shot transfer. Apart from automatic evaluation, we also offer human evaluation on En-De. We choose En-De as its WMT20 test set is intentionally constructed for DocNMT evaluation. Table 4 lists the results. We observe that zero-shot transfer matches and even surpasses SenNMT through N21 transfer, but fails with 12N transfer, although accuracy improvements on contrastive test sets show that both transfers are better than SenNMT. We conjecture that these contrastive test sets only target a limited number of document phenomena and thus can't fully reflect the overall translation quality and represent human preference. These numbers verify the feasibility of document-level zero-shot transfer through multilinguality. Besides, we find that genuine parallel documents benefit the transfer slightly more than BT-based pseudo ones, and that the supervised DocNMT reaches the best result under DocInfer. We surprisingly find that DocNMT with SenInfer yields very competitive performance, although no contextual information is used for decoding. We also observe that such decoding tends to produce longer translations than SenNMT despite using the same decoding hyperparameters. This behaviour should be shaped by the fact that DocNMT is biased towards long concatenated target references. This partially agrees with the recent argument that context improves DocNMT with some sort of regularization rather than teaching the model to deal Models ACC En-Fr ACC En-De SenNMT w/ SenInfer 50.00 52.00 SenNMT w/ DocInfer 58.50 (cid:63) 50.80 DocNMT w/ SenInfer 50.00 51.90 DocNMT w/ DocInfer 64.50  66.80  Table 5: Applying DocInfer and SenInfer to DocNMT and SenNMT for contrastive evaluation. with context (Kim et al., 2019). On the other hand, this challenges how to properly evaluate DocNMT. Another observation is that applying DocInfer to SenNMT delivers a significant accuracy improvement on En-Fr contrastive test set (+8.5%, Table 5), but slightly worse results on En-De. To accurately recognize the correct translation in these test sets, models need to leverage context. Such improvement might suggest that SenNMT has some limited capability of contextual modeling, but might just reflect the instability of small-scale test sets (only 200 cases in En-Fr test set, indicating a radius of around 7% for the 95% confidence interval). To some extent, this devalues the improvement achieved by 12N transfer as shown in Table 3, but strengthens the success of N21 transfer (often > 9% gains). This paper studies the variables playing role in achieving zero-shot document-level translation capability for languages that only have sentence level data (students), through multilingual transfer from languages that have access to document level data (teachers). We make the first step in this direction by extensively exploring properties of transfer by investigating three different variables. Our experiments on Europarl-7 and IWSLT-10 confirm the feasibility, where we discover that increasing document-supervised teacher languages thereby increasing the document training data size, adequately balancing between document and sentence data at training, and leveraging monolingual documents via back-translation all benefit zero-shot transfer in varying degrees. The transferability of contextual modeling in DocNMT demonstrates the potential of delivering multilingual DocNMT with limited document resources. Along with the success of document-level zero-shot transfer, problems with accurately estimating the document-level translation become challenging. BLEU often fails to capture document phenomena, while contrastive test sets only cover few document-level aspects. Neither perfectly corre-4183 lates with human evaluation. Besides, whether the gains really come from contextual modeling is still unclear. Our human evaluation shows some preference to DocNMT with SenInfer where context is not used for decoding at all. Designing better evaluation protocols (either automatic or human) is again confirmed to be critical. Besides, performing analysis beyond 12N and N21 transfer deserves more effort and it is an interesting and plausible future direction to analyze how language similarity affects the transfer. We thank the reviewers for their insightful comments. We want to thank Macduff Hughes and Wolfgang Macherey for their valuable feedback. We would also like to thank the Google Translate team for their constructive discussions and comments.
abstain abstain method objective objective result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method objective abstain abstain method method method method abstain objective objective objective objective objective objective other other other method method other other other objective objective objective method method other method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain method method method method abstain method method method abstain method abstain method abstain abstain method method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method abstain result abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain result objective method abstain abstain abstain abstain abstain result abstain abstain other other other