It has been shown that machine translation models usually generate poor translations for named entities that are infrequent in the training corpus. Earlier named entity translation methods mainly focus on phonetic transliteration, which ignores the sentence context for translation and is limited in domain and language coverage. To address this limitation, we propose DEEP, a DE noising E ntity P retraining method that leverages large amounts of monolingual data and a knowledge base to improve named entity translation accuracy within sentences. Besides, we investigate a multi-task learning strategy that finetunes a pre-trained neural machine translation model on both entity-augmented monolingual data and parallel data to further improve entity translation. Experimental results on three language pairs demonstrate that DEEP results in significant improvements over strong denoising auto-encoding baselines, with a gain of up to 1.3 BLEU and up to 9.2 entity accuracy points for English-Russian translation. 1 1 Introduction Proper translation of named entities is critically important for accurately conveying the content of text in a number of domains, such as news or encyclopedic text (Knight and Graehl, 1998; Al-Onaizan and Knight, 2002a,b). In addition, a growing number of new named entities (e.g., person name, location) appear every day, therefore many of these entities may not exist in the parallel data traditionally used to train MT systems. As a result, even state-of-the-art MT systems struggle with entity translation. For example, Laubli et al. (2020) note that a Chinese-English news translation system that had allegedly reached human parity still lagged far behind human translators on entity translations, and this prob-1 Code/data/models are released at https://github. Because of this, there have been a number of methods proposed specifically to address the problem of translating entities. As noted by Liu (2015), earlier studies on named entity translation largely focused on rule-based methods (Wan and Verspoor, 1998), statistical alignment methods (Huang et al., 2003, 2004) and Web mining methods (Huang et al., 2005; Wu and Chang, 2007; Yang et al., 2009). However, these methods have two main issues. First, as they generally translate a single named entity without any context in a sentence, it makes it di-cult to resolve ambiguity in entities using context. In addition, the translation of entities is often performed in a two-step process of entity recognition then translation, which complicates the translation pipeline and can result in cascading errors (Huang et al., 2003, 2004; Chen et al., 2013). In this paper, we focus on a simple yet eective method that improves named entity translation within context. Specifically, we do so by devising a data augmentation method that leverages two data sources: monolingual data from the target language and entity information from a knowledge base (KB). Our method also adopts a procedure of pre-training and finetuning neural machine translation (NMT) models that is used by many recent works (Luong and Manning, 2015; Neubig and Hu, 2018; Song et al., 2019; Liu et al., 2020). In particular, pre-training methods that use monolingual data to improve translation for low-resource and medium-resource languages mainly rely on a denoising auto-encoding objective that attempt to reconstruct parts of text (Song et al., 2019) or the whole sentences (Liu et al., 2020) from noised input sentences without particularly distinguishing named entities and other functional words in the sentences. In contrast, our method exploits an entity linker to identify entity spans in the monolingual sentences and link them to a KB that contains mul-1753 Krasnodar (Q3646) Language Label Description English Krasnodar capitalofKrasnodarregion(Krai)inSouthernRussia Russian  , : : Language Label ... tilingual translations of these entities (such as Wikidata (Vrandei and Krtzsch, 2014)). We then generate noised sentences by replacing the extracted entity spans with their translations in the knowledge base and pre-train our NMT models to reconstruct the original sentences from the noised sentences. To further improve the entity translation accuracy and avoid forgetting the knowledge learned from pre-training, we also examine a multi-task learning strategy that finetunes the NMT model using both the denoising task on the monolingual data and the translation task on the parallel data. In the experiments on English-Russian, English-Ukrainian, and English-Nepali translations, DEEP outperforms the strong denoising auto-encoding baseline with respect to entity translation accuracy, and obtains comparable or slightly better overall translation accuracy as measured by BLEU. A fine-grained analysis shows that our multi-task finetuning strategy improves the translation accuracy of the entities that do not exist in the finetuning data. Given a set of monolingual text segments for pretraining, i.e.,   D  , a sequence-to-sequence denoising auto-encoder is pre-trained to reconstruct a text segment  from its noised version corrupted by a noise function  () . Formally, the DAE objective is defined as follows: LDAE (D  ,  ) = (cid:213)  D  log  (  |  (  ) ;  ) , (1) where  denotes the model's learnable parameters. For notation simplicity, we drop  in the rest of the sections. This formulation encompasses several dierent previous works in data augmentation for MT, such as monolingual data copying (Currey et al., 2017), where  () is the identity function, back translation (Sennrich et al., 2016), where  () is a backwards translation model, as well as heuristic noise functions (Song et al., 2019; Lewis et al., 2020; Liu et al., 2020) that randomly sample noise according to manually devised heuristics. In particular, as our baseline we focus on the mBART method (Liu et al., 2020), a popular method with two types of heuristic noise functions being used sequentially on each text segment. The first noise function randomly masks spans of text in each sentence. Specifically, a span length is first randomly sampled from a Poisson distribution (  = 0 . 35) and the beginning location for a span in  is also randomly sampled. The selected span of text is replaced by a mask token. This process repeats until 35% of words in the sentence are masked. The second noise function is to permute the sentence order in each text segment with a probability. Our method adopts a procedure of pre-training and finetuning for neural machine translation. First, we apply an entity linker to identify entities in a monolingual corpus and link them to a knowledge base (3.1). We then utilize entity translations in the knowledge base to create noisy code-switched data for pre-training (3.2). Finally, we examine a multi-task learning strategy to further improve the translation of low-frequency entities (3.3). The goal of this part is to identify entities in each monolingual segment and obtain their translations. To this end, we use Wikidata (Vrandei and Krtzsch, 2014) a public multilingual knowledge base that covers 94M entities. 2 Each entity is represented in surface forms from dierent languages in which a Wikipedia article exists. Therefore, linking an entity mention  in a target-language segment  to an entity  in Wikidata allows us to obtain the multilingual translations of the entity, that is,    ,    KB :   = surface ( , KB ) ,     where   denotes a set of multilingual surface forms of  . We can define the translate operation as:  = lookup (   ,  ) which simply looks for the surface form of  in the source language  . Note that this strategy relies on the fact that translations in higher-resource languages are included in   , which we adopt by using English in our experiments. In general, however,   does not universally cover all the languages of interest. For entity recognition and linking, we use SLING (Ringgaard et al., 2017), 3 which builds an entity linker for arbitrary languages available in Wikipedia. After obtaining entity translations from the KB, we attempt to explicitly incorporate these translations into the monolingual sentences for pre-training. To do so, we design an entity-based noise function that takes in a sentence  and the KB, i.e.,  ( , KB ) . First, we replace all detected entity spans in the sentence by their translations from the KB: replace ( , KB ) = swap ( , ,  ) ,     (2) 2 Dump June 14, 2021. where the swap() function swaps occurrences of one entity span  in  with its translation  in the source language. For example, in the second box of Figure 1, the named entities ,  and  in Russian are replaced by Krasnodar, Saratov, and Ulyanovsk in English. After the replacement, we create a noised code-switched segment which explicitly includes the translations of named entities in the context of the target language. For some segments that contain fewer entities, their code-switched segments may be similar to them, which potentially results in a easier denoising task. Therefore, we further add noise to these code-switched segments. To do so, if the word count of the replaced entity spans is less than a fraction (35%) of the word count in the segment, we randomly mask the other non-entity words to ensure that about 35% of the words are either replaced or masked in the noised segment. Finally, we follow Liu et al. (2020) to randomly permute the sentence order in  . We then train a sequence-to-sequence model to reconstruct the original sentence  from its noised code-switched sentence as follows: LDEEP (D  , KB ) = (cid:213)  D  log  (  |  ( , KB )) 3.3 Multi-task Finetuning After pre-training, we continue finetuning the pre-trained model on a parallel corpus ( ,  )  D  for machine translation. To avoid forgetting the entity information learned from the pre-training stage, we examine a multitask learning strategy to train the model by both the pre-training objective on the monolingual data and the translation objective on the parallel data. Since monolingual segments are longer text sequences than sentences in D  and the size of D  is usually larger than that of D  , simply concatenating both data for multi-task finetuning leads to bias toward denoising longer sequences rather than actually translating sentences. To balance the two tasks, in each epoch we randomly sample a subset of monolingual segments D (cid:48)  from D  , where the total subword count of D (cid:48)  equals to that of D  , i.e., (cid:205)  D (cid:48)  |  | = (cid:205) ( , )D  max (|  | , |  |) . We 1755 Lang. where the pre-training objective L Pre-train is either DAE or DEEP with DEEP having an additional input of a knowledge base. Notice that with the sampling strategy for the monolingual data, we double the batch size in the multi-task finetuning setting with respect to that in the single-task finetuning setting. Therefore, we make sure that the models are finetuned on the same amount of parallel data in both the single-task and multi-task settings, and the gains from the multi-task setting sorely come from the additional task on the monolingual data. To distinguish the tasks during finetuning, we replace the start token ( [BOS] ) in a source sentence or a noised segment by the corresponding task tokens for the translation or the denoising task ( [MT] , [DAE] or [DEEP] ). We initialize these task embeddings by the start token embedding and append them to the word embedding matrix of the encoder. Pre-training Data: We conduct our experiments on three language pairs: English-Russian, English-Ukrainian and English-Nepali. We use Wikipedia articles as the monolingual data for pre-training and report the data statistics in Table 1. We tokenize the text using the same sentencepiece model as Liu et al. (2020), and train on sequences of 512 subwords. Finetuning & Test Data: We use the news commentary data from the English-Russian translation task in WMT18 (Specia et al., 2018) for finetuning and evaluate the performance on the WMT18 test data from the news domain. For English-Ukrainian, we use the TED Talk transcripts from July 2020 in the OPUS repository (Tiedemann, 2012) for finetuning and testing. For English-Nepali translation, Lang. we use the FLORES dataset in Guzmn et al. (2019) and follow the paper's setting to finetune on parallel data in the OPUS repository. Table 2 shows the data statistics of the parallel data for finetuning. Notice that from the last four columns of Table 2, the entities in the pre-training data cover at least 87% of the entity types and 91% of the entity counts in both finetuning and test data except the En-Ne pair. Architecture: We use a standard sequence-to-sequence Transformer model (Vaswani et al., 2017) with 12 layers each for the encoder and decoder. We use a hidden unit size of 512 and 12 attention heads. Following Liu et al. (2020), we add an additional layer-normalization layer on top of both the encoder and decoder to stabilize training at FP16 precision. We use the same sentencepiece model and the vocabulary from Liu et al. (2020). Random  MT : We include a comparison with a randomly initialized model without pre-training and finetune the model for each translation task. DAE  MT : We pre-train a model by DAE using the two noise functions in Liu et al. (2020) and finetune the model for each translation task. DEEP  MT : We pre-train a model using our proposed DEEP objective and finetune the model on the translation task. DAE  DAE+MT : We pre-train a model by the DAE objective and finetune the model for both the DAE task and translation task. DEEP  DEEP+MT : We pre-train a model by the DEEP objective and finetune the model for both the DEEP task and translation task. batch of 64 text segments, each of which has 512 subwords. We use the Adam optimizer (  =1e-6,  2 =0.98) and a polynomial learning rate decay scheduling with a maximum step at 500K. All models are pre-trained on one TPUv3 (128GB) for about 12 hours for 50K steps. 4 We apply the noise function on the monolingual data on the fly for each epoch, and this takes only a few minutes by multiprocessing in Fairseq (Ott et al., 2019). We then reset the learning rate scheduler and continue finetuning our pre-trained models on the MT parallel data for 40K steps. Single-task (multi-task) finetuning takes about 16 (32) hours on 2 RTX 3090 GPUs. We set the maximum number of tokens in each batch to 65,536 in the single task setting and double the batch size in the multi-task setting to ensure that models in both settings are trained on an equal amount of parallel data, and thus any performance gain can only be attributed to monolingual data during finetuning. We use 2,500 warm-up steps to reach a maximum learning rate of 3e-5, and use 0.3 dropout and 0.2 label smoothing. After training, we use beam search with a beam size of 5 and report the results in sacreBLEU (Post, 2018) following the same evaluation in Liu et al. (2020). In Table 3, we compare all methods in terms of BLEU (Papineni et al., 2002) and chrF (Popovi, 2015) on the test data for three language pairs. First, we find that all pre-training methods significantly outperform the random baseline. In particular, our DEEP method obtains a gain of 3.5 BLEU points in the single task setting for the low-resource En-Ne translation. Second, we compute statistical significance of the BLEU and chrF scores with bootstrap resampling (Koehn, 2004), and we observe significant improvements with the multi-task finetuning strategy over the single-task finetuning for En-Ru and En-Ne. Our DEEP method outperforms the DAE method for En-Ru translation by 1.3 BLEU points in the multi-task setting. It is also worth noting that DEEP obtains higher BLEU points than DAE at the beginning of the multi-task finetuning process, however the gap between both methods decreases as the finetuning proceeds for longer steps (See Appendix A). One possible reason is that models trained by DEEP benefit from the entity trans-4 As we show in Figure 4, models pre-trained for 50K steps provide a reasonably good initialization. lations in the pre-training data and obtain a good initialization for translation at the beginning of the finetuning stage. As the multi-task finetuning proceeds, the models trained by both DAE and DEEP rely more on the translation task than the denoising task for translating a whole sentence. Thus the nuance of the entity translations might not be clearly evaluated according to BLEU or chrF. Since corpus-level metrics like BLEU or chrF might not necessarily reveal the subtlety of named entity translations, in the section we perform a fine-grained evaluation by the entity translation accuracy which counts the proportion of entities correctly translated in the hypotheses. Specifically, we first use SLING to extract entities for each pair of a reference and a hypothesis. We then count the translation accuracy of an entity as the proportion of correctly mentioning the right entity in the hypotheses, followed by macro-averaging to obtain the average entity translation accuracy. We also show the accuracy scores in Table 3. First, our method in both singleand multi-task settings significantly outperformed the other baselines. In particular, the gains from DEEP are much clear for the En-Uk and En-Ru translations. One possible reason is that Russian or Ukrainian entities extracted from the pre-training data have a relatively higher coverage of the entities in both the finetuning and test data as reported in Table 2. However, SLING might not detect as many entities in Nepali as in the other languages. We believe that future advances on entity linking in low-resource languages could potentially improve the performance of DEEP further. We leave this as our future work. In this section, we further analyze the eect on dierent categories of entities using our method. Performance of Entity Groups over Finetuning: The model is exposed to some entities more often than others at dierent stages: pre-training, finetuning and testing, which raises a question: how is the entity translation aected by the exposure during each stage? To answer this question, we divide the entities appearing in the test data into three groups:  PFT : entities appearing in the pre-training, finetuning, and test data. FT We show the English-to-Russian entity translation accuracy scores for each group over finetuning steps in Figure 2. Overall, accuracies are higher for the entities that appear in the finetuning data ( PFT , FT ), which is due to the exposure to the finetuning data. Our proposed method consistently outperformed baseline counterparts in both singleand multi-task settings. The dierences in accuracy are particularly large at earlier finetuning steps, which indicates the utility of our method in lower-resource settings with little finetuning data. The eect of multi-task finetuning is most notable for entities in PT . Multi-task finetuning continuously exposes the model to the pre-training data, which as a result prevents the model from forgetting the learned entity translations from PT . Performance according to Entity Frequency: We further analyze the entity translation accuracy scores using entity frequencies in each group introduced above. This provides a more fine-grained perspective on how frequent or rare entities are translated . To do so, we take Russian hypotheses from a checkpoint with 40K steps of finetuning, bin the set of entities in three data ( i.e. PFT , PT , FT ) according to frequencies in each of the data. We then calculate the entity translation accuracy within each bin by comparing them against reference entities in the respective sentences. Figure 3 shows the accuracy gain of each pre-training methodologies from Random  MT ( i.e. no pre-training) on test data, grouped by the entity frequency bins in pre-training and finetuning data. Note that leftmost column and the bottom row represent PT , FT , respectively. As observed earlier, the proposed method improves more over most frequency bins, with greater dierences on entities that are less frequent in finetuning data. This tendency is observed more significantly for the multi-task variant ( DEEP  DEEP + MT ), where the gains are mostly from entities that never appeared in finetuning data ( i.e. leftmost column). Multi-task learning with DEEP therefore prevents the model from forgetting the entity translations learned at pre-training time. Analytical results on Ukrainian and Nepali are in Appendix B. 1758 0 ( 1 , 2 ] ( 2 , 6 ] ( 6 , 23 ] ( 23 , 1878 ] Freq. Finetuning Data Size vs Entity Translation: While DEEP primarily focuses on a low-resource setting, the evaluation with more resources can highlight potential use in broader scenarios. To this end, we expand the finetuning data for English-Russian translation with an additional 4 million sentence pairs from ParaCrawl (Ban et al., 2020), a parallel data collected from web pages. Although web pages might contain news text, ParaCrawl data covers more general domains. We finetune models on the combined data and evaluate with BLEU and entity translation accuracy. Table 4 shows the comparisons across dierent finetuning data sizes. When the model is initialized with pre-training methods, we observed decreased BLEU points and increased entity translation accuracy scores. This is partly due to the discrepancy of domains between our finetuning data (news) and ParaCrawl. Regardless, DEEP is consistently equal to or better than DAE in all tested settings. Pre-training Steps vs Entity Translation: Since DEEP leverages entity-augmented monolingual data, the model trained by DEEP revisits more entities in dierent context as the pre-training 0 25 50 100 150 200 Pre-training Steps (x 1000) 15 16 17 18 19 BLEUBLEU Entity Translation Accuracy 30 32 34 36 38 40 E n t i t y T r a n s l a t i o n A cc u r a c y Figure 4: English-to-Russian BLEU and Entity translation accuracy scores after finetuning from variable pretraining steps. proceeds. To analyze the eciency of learning entity translation during pre-training, we focus on the question: how many pre-training steps are needed for named entity translation? To examine this question, we take the saved checkpoints trained by DEEP from various pre-training steps, and apply the single-task finetuning strategy on the checkpoints for another 40K steps. We plot the entity translation accuracy and BLEU on the test data in Figure 4. We find that the checkpoint at 25K steps has already achieved a comparable entity translation accuracy with respect to the checkpoint at 150K steps. This shows that DEEP is ecient to learn the entity translations as early as in 25K steps. Besides, both the BLEU and entity translation accuracy keep improving as the pre-training steps increase to 200K steps. In this section, we select two examples that contain entities appearing only in the pre-training and testing data. The first example contains three location names. We find that the model trained by the single-task DAE predicts the wrong places which provide the wrong information in the translated sentence. In addition, the model trained by the multi-task DAE just copies the English named entities (i.e., Krasnodar, Saratov and Ulyanovsk) to the target sentence without actual translation. In contrast, our method predicts the correct translation for Krasnodar in both single-task and multi-task setting, while the multi-task DEEP translates all entities correctly. In the second example, although our method in the single-task setting predicts wrong for all the entities, the model generates partially correct translations such as  for  and  @-@  for -. Notice that DEEP in the multi-task setting translates the correct entities asphalt and Kras-noarmeyskiy which convey the key information in this sentence. In contrast, the translation produced by the multi-task DAE method literally means (Barnaul),  (new)  (myth)  (at) Krasnoarmey Prospekt,  (grow) Krasnoarmeski., which is incomprehensible due to the entity translation errors. Named Entity Translation has been extensively studied for decades (Arbabi et al., 1994; Knight and Graehl, 1998). Earlier studies focus on rule-based methods using phoneme or grapheme (Wan and Verspoor, 1998; Al-Onaizan and Knight, 2002b), statistical methods that align entities in parallel corpus (Huang et al., 2003, 2004; Zhang et al., 2005) and Web mining methods built on top of a search engine (Huang et al., 2005; Wu and Chang, 2007; Yang et al., 2009). Recently, Finch et al. (2016); Hadj Ameur et al. (2017); Grundkiewicz and Heafield (2018) used NMT to transliterate named entities without any sentence context . Another line of research (Ugawa et al., 2018; Li et al., 2018; Torregrosa et al., 2020; Modrzejewski et al., 2020; Zhou et al., 2020) only performs entity recognition and uses entity tags (e.g., person) which are not directly informative to the translation task, in contrast to the entity translations obtained by entity linking in our work. Besides, these methods modify model architecture to integrate entity tag embeddings or knowledge graph entity embeddings (Moussallem et al., 2019), which also require extracting entity information for both training and test data. In contrast, we focus on data augmentation methods to improve name entity translation within context , so our method is easily applicable to any architectures and test data without preprocessing. Pre-training of Neural Machine Translation has been shown eective by many recent works (Con-neau and Lample, 2019; Song et al., 2019; Liu et al., 2020; Lin et al., 2020), where dierent pre-training objectives are proposed to leverage monolingual data for translation. These methods adopt a denoising auto-encoding framework, which encompasses several dierent works in data augmentation on monolingual data for MT (Lambert et al., 2011; Currey et al., 2017; Sennrich et al., 2016; Hu et al., 2019). However, named entity translations during pre-training is under-explored. We fill this gap by integrating named entity recognition and linking to the pre-training of NMT. Moreover, while recent 1760 work shows that continue finetuning a pre-trained encoder with the pre-training objective improves language understanding tasks (Gururangan et al., 2020), this finetuning paradigm has not been explored for pre-training of a sequence-to-sequence model. Besides, previous works on multi-task learning for MT focus on language modeling (Gulcehre et al., 2015; Zhang and Zong, 2016; Domhan and Hieber, 2017; Zhou et al., 2019), while we examine a multi-task finetuning strategy with an entity-based denoising task in this work and demonstrate substantial improvements for named entity translations. In this paper, we propose an entity-based pretraining method for neural machine translation. Our method improves named entity translation accuracy as well as BLEU score over strong denoising auto-encoding baselines in both single-task and multi-task setting. Despite the eectiveness, several challenging questions remain open. First, recent works on integrating knowledge graphs (Zhao et al., 2020a,b) in NMT have shown promising results for translation. Our method links entities to a multilingual knowledge base which contains rich information of the entities such as entity description, relation links, and alias. How to leverage these richer data sources to resolve entity ambiguity deserves further investigation. Second, finetuning pre-trained models on in-domain text data is a potential way to improve entity translations across domains. This work was supported in part by a grant from the Singapore Defence Science and Technology Agency.
abstain abstain objective objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain method abstain objective abstain method result abstain result abstain abstain abstain other method abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain method method abstain method abstain method abstain abstain abstain method abstain method method method method abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method method abstain method abstain result result result abstain result abstain abstain abstain result abstain result result abstain abstain abstain abstain result abstain method abstain method abstain result abstain abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain result method abstain other abstain method method abstain result abstain abstain abstain abstain result abstain abstain method abstain abstain other other other abstain other objective other other other abstain other abstain objective method abstain abstain abstain abstain abstain other