Recent work in multilingual machine translation (MMT) has focused on the potential of positive transfer between languages, particularly cases where higher-resourced languages can benefit lower-resourced ones. While training an MMT model, the supervision signals learned from one language pair can be transferred to the other via the tokens shared by multiple source languages. However, the transfer is inhibited when the token overlap among source languages is small, which manifests naturally when languages use different writing systems. In this paper, we tackle inhibited transfer by augmenting the training data with alternative signals that unify different writing systems, such as phonetic, romanized, and transliterated input. We test these signals on Indic and Turkic languages, two language families where the writing systems differ but languages still share common features. Our results indicate that a straightforward multi-source self-ensemble  training a model on a mixture of various signals and ensembling the outputs of the same model fed with different signals during inference  outperforms strong ensemble baselines by 1.3 BLEU on both language families. Further, we find that incorporating alternative inputs via self-ensemble can be particularly effective in low-resource settings, leading to +5 BLEU when only 5% of the total training data is accessible. Finally, our analysis demonstrates that including alternative signals yields more consistency and translates named entities more accurately, which is crucial for increased factuality of automated systems. improvement benefits languages with large quantities of high-quality training data (high-resource languages). Recently, researchers have focused on the development of multilingual translation models (Aharoni et al., 2019; Fan et al., 2020) capable of translating between many different language pairs rather than specialized models for each translation direction. In particular, such multilingual models hold great promise for improving translation quality for low-resource languages, as grouping languages together allows them to benefit from linguistic similarities as well as shared data between related languages. For example, training a translation system with combined Assamese and Bengali data would enable transfer learning between the two languages. We investigate how to enable multilingual translation models to optimally learn these similarities between languages and leverage this similarity to improve translation quality. The fundamental unit representing lingual similarity is the token  languages that are similar often have similar words or phrases  and during training, translation models can learn strong representations of tokens in low-resource languages if they are also present in high-resource languages. However, a challenge arises when similar languages share only a small amount of tokens, which inhibits the transfer to limited and trivial cases of token sharing, e.g., punctuation marks and digits. This is particularly clear in cases where similar languages are written in different scripts, as the amount of shared tokens is small compared to languages using the same written script. An example would be Hindi and Gujarati, which have phonetic similarity but are written in their own native scripts. To tackle inhibited transfer due to distinct writing systems, we transform the original input via transliteration , the process of converting text from one script to another, to get alternative signal from the original source sentences. Transliteration has 5291 Model __bn_romani__ pradhanmantri __bn__  __bn_ipa__ p  hanm  nt  i Average Input Log probabilities Prime Minister Output Decoding __bn_inscrip__     Figure 1: A generic illustration of self-ensemble for a multilingual translation system while translating Bengali to English. been used in many real world cases, such as converting Cyrillic Serbian to Latin Serbian, as the language is commonly written with both scripts, or typing in romanized Hindi for convenience on a Latin-script keyboard. To unify various writing scripts to increase token overlap, we experiment with three types of transliteration: (1) transliterate into phonemes expressed by international phonetic alphabet (IPA), (2) transliterate into Latin script (ROMANI ), and (3) transliterate into a script used by another language within the same language family (INSCRIP ). Beyond training on alternative inputs created through transliteration, we also systematically examine approaches to combining different signals. Our experimental results on Indic and Turkic datasets demonstrate that (i) a self-ensemble (Figure 1)  training a model on the mixture of different signals and using an ensemble of the same model given different input signals during inference time, outperforms other methods such as multi-source ensemble and multi-encoder architecture, which require training multiple models or significant architectural changes. (ii) Further, without the need for additional bitext, a self-ensemble over the original and transliterated input consistently outperforms baselines, and is particularly effective when the training set is small (e.g. low-resource languages) with improvements of up to +5 BLEU. (iii) Finally, the improvements in BLEU originate from clear gain in the accuracy and consistency in the translation of named entities, which has strong implications for increased factuality of automated translation systems. languages can benefit from similarities to high-resource languages where data is plentiful. However, surface-level differences between languages, such as writing system, can obscure semantic similarities. We describe an approach to transliterating input sentences to various alternative forms that maximize transfer learning between different languages, and various modeling approaches to incorporating such varied inputs. While training a multilingual translation system, tokens shared by multiple source languages serve as anchors to transfer information obtained from learning one language pair to the other. For example, the translation of terisini' in low-resourced Uzbek data can benefit from the word derisinin' in relatively high-resourced Turkish data after tok-enizing into sub-word units. However, the transfer is hindered when the amount of shared tokens is small  exacerbated by cases where the source and target languages are written in different scripts. 1 To alleviate the issue of various writing systems and encourage languages to transfer, we focus on alternative signals that unify the script of source languages and have larger token overlap. The core concept we explore is how to best leverage transliteration , or the process of converting the text from one script to the other. We demonstrate that transliteration can be an effective data augmentation approach that improves the translation performance without the need of acquiring additional parallel data. We explore three alternative inputs that allow 1 Muller et al. (2021) show that the discrepancy in scripts causes failure to transfer in multilingual models and further hurts performance in downstream tasks. models to share information more easily across languages with low token overlap but high semantic similarity. Figure 4 in Appendix C shows example alternative signals of the same Oriya sentence. Phonetic Input. Related languages in the same language family usually sound similar, such as languages in the Romance language family and those in the Indo-Aryan language family. Although cognates can be captured to some degree for Romance languages on subword-level, it is difficult for the Indo-Aryan family as those languages use different writing systems. Therefore, to fully exploit shared information, we transform the original textual input (BASE ) into the phonetic space, where the basic units are phonemes expressed in international phonetic alphabet (IPA). For example, ' in Bengali looks like phanmnti' in IPA form. Romanized Input. Many languages use Latin alphabet (or Roman alphabet) in their default writing system, if not, they more or less have romanization of their default script in order to accommodate conventional keyboards, e.g., Chinese can be typed on U.S. keyboards through Pinyin, the romanization of Chinese. To utilize this existing form of alternative input, the romanized input is another signal we explore in this work. For example, ' looks like pradhanmantri' in romanized form. In-family Script Input. The two previous alternative representations introduce tokens not present in the existing vocabulary, which increases the number of input and output representations the translation models must learn. Further, phonetic input is artificial in the sense that it is not used by people to communicate to each other in written form  and only used for pronunciation. Romanization naturally would introduce many additional tokens if the source language does not use Latin script. A third alternative that does not suffer these drawbacks is transliterate source language into the script of any of the other source languages in the multilingual translation model. To take advantage of language relatedness (Dhamecha et al., 2021), we unify the source languages with the script used by a language within the same language family (INSCRIP ). This method has the additional advantage of not needing to learn new subword tokenization models or replace the old vocabulary with a new one since all the inputs are expressed in one of the existing multilingual model's source language scripts. For example, ' looks like    ' when transliterated into Hindi script. Advantages of Transliterated Inputs. Various different input representations have been inserted into translation models, from parse trees (Li et al., 2017; Currey and Heafield, 2018) to pretrained embeddings (Artetxe et al., 2018; Conneau et al., 2018). Compared to these alternatives, transliteration has several clear advantages. Most importantly, transliteration is fast and accurate. Several existing alternatives often use other models to produce a different input, such as a parse tree, which cascades error from the first model into the translation model. Comparatively, the alphabet alignment between various writing systems is quite well known, even for many low-resource languages, as alphabet is one of the foundational aspects of studying any new language. Similarly, phonetic pronunciation guides are often widely available. These resources are also easily accessible programmatically, making them ideal for converting large quantities of supervised training data, for instance, the espkea-ng tool supports phonemization of more than 100 languages and accents. Beyond the ease of creating transliterations, we emphasize that this technique does not require any data annotation or collection of parallel data. Thus, it can be utilized in any existing translation system. How can additional transliterated inputs be incorporated into modern machine translation architectures? Since each alternative signal could capture a different view of the original input, in addition to training on each of the individual alternative signal alone, we investigate different approaches to combining them. Straight Concatenation The simplest combination strategy is to concatenate different input signals and separate them by a special token. For instance, to combine the original and phonetic input, we re-arrange the input to be of the format:  [original input] [SEP] [phonetic input] . During training, the decoder explicitly attends to tokens in both input signals. The advantage of this method is that no architectural change is required as all modi-fication is operated on the input data. However, as the concatenated input becomes longer, this method requires more computation to train compared to the baseline model trained on the original input only. Multi-Encoder Architectures Prior works have found multi-encoder architecture to be effective for multi-source machine translation (Nishimura et al., 2018). To cope with input from different sources, each encoder in the multi-encoder architecture deals with one type of input. To attend to multiple encoders on the decoder side, four cross-attention mechanisms can be adopted. We direct the reader to Appendix A for a detailed description of these attention variations. Although prior work investigates the efficacy of this approach, it is a complicated model choice requiring non-trivial architectural changes. Multi-Source Ensemble Ensembles are usually employed to boost the performance of a translation system. In a standard setup, each ensemble component is trained with identical configuration except for the random seed. We generalize this method to multi-source ensemble, i.e., individual ensemble components are trained on different transliterated inputs. During inference time, each component is fed with the type of transliteration it was trained on and produces the predicted log probabilities, which are averaged over all components for the subsequent decoding process. It is important for models trained on different source signals to have the same target vocabulary so that the average of log probabilities can happen. Unlike the previous two methods, this approach requires training multiple full models, thus requiring even more computation. Multi-Source Self-Ensemble Ensembling models that are trained on different input transliterations has the advantage that each individual model is maximally simple  only the input data for training changes. However, it comes with the downside that multiple different models need to be trained. This creates challenges particularly when models grow in size, as a new model would need to be created for each different transliterated input. Instead, we propose the Multi-Source Self-Ensemble , which has all the advantages of traditional ensembling, but only requires one model to be trained. Previous works in self-ensembles have focused on model robustness (Liu et al., 2018), which is distinct from varying input representations. Other work creates inputs in different languages (Fan et al., 2020), but have to use a translation model to create those inputs first. target sentence. Concretely, the model is trained on the mixture of various input signals, each preceded by a special language token indicating which type of signal this input belongs to. At inference time, the alternative transliterated signals of the same test sentence are fed to the same model and the log probabilities produced by these separate passes are averaged as in multi-source ensemble. This approach is simple to implement as it requires no architectural change, meaning the transliterated inputs we propose can be added seamlessly to any existing translation library. Unlike multi-source ensemble, only one model needs to be trained, stored and loaded for inference, greatly simplifying the ensembling process and increasing the scalability of our approach (particularly as translation models increase in size). To enforce fair comparison between multi-source self-ensemble and multi-source ensemble, we scale the former so that it has the same number of parameters as that of all ensemble components of the latter. For the purpose of minimally impacting inference speed, the scaling is done only to the encoder embedding dimension so that the decoder remains the same. Dataset We train our model on two language families: Indic and Turkic. The Indic dataset is from the WAT MultiIndic MT task 2 , including 10 Indic languages and in total around 11 million Indic-English bi-texts. Six of the Indic languages are Indo-Aryan languages and the rest are Dravidian languages. All of these languages use a different writing system. The Turkic dataset is collected from the open parallel corpus (Tiedemann, 2012) 3 . For relatively high-resourced language Turkish, we randomly select 4 million subset from the CCAligned (El-Kishky et al., 2020) corpus. Within this dataset, two languages use Cyrillic alphabet (Kazakh and Kyrgyz) and the rest use Latin alphabet. Detailed dataset statistics are displayed in Table 7 in Appendix B. Single-input model To test the effectiveness of each input signal, we train models on each single type of input: original input (BASE ), phonetic input (IPA), romanized input (ROMANI ) or input all expressed in the script of a language within the same language family (INSCRIP ). On the Indic dataset, 2 https://lotus.kuee.kyoto-u.ac.jp/WAT/ indic-multilingual/index.html 3 https://opus.nlpl.eu/ 5294  93 M parameters  2  93 M parameters Indic Turkic Indic Turkic Single-input Original Standard Ensemble BASE 33.6 20.3 BASE +B ASE 34.5 21.1 Single-input Alternative Multi-Source Ensemble IPA 32.7 17.9 BASE +IPA 34.3 20.9 ROMANI 32.5 20.7 BASE +R OMANI 34.4 21.4 INSCRIP 33.4 20.5 BASE +I NSCRIP 34.5 21.5 Multi-Source Self-Ensemble Multi-Source Self-Ensemble BASE +IPA 34.1 20.5 BASE +IPA 35.7 21.9 BASE +R OMANI 33.8 20.9 BASE +R OMANI 35.7 22.2 BASE +I NSCRIP 34.2 21.3 BASE +I NSCRIP 35.8 22.4 Table 1: BLEU scores on Indic test set and FloRes Turkic Devtest set. for the INSCRIP signal, all Indo-Aryan languages are transliterated into Hindi script, and all Dravidian languages into Tamil script. On the Turkic dataset, all languages in Latin script are transliterated into Cyrillic script. Multi-Source Ensemble A baseline for ensembling models trained on different signals is the standard ensemble (BASE +B ASE ) where two BASE models are ensembled, each trained with a different random seed. Although there are multiple combinations of input signals, we only discuss the cases where BASE is combined with one of {IPA, ROMANI , INSCRIP }, since in our preliminary experiments, we found dropping the BASE model leads to significantly degraded performance. Multi-Source Self-Ensemble Similar to above, we train a single model on the mixture of original input and one of {IPA, ROMANI , INSCRIP } input for multi-source self-ensemble. To enforce fair comparisons with the ensembled models, which have more parameters in total, we train two sizes of the self-ensemble (SE) model, one having the same size of a single baseline model, the other scaled to have twice the number of parameters of a single BASE model. Data Preprocessing We use espeak-ng 4 to convert the original input to phonetic input. For Indic languages, we use indic-trans 5 (Bhat et al., 2015) to obtain the romanized as well as the in-family transliterated input. On the 4 https://github.com/espeak-ng/ espeak-ng 5 https://github.com/libindic/ indic-trans Turkic dataset, we manually align the Cyrillic and Latin alphabet and substitute the letter(s) in one script with the corresponding one in another. 6 The Indic languages are tokenized with indic_nlp_library and the rest are tokenized with mosesdecoder 7 . We use sentencepiece 8 to create 32K BPE (Sennrich et al., 2016) subword vocabularies for each type of input signal. Examples longer than 250 tokens are discarded. We merge the source dictionaries of different signals by dropping duplicated tokens, while keeping the decoder dictionaries all the same in order to compute the average log probabilities in ensemble settings. Training & Evaluation We train many-to-En language directions during training (10 and 5 directions for Indic and Turkic dataset respectively). The architecture is a standard 6-layer encoder 6-layer decoder Transformer model, with 512 embedding dimension and 2048 hidden dimension in the default setting. For the scaled self-ensemble model, we increase the encoder hidden dimension such that the number of parameters in this model approximately matches that of n baseline models ( n = 2 for results in Table 1). We use 4000 warmup steps and learning rate 0 . 0003 . Both the dropout and attention dropout rate are set to 0 . 2 . Label smoothing is set to 0.1. Data from different language pairs are sampled with 1.5 temperature sampling. We 6 The substitution process starts from the letter in the target script that corresponds to the most number of letters in the source script. train all models for 18 epochs and 40 epochs for Indic and Turkic dataset respectively and evaluate the best checkpoint selected by dev loss. We use spBLEU 9 (Goyal et al., 2021; Guzmn et al., 2019) to compute the BLEU scores. 10 4 Results In this section, we compare the performance of our proposed multi-source self-ensemble model to various alternative ways of input combinations on two low-resource language families: Indic and Turkic languages. Furthermore, we show multi-source self-ensemble learns faster and generates more consistent and accurate translations. Our method is based on the hypothesis that incorporating alternative inputs increases the token overlap of source languages, which benefits the transfer during training. To verify this, we compute average sentence-level uni-gram overlap of all source language pairs (Table 2) and find that alternative signals do have higher token overlap compared to the original input. For instance, the IPA signal, having similar average sentence length as BASE , has much higher token overlap (0.15 vs. 0.03). Do increased token overlaps result in better translation performance? We train models on each of the alternative inputs alone and report the results in the left column of Table 1. We find that using only one alternative input in the source has either worse or similar performance as the original baseline, indicating higher token overlap among source languages does not guarantee better BLEU scores. The degraded performance is likely due to unfavorable interference introduced by shared tokens in 9 https://github.com/facebookresearch/ flores#spm-bleu 10 While prior work (Kocmi et al., 2021) has shown better correlation between neural metrics and human ratings, there have not been extensive evaluations for low-resource languages, especially for systems dealing with various writing scripts. Therefore we use spBLEU which is consistent with previous works (Goyal et al., 2021). the alternative signals. The interference may create information loss 11 or increased ambiguity 12 , which reinforces the importance of combining alternative inputs with the original input. Due to undesired interference exhibited in the alternative input spaces, we therefore adopt the input combination using our proposed Multi-Source Self-Ensemble to combine the original input and alternative signals. Results in left lower part of Table 1 demonstrate improvements over the single-input baseline. Our best performing alternative input configuration improves +1.0 BLEU on Turkic languages and +0.6 BLEU on Indic languages for 93M parameter models. In production, model ensembles are often employed to achieve the best possible performance. This is usually done by training multiple models each initialized with a different random seed (Baw-den et al., 2020; Tran et al., 2021b), and averaging the predicted next token probabilities at inference time. We also provide results against these strong ensemble baselines and observe +1.3 BLEU improvements on both Indic and Turkic languages. Note that, to enforce a fair comparison, we compare a scaled version of the multi-source self-ensemble model which has the same number of parameters as multiple ensemble baseline components. 11 For example, the punctuation marks are lost during phonemization process. 12 For instance, multiple words may have the same pronunciation and thus have the same input in IPA form, which makes the learning harder. Architectural Simplicity. As introduced in 2.2, there are various ways to incorporate multiple inputs, such as concatenation to form a longer input or using multiple encoders networks. In Table 3, we show that using multiple encoders has no improvements over the comparable baseline with raw text input, and straight concatenation only brings marginal gains (+0.1 BLEU). Further, our simple but effective Multi-Source Self-Ensemble technique reaches the same performance as that of a much larger quad-encoder model, which requires non-trivial architectural changes and takes more compute to train. Thus, our technique is suitable to be used out of the box in any seq-to-seq library. Faster Learning in Low-Resource Settings. To understand how self-ensemble performs with different amounts of data, we plot the learning curve of both the baseline and the self-ensemble model on 5% 13 to 80% of the total Indic training set. 14 As 13 When the training set is very small ( 5% and 10% ), we train for 60 epochs and select the model by dev loss. shown in Figure 2, the self-ensemble model outperforms the baseline model by a large margin when the amount of training data is small ( +5 BLEU when only 5% of the total set is used for train-ing). This is the scenario for most low-resource languages, as the gap gradually closes when more data is available. Overall, the multi-source self-ensemble model is consistently better than the baseline model irrespective of training data scale. This suggests that transliteration can be a cheap and effective data augmentation approach when used in conjunction with multi-source self-ensemble. Improved Output Consistency. We conduct a deeper analysis to understand the performance improvement of Multi-Source Self-Ensembles beyond BLEU scores alone. We find that our proposed technique generates much more consistent output, which could be a benefit of alternative signals transferring information more easily amongst source languages. We propose consistency BLEU ( C-BLEU ) to quantify the consistency of multi-way evaluation output of a many-to-En translation model. We treat the output of L 1 -En direction as reference and output of all other L i -En directions as hypothesis. We compute this for all N source languages in the dataset, accounting for total N ( N  1) C-BLEU scores, then take the average of all (Table 4). While training on IPA or ROMANI alone does not outperform the baseline in terms of C-BLEU, model trained on INSCRIP input improves the score by +1.3. Self-ensemble over BASE and IPA increases the C-BLEU to 36.2 (and from 36.3 to 38.1 with scaled model), indicating the alternative signals are best trained together with the original input. Improved Named Entity Accuracy. The previous analysis implies the self-ensemble model outputs more consistent translation, yet this does not mean the consistent translations are accurate. In this section, we conduct an analysis targeted at named entities. We use spaCy (Honnibal et al., 2020) NER tagger to extract all named entities, and then compute the exact match of the extracted entities. According to the results in Table 4, self-ensemble introduces small gains (+0.5) in terms of named entity F1 (NE-F1), whereas the scaled self-ensemble boosts NE-F1 score by +1.1 . Although the improvement is small in aggregate, we find significant improvement when breaking down by entity type. As shown in Figure 3, the multi-source self-ensemble model (without scaling) outperforms the baseline model on certain entity types, e.g., person, organization, time and event by a large margin. Our work can be viewed as multilingual MT (Firat et al., 2016) combined with multi-source MT (Zoph and Knight, 2016), where the sources are not other languages but rather alternative transliterated signals. The transliterated input has been explored in the past for translation system. Nakov and Ng (2009) use transliteration as a preprocessing step for their phrase-based SMT model to tackle systematic spelling variation. Both Chakravarthi et al. (2019) and Koneru et al. (2021) convert Dravidian languages to Latin script and train multilingual models with both source and target in Latin script; the latter identify code-switching to be a challenge during back-transliteration. Besides converting to Latin script, Dabre et al. (2018) use another common script, Devanagari, for Indic languages. In addition to the natural written scripts, previous works also explored artificial script, such as IPA. Liu et al. (2019) incorporate phonetic representations, specifically for Chinese Pinyin, to cope with homophone noise. Unlike our work, Chakravarthi et al. (2019) adopt transliteration to IPA for both the source and target. Apart from transliterated input, other potential alternative signals we did not fully explored include orthographic syllable units (Kunchukuttan and Bhattacharyya, 2016, 2020), morpheme-based units (Ataman et al., 2017; Dhar et al., 2020), and character (Lee et al., 2017) or byte (Wang et al., 2019a) level input in addition to the subword-level units (Sennrich et al., 2016). Multi-encoder architecture is the most common way to combine input from different sources. While previous works mainly use additional encoders to encode syntactic information (Li et al., 2017; Currey and Heafield, 2018) or input in another language (Nishimura et al., 2018), we feed in each encoder with different signals of the same sentence. Prior works also investigated approaches to combining input at different granularity (Ling et al., 2015; Chen et al., 2018; Casas et al., 2020). Wang et al. (2019b) combine the decoupled lexical and semantic representations through an attention mechanism. Another common method of utilizing additional input signal is multi-task learning, force the model to output extra labels (Luong et al., 2016; Grnroos et al., 2017). Apart from combining the sources during training, inference-time ensemble (Garmash and Monz, 2016) is often adopted by recent submissions to shared MT tasks (Ng et al., 2019; Tran et al., 2021a). The ensemble components are usually separate systems trained with different random initialization or language pairs. Fan et al. (2020) ensemble the same 5298 model by feeding in source sentences in different languages. The self-ensemble approach was also found to make networks more robust after adding random noises (Liu et al., 2018). Prior work also uses the term "self-ensemble" to refer to an ensemble of models using weights from different time steps during training (Xu et al., 2020). To overcome the low token-overlap issue exhibited in multilingual MT systems due to distinct writing system, we examined three alternative signals (phonetic, romanized and in-family transliterated input) and investigated four approaches (input concatenation, multi-encoder, multi-source ensemble, self-ensemble) to combining them with the original input. Our results show that training a single model with a mixture of diverse signals and performing self-ensemble during inference time can improve BLEU by 1.3 points on Indic and Turkic dataset. The improvements can reach +5 BLEU when training data size is small. Further, we show this approach generate more accurate and consistent translation of named entities which greatly impacts the factuality accuracy of news translation. We thank Shiyue Zhang, Xiang Zhou, Jean Mail-lard, Yixiao Song, and Marzena Karpinska for the helpful discussions during the course of this work.
abstain abstain abstain objective objective result result objective abstain abstain abstain abstain objective abstain abstain abstain abstain method abstain abstain method method objective abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain method other abstain abstain abstain method method method method method other abstain method abstain method method abstain method method abstain abstain abstain abstain abstain method abstain method method method abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain method other other other other other other method other other method other other other other other other other other objective result abstain result other