Synthetic translations have been used for a wide range of NLP tasks primarily as a means of data augmentation. This work explores, instead, how synthetic translations can be used to revise potentially imperfect reference translations in mined bitext. We find that synthetic samples can improve bitext quality without any additional bilingual supervision when they replace the originals based on a semantic equivalence classifier that helps mitigate NMT noise. The improved quality of the revised bitext is confirmed intrinsically via human evaluation and extrinsically through bilingual induction and MT tasks. While human-written data remains the gold standard to train Neural Machine Translation ( NMT ) and Multilingual NLP models, there is growing evidence that synthetic bitext samplessentence-pairs that are translated by NMT benefit a wide range of tasks. They have been used to enable semi-supervised MT training from monolingual data (Sennrich et al., 2016a; Zhang and Zong, 2016; Hoang et al., 2018), to induce bilingual lexicons (Artetxe et al., 2019; Shi et al., 2021), and to port models trained on one language to an-other (Conneau et al., 2018; Yang et al., 2019). While synthetic bitexts are useful additions to original training data for downstream tasks, it remains unclear how they differ from naturally occurring data. Some studies suggest that synthetic samples might be simpler and easier to learn (Zhou et al., 2020; Xu et al., 2021). Recognizing that naturally occurring bitext can be noisy, for instance, when they are mined from comparable monolingual corpora (Resnik and Smith, 2003; Fung and Yee, 1998; Espl et al., 2019; Schwenk et al., 2021), we hypothesize that synthetic bitext might also directly improve the equivalence of the two bitext sides. Thus synthetic samples might be useful not only for data augmentation but also to revise potentially noisy original bitext samples. In this paper, we present a controlled empirical study comparing the quality of bitext mined from monolingual resources with a synthetic version generated via MT . We focus on the widely used WikiMatrix bitexts for a distant (i.e, EN-EL ) and a similar language-pair (i.e, EN-RO ), since it has been shown that this corpus contains a significant proportion of erroneous translations (Caswell et al., 2021). We generate synthetic bitext by translating the original training samples using MT systems trained on the bitext itself and therefore do not inject any additional supervision in the process. We also consider selectively replacing original samples with forward and backward synthetic translations based on a semantic equivalence classifier, which is also trained without additional supervision. We show that the resulting synthetic bitext improves the quality of the original intrinsically using human assessments of equivalence and extrinsically on bilingual induction ( BLI ) and MT tasks. We present an extensive analysis of synthetic data properties and of the impact of each step in its generation process. This study brings new insights into the use of synthetic samples in NLP . First, intrinsic evaluation shows that synthetic translations, in addition to normalizing the bitext (Zhou et al., 2020; Xu et al., 2021), could potentially provide reference translations that are more semantically equivalent to the source than the original ones. Furthermore, the improved bitext provides more useful signals for BLI tasks and NMT training in two settings (training from scratch; continued training), as confirmed by our extrinsic evaluations. Finally, ablation analyses that compare different ways to combine synthetic translations show that using both translation directions and filtering using semantic equivalence is key to improving bitext quality and calls for further exploration of best practices for using synthetic translations in NLP tasks. Synthetic Translations Generating synthetic translations has mainly been studied as a means of data augmentation for NMT through forward translation (Zhang and Zong, 2016) or backtranslation (Sennrich et al., 2016a; Marie et al., 2020) of monolingual resources. Moreover, recent lines of work use synthetic translations to augment the original parallel data: Nguyen et al. (2020) diversify the parallel data via translating both sides using multiple models and then merging them with the original to train a final NMT model; Jiao et al. (2020) employ a similar approach to rejuvenate inactive examples that contribute the least to the model performance. Sequence-level knowledge distillation (Kim and Rush, 2016) can also be viewed as replacing original bitext with synthetic translations. While its original goal was to guide the training of a student model of small capacity with the output of a teacher of high capacity, distillation is also necessary to effectively train some categories of MT architectures such as non-autoregressive models (Gu et al., 2018). While it is not entirely clear why synthetic distilled samples are superior to original bitext in this case, recent studies suggest that the synthetic samples are simpler and thus easier to learn from (Zhou et al., 2020; Xu et al., 2021). Synthetic Data Selection Prior work covers a wide spectrum of different selection strategies on top of synthetic translations generated from monolingual samples. Each of them focuses on identifying samples with specific properties: Axelrod et al. (2011) sample sentences that are most relevant to a target domain with the goal of creating pseudo in-domain bitext; Hoang et al. (2018) generate synthetic parallel data iteratively from increasingly better back-translation models for improving unsupervised NMT ; Fadaee and Monz (2018) focus on the diversity of synthetic samples and sample synthetic translations containing words that are difficult to predict using prediction losses and frequencies of words. By contrast, our empirical study investigates whether synthetic translations can be used to selectively replace original references to improve bitext quality rather than augmenting it. Bitext Quality Mining bitext from the web results in large-scale corpora that are usually collected without guarantees about their quality. For instance, they contain noisy samples, ranging Algorithm 1 Revising Bitext: Given a bitext D = ( S, T ) , a divergent scorer R , and a margin score t , return revised bitext  D 1: procedure TRAIN ( D = ( S, T ) ) 2: Train MS  T on D until convergence 3: return MS  T 4: end procedure 1: procedure EQUIVALIZE ( D = ( S, T ) ) 2: MS  T  TRAIN ( D = ( S, T ) ) 3: MT  S  TRAIN ( D = ( T, S ) ) 4:  D   5: for i  1 ,..., |D| do 6: ( S i ,  T i )  ( S i , MS  T ( S i ) ) 7: (  S i , T i )  ( MT  S ( T i ) , T i ) 8: d F  R ( S i ,  T i )  R ( S i , T i ) 9: d B  R (  S i , T i )  R ( S i , T i ) 10: if max ( d F , d B ) > t then 11: if max = d F then 12:  D   D  {( S i ,  T i )} 13: else 14:  D   D  {(  S i , T i )} 15: end if 16: else 17:  D   D  {( S i , T i )} 18: end if 19: end for 20: return  D 21: end procedure from untranslated sentences to sentences with no linguistic content (Khayrallah and Koehn, 2018; Caswell et al., 2020). Some of this noise is typically filtered out automatically using heuristics (Ramrez-Snchez et al., 2020) or NMT model scores (Junczys-Dowmunt, 2018; Koehn et al., 2019). Yet, even after this noise filtering, a wide range of the remaining samples contains fine-grained semantic divergences (Briakou and Carpuat, 2020). Our past work explored strategies to mitigate the impact of these divergences on MT models by incorporating divergence tags as token-level factors (Briakou and Carpuat, 2021), and designing an approach to automatically edit divergent samples with noisy supervision from monolingual resources (Briakou et al., 2021). By contrast, this work explores whether synthetic translations can be used to replace potentially fine-grained divergences using only the bitext we seek to revise. We rely on established techniques that can be applied using only the bitext that we seek to revise. First, we train NMT models on the original bitext to translate in both directions. For each original sentence-pair, we generate a pool of synthetic translations using NMT and apply a divergence ranking criterion to decide whether and how to replace the original references with a better translation. Algorithm 1 gives an overview of the process, and we describe each step below. Generating synthetic translations We train NMT models MS  T and MT  S on the original bitext to translate in each direction (lines 2 3 ). For each sentence-pair, they are used to generate two candidates for replacement by forward and backward translation (lines 6 7 ): ( S i , MS  T ( S i ) ) and ( MT  S ( T i ) , T i ). As a result, NMT models translate the exact same data that they are trained on. We thus expect translation quality to be high , and that local errors in the original bitext might be corrected by the translation patterns learned by NMT models on the entire corpus. Selective Replacement We propose to replace an original pair by a candidate only if the candidate is predicted to better convey the meaning of the source than the original, which we refer to as the semantic equivalence condition . We implement this by ranking the original sample ( S i , T i ), its revision by forward translation ( S i , MS  T ( S i ) ) and its revision by back-translation ( MT  S ( T i ) , T i ), according to their degree of semantic equivalence. If none of the synthetic samples score higher than the original, it is not replaced (line 17 ). Otherwise, the original is replaced by the highest scoring synthetic sample (lines 10 15 ). As a result the cardinality of the bitext remains constant. The semantic equivalence condition ( d F and d B (lines 8 9 )) is implemented using divergentm BERT , a divergent scorer introduced in our prior work (Briakou and Carpuat, 2020) that is trained on synthetic samples generated by perturbations of the original bitext (e.g., deletions, lexical or phrasal replacements) performed without any bilingual information. Bitext We evaluate the use of synthetic translations for revising bitext on two language pairs of the WikiMatrix corpus (Schwenk et al., 2021). WikiMatrix consists of sentence-pairs mined from Wikipedia pages using language agnostic sentence embeddings ( LASER ) (Artetxe and Schwenk, 2019). Prior work indicates that, as expected, the corpus as a whole comprises many samples that are not exact translations: Caswell et al. (2021) report that for more than half of the audited low-resource language-pairs, mined pairs are on average misaligned; Briakou and Carpuat (2020) find that 40% of a random sample of the English-French bitext are not semantically equivalent, and include fine-grained meaning differences in addition to alignment noise. We focus on bitexts with fewer than one million sentence pairs in Greek  English ( EL  EN , with 750 , 585 pairs) and Romanian  English ( RO  EN , with 582 , 134 pairs), because improving bitext is particularly needed in this data regime. In much higher resource settings, filtering strategies might be sufficient as there might be more high quality samples overall. In much lower resource settings, the data is likely too noisy or too small to effectively revise bitexts using NMT . We filter out noisy pairs in the training data using bicleaner (Ramrez-Snchez et al., 2020) so that our empirical study excludes the most obvious forms of noise, and focuses on the harder case of revising samples that standard preprocessing pipelines consider to be clean. 1 Preprocessing We use Moses (Koehn et al., 2007) for punctuation normalization, true-casing, and tokenization. We learn 32 KBPE s (Sennrich et al., 2016b) per language using subword-nmt 2 . NMT Models We use the base Transformer architecture (Vaswani et al., 2017) and include details on the exact architecture and training in Apendix C. Selective Replacement The divergence ranking models are trained using our public implementation of divergentm BERT (Briakou and Carpuat, 2020). 3 Synthetic divergences are generated starting from the 5 , 000 top scoring WikiMatrix sentences based on LASER score (i.e., seed equiva-lents). We fine-tune the  BERT -Base Multilingual Cased model (Devlin et al., 2019) and set the margin equal to 5 as per our original implementation. We use the same margin value for the margin score of Algorithm 1. 4 1 https://github.com/bitextor/bicleaner 2 https://github.com/rsennrich/subword-nmt 3 https://github.com/Elbria/xling-SemDiv 4 Our divergentm BERT yields 84 F 1 on a set of English-French human-annotated fine-grained divergences in WikiMatrix collected in our prior work (Briakou and Carpuat, 2020). We ask 3 bilingual speakers to evaluate the quality of the EN-EL bitexts. Given an original source sentence, they are asked to rank the original target and the candidate target in the order of their equivalence to the source. They are asked Which sentence conveys the meaning of the source better?, and ties are allowed. A random sample of 100 pairs from forward and backward MT is annotated. As can be seen in Table 2, 60% of ALL synthetic candidates are better translations of the WikiMatrix reference, which confirms the potential of NMT for improving over original translations. Further ablations confirm the benefits of selecting these synthetic candidates with the semantic equivalence condition. When the divergent scorer ranks a candidate higher than the original by a small margin (i.e., 0  d  5 given d = R ( S i , MS  T ( T i ))  R ( S i , T i )) ), human evaluation shows that the candidate is actually better than the original only 51% of the times. When using our exact semantic equivalence condition ( d > 5 ), can-Candidate set % Equivalized Kendall's  ALL 60 . didates are judged as more equivalent than the original 87 . 5% of the times, and annotations within this set have a stronger agreement (i.e., 0 . 688 Kendall's  ). This indicates that the condition d > 5 identifies more clear-cut examples of synthetic translations that fix semantic divergences in the original data and can be thus used for selective replacement of imperfect references by better quality translations. Further inspection of the annotations reveals that most source-target WikiMatrix examples contain fine meaning differences ( 56% ). In those cases, we observe that most of the content between the sentences is shared, but either small segments are 4756 PROPERTY ORIGINAL REVISED   1 : # Sentences 750 , 585 750 , 585 0 . mistranslated (e.g., London instead of Athens in the first example of Table 1), or some information is missing from either side of the pair (e.g., all six missing from the target side in the third example of Table 1). Furthermore, more coarse-grained divergences are found less frequently ( 12% )in those cases, we notice that sentences are usually either topically related or structurally similar (e.g., length, syntax) with a few anchor words (e.g., last example in Table 1). Finally, 32% of the times the original WikiMatrix pairs are perfect translations of each other. Figure 1 presents the distribution of lexical differences (i.e., computed using LeDa score that captures lexical differences based on the percentages of tokens that are not found in two sentences (Niu and Carpuat, 2020)) between original and synthetic translations (in EN ) for candidates that replace and do not replace the originals. 5 First, we observe that a substantial amount of synthetic translations that do not replace original references ( 40% ) corresponds to small LED scores ( < 0 . 1 ), suggesting that the equivalence criterion could fall back to the original sentence not because of the poor quality of candidate references, but rather due to them being already close to the originals. Furthermore, all synthetic translated instances are represented in almost all bins, with fewer instances found on the 5 LeD details are in Appendix A. Replaced Not Replaced Figure 1: LeD differences of original vs. synthetic translations ( EL  EN ). extreme bins of > 0 . 7 LED scores. Finally, synthetic translations that replace original references are mostly concentrated within the range [0 . 2 , 0 . 6] of LeD scores. This indicates that they share lexical content with the original, which further supports the hypothesis that synthetic translations revise fine-grained meaning differences in WikiMatrix in addition to alignment noise. Table 3 presents differences in statistics of the original vs. revised WikiMatrix EN-EL bitexts to shed more light on the impact of selectively using synthetic translation for bitext quality improvement. 6 The refined bitext exhibits higher coverage (i.e., ratio of source words being aligned by any target words; rows 5 and 13 ) and smaller complexity (i.e., 6 Details on the metrics are in Appendix A. 4757 the diversity of target word choices given a source word (Zhou et al., 2020)) compared to the original bitext. Moreover, the use of synthetic translations introduces small decreases in the lexical types covered in the final corpus (i.e., rows 3 and 11 ), which is expected as the additional coverage in the original corpus might be a result of divergent texts. Those observations are in line with prior work that seeks to characterize the nature of synthetic translations used in other settings, such as knowledge distillation (Zhou et al., 2020; Xu et al., 2021). While fixing divergent references contributes to this simplification effect, NMT translations might also reinforce unwanted biases from the original bitext. For instance, the distribution of two grammatical gender pronouns on the English side is a little more imbalanced in the improved bitext than in the original (rows 6 7 and 14 15 ), 7 likely due to gender bias in NMT (Stanovsky et al., 2019). This calls for techniques to mitigate such biases (Saun-ders and Byrne, 2020; Stafanovics et al., 2020) for NMT and other downstream tasks. Our previous analysis suggests that selective replacement of divergent references with synthetic translations results in bitext of improved quality , with reduced level of noises and easier word-level mappings between the two languages, when compared to the original WikiMatrix corpus. To better understand how those differences impact downstream tasks, we contrast the improved bitext with the original through a series of extrinsic evaluations for EN-EL and EN-RO languages that rely on parallel texts as training samples (see 5.2). First, we focus on the recent state-of-the-art unsupervised BLI approach of Shi et al. (2021) that relies on word-alignments of extracted bitexts. Second, we follow the recent bitext quality evaluation frameworks adopted by the Shared Task on Parallel Corpus Filtering and Alignment (Koehn et al., 2020) and built neural machine translation systems from scratch and by continued training on a multilingual pre-trained transformer model. Finally, we conduct extensive ablation experiments to test the impact of using synthetic translations without the semantic equivalence condition and contrast with familiar techniques used by prior work (see 5.3). BLI The task of BLI aims to induce a bilingual lexicon consisting of word translations in two languages. We experiment with the recently proposed method of Shi et al. (2021) that combines extracted bitext and unsupervised word alignment to perform fully unsupervised induction based on extracted statistics of aligned word pairs. The induced lexicons are evaluated based on MUSE (Lample et al., 2018) consisting of 45 , 515 and 80 , 815 dictionary entries for EL-EN and EN-RO , respectively. 8 We extract word alignments using m BERT -based Sima-lign 9 (Jalili Sabet et al., 2020) and statistics based on the implementation of Shi et al. (2021). 10 MT We experiment with MT tasks following two approaches: (1) training standard transformer seq2seq models from scratch; (2) continued training for mT5 (Xue et al., 2021), a multilingual pre-trained text-to-text transformer. We evaluate translation quality with BLEU (Papineni et al., 2002) 11 on the official development and test splits of the TED corpus (Qi et al., 2018). 12 For (1) we follow the experimental settings described in 3.2. For (2) we initialize the weights of transformer with mT5-small which consists of 300 M parameters, 13 . We use the simpletransformers implementation. 14 We fine-tune for up to 5 epochs and include the parameter settings in Appendix D. Ablation Settings We compare the NMT models trained on the variants of the synthetic bitext to isolate the impact of replacement criteria and different candidates. 15 For the former, we experiment with the rejuvenation approach of Jiao et al. (2020) that replaces original references with forward translated candidates for the 10% least active original samples measured by NMT probability scores. Moreover, we experiment with forward and backtranslation baselines trained on bitexts that consist solely from targetor source-side candidate sentences (i.e., original references are entirely excluded) and with ablations that consider either forward or backward 8 https://github.com/facebookresearch/MUSE 9 https://github.com/cisnlp/simalign 10 https://github.com/facebookresearch/ bitext-lexind 11 https://github.com/mjpost/sacrebleu 12 Data statistics are found in Appendix E. 13 https://github.com/google-research/ multilingual-t5 14 https://github.com/ThilinaRajapakse/ simpletransformers 15 Results on development sets are in Appendix B. 4758 All Low Medium High PAIR BITEXT Precision Recall F1 OOV rate Precision EL-EN (cid:110) Original 76 . candidates for the proposed semantic equivalence condition. Finally, we consider two alternatives to the semantic equivalence condition based on divergent scores: the ranking condition replaces a candidate if it scores higher than the original (i.e., margin with d = 0 ) and the thresholding condition adds the additional constraint that candidates should rank higher than a threshold to replace the original pair. BLI Table 4 presents results for unsupervised BLI on the MUSE gold-standard dictionaries, for EL-EN and EN-RO . Across languages, the revised bitexts induce better lexicons compared to the original WikiMatrix. Crucially, improvements are reported both in terms of Recallwhich connects to the observation that the revised bitext exhibits higher coverage than the original and in terms of Precision which connects to the noise reduction effect that impacts the extracted word alignments. Additionally, a break-down on the Precision of the induced lexicons binned by the frequency of MUSE source-side entries (i.e., last 3 columns in Table 4) reveals that the improvements come from better induction of lowand medium-frequency words, which we expect are more sensitive to noisy misalignments that result from divergent bitext. Finally, those improvements are reported despite the small increase of the OOV rate in the revised lexicons that results from the decrease in the lexical types covered in it, as mentioned in the analysis (i.e., 4.3). Furthermore, following the advice of Ke-mentchedjhieva et al. (2019) who raise concerns on BLI evaluations based on gold-standard pre-defined dictionaries, we accompany our evaluation with manual verification to confirm that our conclusions are consistent with those of the automatic evaluation. Concretely, we manually check the false positives induced translation pairs from the origi-PAIR ORIGINAL REVISED EL  EN 28 . nal vs. the improved bitext. We found that 65 / 80 are false false positives (due to incompleteness of pre-defined dictionaries) for the improved bitext and 51 / 80 for the original (see Appendix F for the complete list). This confirms that the metric improvements we observe are meaningful and suggests that the improved bitext help learn better mappings between source and target words. MT Table 5 presents translation quality ( BLEU ) on EN  RO and EN  EL tasks for MT training from scratch and Figure 2 shows translation quality of 4759 SELECTIVE DATABITEXT STATISTICS REPLACEMENT TYPES BLEU  O F BVIS . mT5 continued training across epochs. Across tasks and settings, the revised bitext yields better translation quality than the original WikiMatrix data. The consistent improvements we observe across the two settings suggest that the properties of the synthetic translations that replace original samples and bring those improvements are invariant to specific models. Moreover, the magnitude of improvements is larger in the continued training setting compared to training from scratch (e.g.,  +0 . 8 vs.  +1 . 5 , for EN  EL ;  +0 . 2 vs.  +1 . 5 , for RO  EN ). The latter suggests that improvements from using synthetic samples do not only come from the normalization effect (i.e., synthetic samples are easier to model by NMT ) but also connect to the reduced noise in the training samples. This further complements our hypothesis that synthetic translations can improve the quality of imperfect references that should, in principle, yield noisy training signalsand thus impact the resulting qualityof different MT models. Table 6 compares the translation quality ( BLEU ) of NMT systems trained on different synthetic translations. By forcing the semantic equivalence condition when deciding whether a synthetic translation replaces an original, we revise 50% of the latter yielding the best results across directions with significant improvements (i.e, increases do not lie within 1 stdev of the original's bitext performance) of +0 . 81 ( EN  EL , row 9 ) and +1 . 49 ( EL  EN , row 18 ) points over the original bitext. Impact of semantic equivalence condition Table 6 shows that naively disregarding the original references and training only on synthetic translations gives mixed results: training on forward-translated references only (i.e., row 2 ) gives small improvements ( +0 . 36 ) over the model trained on WikiMatrix for EN  EL , while it performs comparably to it for EL  EN (i.e., row 11 ). On the other hand, training on backward data only (i.e., row 12 ) improves BLEU by a small margin ( +0 . 23 ) for MT into EN while it hurts BLEU when translating into EL (i.e., row 3 ). This indicates that the good quality of the synthetic translations cannot be taken for granted and motivates replacing original pairs under conditions that account for semantic controls. The latter is further confirmed by results on the rejuvenation baseline: replacing candidates for the 10% of the most inactive WikiMatrix samples results in small and insignificant increases in BLEU when compared to models trained on original WikiMatrix data (i.e., rows 1 4 and 10 13 ). This indicates that rejuvenation might not be well-suited to lower resource settings than the ones it was originally tested on (Jiao et al., 2020). The rejuvenation technique might be affected by the decreased NMT 4760 quality and calibration in lower resource settings. By contrast, using synthetic translations with semantic control mitigates their impact. Finally, all three semantic control variants based on divergent scores yield bitexts that improve BLEU compared to the original WikiMatrix (i.e., rows 5 8 and 14 18 ). Among them, the margin condition is the most successful, followed by the thresholding variant. The breakdown of training statistics reveals the reason behind their differences: the thresholding condition is a more strict constraint as it only allows synthetic candidates to replace the original pairs if they are predicted as exact equivalents, allowing for fewer revisions of divergent pairs in WikiMatrix. By contrast, the condition based on margin is a contrastive approach that allows for more revisions of the original data (i.e., a candidate might be a more fine-grained divergent of the source). The ranking criterion is the least successful methodthis is expected as the divergence ranker is not trained as a regression model. Impact of bi-directional candidates Considering both forward ( F ) and backward ( B ) translated candidates during selective replacement yields to further improvements ( 0 . 22 0 . 44 points) over bitext induced by the semantic equivalence condition with candidates from a single NMT model (i.e., rows 7 9 and 16 18 ). When forward and backward candidates are considered independently, they replace 34  37% of the original pairs; in contrast, when considered together, they replace 50% of original WikiMatrix pairs. As a result, there is no perfect overlap between the original pairs replaced by the forward vs. backward model, which motivates the use of both to revise more divergences in WikiMatrix. This finding raises the question of whether using synthetic translations from both directions might benefit other scenarios, such as knowledge distillation. This paper explored how synthetic translations can be used to revise bitext, using NMT models trained on the exact same data we seek to revise. Our extensive empirical study surprisingly shows that, even without access to further bilingual data or supervision, this approach improves the quality of the original bitext, especially when synthetic translations are generated in both translation directions and selectively replace the original using a semantic equivalence criterion. Specifically, our intrinsic evaluation showed that synthetic translations are of sufficient quality to improve over the original references, in addition to normalizing the bitext as suggested by prior work and corpus level statistics (Zhou et al., 2020; Xu et al., 2021). Extrinsic evaluations further show that the replaced synthetic translations provide more useful signals for BLI tasks and NMT training in two settings (i.e., training from scratch and continued training). These findings provide a foundation for further exploration of the use of synthetic bitext. First, we focused our empirical study on language pairs and datasets where revising bitexts is the most needed and most likely to be useful: the resources available for these languages are not so large that mined bitext can simply be ignored or filtered with simple heuristics, yet there is enough data to build NMT systems of reasonable quality (i.e.,  600 K segments for EN-RO , and  750 K for EN-EL ). While in principle, selective replacement of divergent references with synthetic translations should port to high-resource settings, where NMT is as good or better than for the languages considered in this work, other techniques are likely needed in low-resource settings where NMT quality is too low to provide reliable candidate translations. Second, having established that the revised bitext improves the quality of the original bitext in isolation, it remains to be seen how to best revise bitexts in more heterogeneous scenarios with diverse sources of parallel or monolingual corpora. Overall, as synthetic data generated by NMT is increasingly used to improve cross-lingual transfer in multilingual NLP, our study motivates taking a closer look at the properties of synthetic samples to better understand how they might impact downstream tasks beyond raw performance metrics. All bitexts are available at: https: //github.com/Elbria/xling-SemDiv-Equivalize . We thank Marjan Ghazvininejad, Luke Zettle-moyer, Sida Wang, Sweta Agrawal, Jordan Boyd-Graber, Pedro Rodriguez, the anonymous reviewers and the CLIP lab at UMD for helpful comments. This material is based upon work supported by the National Science Foundation under Award No. 1750695 . Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
abstain objective result abstain abstain other abstain abstain abstain abstain method result method method result method objective abstain result abstain other other other other other other other objective other other other other abstain objective method method method method method abstain abstain method method method abstain abstain abstain method method abstain other method abstain abstain method method method method abstain method method other method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method method method method method method method method other abstain method abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result result abstain abstain other method abstain objective other other other other