We show that margin-based bitext mining in a multilingual sentence space can be successfully scaled to operate on monolingual corpora of billions of sentences. We use 32 Common Crawl snapshots (Wenzek et al., 2019), totalling 71 billion unique sentences. Using one unified approach for 90 languages, we were able to mine 10.8 billion parallel sentences, out of which only 2.9 billions are aligned with English. We illustrate the capability of our scalable mining system to create high quality training sets from one language to any other by training hundreds of different machine translation models and evaluating them on the many-to-many TED benchmark. Further, we evaluate on competitive translation benchmarks such as WMT and WAT. Using only mined bitext, we set a new state of the art for a single system on the WMT'19 test set for English-German/Russian/Chinese. In particular, our English/German and English/Russian systems outperform the best single ones by over 4 BLEU points and are on par with best WMT'19 systems, which train on the WMT training data and augment it with backtrans-lation. We also achieve excellent results for distant languages pairs like Russian/Japanese, outperforming the best submission at the 2020 WAT workshop. All of the mined bitext will be freely available. Parallel data, i.e. sentences in two languages which are mutual translations, are a crucial resource for many multilingual natural language processing tasks. Traditionally, high quality parallel texts are obtained from the publications of international organizations like the the United Nations (Ziemski et al., 2016) or the European Parliament (Koehn, 2005). These are professional human translations, but they are in a more formal language and tend to be limited to political topics. Another direction is to rely on volunteers to provide translations for public texts, such as the TED corpus (Qi et al., 2018), news commentary (Tiedemann, 2012) or OpenSubtitles (Lison and Tiedemann, 2016), but this approach lacks scalability. There is also a large body of works which aims in mining bitexts by comparing huge collections of monolingual data. Our aim is to mine at massive scale, both in number of possible languages and in quantity of mined parallel sentences. Most existing large scale bitext mining techniques use a hierarchical approach. First, a subset of texts that may contain parallel sentences are selected at the document level. Subsequently, sentences within these aligned documents are compared to identify parallel ones. This local mining is potentially fast since only a few thousand sentences need to be compared for each document pair. However, sentences not present in these pre-selected documents cannot be aligned, which vastly limits the quantity of mineable bitext. A first system to globally compare all sentences in monolingual collections for many language pairs was presented in Schwenk et al. (2019), but was limited to only Wikipedia. In this paper, we show that this type of global mining scales to extremely huge corpora: 71 billion sentences, about 120x larger than the work of Schwenk et al. (2019). Our contributions are:  development of a new highly efficient and parallelized processing pipeline to confront the substantial computational challenge;  unprecedented size: 10.8 billion mined parallel sentences in 90 different languages;  all these resources are freely available;  we demonstrate the quality of our mined data on a variety of machine translation benchmarks, such as TED, WMT, and WAT, achieving highly competitive results. Much previous work has explored the automatic creation of parallel data from monolingual resources. In this section, we detail various approaches and illustrate the differences of our algorithmic approach and the scale of our mining. Mining Methodology At the start, various approaches used alignment on information beyond text itself, such as with document metadata (Resnik, 1999; Resnik and Smith, 2003). Later, work aligned based on text with techniques such as Jaccard similarity (Etchegoyhen and Azpeitia, 2016; Azpeitia et al., 2017, 2018), crosslingual document retrieval (Utiyama and Isahara, 2003; Munteanu and Marcu, 2005), language models (Buck and Koehn, 2016), translation (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018), or bag-of-words (Buck and Koehn, 2016). In contrast, we use massively multilingual sentence embeddings trained on almost 100 languages, and then conduct margin-based mining in the multilingual embedding space (Schwenk, 2018; Artetxe and Schwenk, 2018a,b; Kvapilkova et al., 2020). Previous work such as Espana-Bonet et al. (2017); Hassan et al. (2018); Guo et al. (2018); Yang et al. (2019) used bilingual embeddings, which is not scalable for mining many different languages. Compared to work such as Schwenk (2018), we drastically increase the scale of our mining and produce two orders of magnitude more data  this is possible by the increased efficiency and scala-bility of our improved mining methods. A few mining approaches were applied to large quantities of language pairs. For example, the ParaCrawl project 1 mined data for all European languages. Bitextor (Espl`a-Gomis and Forcada, 2010) was applied to many languages, but took an approach that required identifying parallel documents first and then extracting aligned sentences. This is similar to the ccAligned project (El-Kishky et al., 2020). In contrast to these, we mine much larger quantities of parallel data due to the global margin-based mining approach that we take. Data used to Mine Many previous methods for data mining focused on Wikipedia. Otero and Lopez (2010) and Patry and Langlais (2011), for instance, aligned entire parallel documents. For example, Adafre and de Rijke (2006) and Mohammadi and GhasemAghaee (2010) used machine translation systems to compare Dutch and Persian Wikipedias to English, to identify aligned sentences. Various other worked used similarities in mentioned entities to align text, such as Gottschalk and Demidova (2017) and Tsai and Roth (2016). Work such as Smith et al. (2010); Tufis et al. (2013); Aghaebrahimian (2018) used Wikipedia to mine parallel sentences, but focused on fewer languages, often high resource. In contrast, our system mines not in Wikipedia but in CommonCrawl, a much larger source of data  and is applied to a much larger quantity of languages. Work has extended mining beyond Wikipedia. For example, ParaCrawl 1 has been heavily used (e.g. in WMT), which is based on several noisy multilingual crawls (Koehn et al., 2018, 2019). El-Kishky et al. (2019) focused on mining documents in Common Crawl rather than sentences. Our work continues this line of scalable mining on the web, but pushes to large-scale mining to produce billions of aligned sentences. We leverage massively multilingual sentence embeddings and a margin-based criterion to mine parallel sentences. The core idea is to learn a multilingual sentence embedding, or an embedding space in which semantically similar sentences are close, independent of the language they are written in. This means that distance in the embedding space can be used to determine if two sentences are mutual translations or not. We use the open source LASER (Artetxe and Schwenk, 2018b) embeddings as they cover over 90 different languages. 2 Another recent multilingual sentence embedding is LaBSE (Feng et al., 2020). Given two sentence embeddings, how can we decide if they are mutual translations? Using an abso-lute threshold on the cosine distance was shown to achieve competitive results (Schwenk, 2018), but is globally inconsistent (Guo et al., 2018). Therefore, we use margin-based mining (Artetxe and Schwenk, 2018a). The margin M( x, y ) between two sentence embeddings x and y is defined as the ratio between the cosine distance between x and y , and the average cosine similarity of its nearest 1 http://www.paracrawl.eu/ 2 https://github.com/facebookresearch/ LASER text 1 text N text FAISS embed 1 embed N embed FAISS index 1 index N trained index dedup. where NN k ( x ) denotes the k unique nearest neighbors of x in the other language, and analogously for NN k ( y ) . We set k to 16. Artetxe and Schwenk (2018a) describe the max-strategy as one of the best performing ones: the margin is calculated in both directions for all sentences in languages L 1 and L 2 . Then, the union of forward and backward candidates is built, candidates are sorted, and pairs with source or target sentences which were already used are omitted. Finally, a threshold is applied to the margin score to decide if two sentences are mutual translations. This strategy was motivated by evaluation on the BUCC corpus (Zweigenbaum et al., 2018), where the reference alignments are known to be strictly 1:1. Our aim is to mine at the billion-scale, and at this size, the probability of finding multiple per-fect translations increases. Therefore, we take the union of the best forward and backward alignments, excluding duplicate bitexts. In this work, we mine billions of parallel sentences from the Web by using the data released in Common Crawl. 3 We preprocess the raw text following the pipeline used to create the CCNet dataset (Wen-zek et al., 2019). We use 32 crawls spanning the period from December 2017 to February 2020. Our CCNet corpus is about 120 times larger than Wikipedia: 71 billion compared to 595 million unique sentences (Schwenk et al., 2019). The largest corpora are English (14.3 billion), then German, French, and Spanish (more than 5.2 billion 3 https://commoncrawl.org/ sentences). For 17 different languages, CCNet contains over one billion unique sentences (see Table 1). This requires a carefully designed mining approach in order to tackle the substantially computational complexity and successfully scale. We developed a multi-step mining procedure that is structured into three distinct tasks: 1. text extraction and processing including sentence splitting and language identification; 2. creation of a FAISS index for each language; 3. mining parallel data for each language pair using the sentence embeddings and indices. Each step is parallelized as much as possible by splitting the data into several blocks. Text extraction. The first task, text extraction and processing, consists of three steps: 1) extract text from the JSON data of CCNet and split the paragraphs into sentences; 2) mark duplicate sentences; and 3) perform language identification (LID) and exclude sentences not in the expected language. Each of these three steps processes blocks in parallel. At the final step, we merge all the block-wise deduplicated sentences and create one set of globally unique sentences for each language. We used a Python library 4 to detect sentence boundaries. If specific rules for a language are not available, we fall-back to a linguistically similar languages, e.g. using Spanish rules for Gallican, and default to English otherwise. Most of the Asian languages are handled by regular expressions. We exclude sentences with more than 500 characters. A major challenge of web data is noise. This particularly manifests in text that has the wrong language label. As noise in this stage will affect our mining process, we perform strict filtering using two LID systems on each sentence, fastText (Grave et al., 4 https://pypi.org/project/ sentence-splitter/ text 1FR embed 1FR index DE embed 1DE text 1DE index FR margin based mining text NFR embed NFR embed MDE text MDE       Idx NDND 1 Idx 1 Idx 1 D 1 DM Idx M Figure 2: Parallelized processing flow to mine parallel sentences . Left : forward distances; Right : backward distances. Middle : both distances are combined according to Equation 3.1 and the extracted bitext. 2018) and LangID (Lui and Baldwin, 2011), and discard the data if the two disagree or have low confidence. This processing yields a corpus of N i unique sentences for each language L i . These texts are the basis for index creation and mining (see column size in Table 1). Index creation. We follow Schwenk et al. (2019) and use the highly optimized FAISS library (John-son et al., 2017) 5 to create compact indices of the sentence embedding. LASER's sentence representations are 1024-dimensional, which means that the embeddings of all sentences would require 71  10 9  1024  4  290 TB to store. To practically handle this scale, we use an aggressive vector compression based on a 64-bit product-quantizer (Jegou et al., 2011), and 64k cells to partition the search space. This corresponds to the index type OPQ64,IVF65536,PQ64 in FAISS. Exhaustive search in huge indices is tractable only if performed on GPU. FAISS supports sharding of a single index on multiple GPUs this is most efficient if the GPUs are in the same machine and communicate very quickly. Our index type, using eight GPUs with 32GB of memory each, allows us to handle an index size of 3.2 billion sentences. Seven languages exceed this threshold, so we proceed to create multiple indices (English, German, French, Spanish, Russian, Chinese, and Japanese). The processing pipeline to train and create the indices is summarized in Figure 1. We train an index on 40 million sampled sentences of the whole corpus. Once the index is trained, the data in each block is independently added to this common index, which can be performed in parallel. The individual indices are subsequently merged into one index per language. The largest indices have a size of around 210GB, making 90 indices total almost 4TB. Mining. After indices for all languages are created, we begin the mining process for each language pair. To illustrate the process, we describe it concretely with the example of two high resource languages, Italian and Portuguese, which have 2.5 billion sentences each. This requires 2 . 5 10 9  2 . 5 10 9 = 6 . 25 10 18 distance calculations. Performing this on a single node with 8 GPUs would require more than 6 months. Instead, we tackle this computational challenge by decoupling the distance calculations of the forward and backward direction and the margin calculation, and processing these in parallel. This processing pipeline is illustrated in Figure 2. For all language pairs, we compute both forward and backward distances, even for languages with multiple indices, such as English, French and German. All available alignments for one pair are merged, excluding duplicate sentence pairs. In the current CCMatrix corpus, we have mined data for a diverse set of 90 languages, covering a variety of different language families and scripts (full list in the Appendix). As the mining process is computationally intensive, we focus on many commonly spoken languages to support existing translation systems, as well as mine several mid to low resource languages to provide parallel data for directions with limited to no public training data. We organized all languages into twelve groups which mostly correspond to well established linguistic language families, but we have also performed some geographic groupings, in particular for small language families or isolated languages. In addition, we have identified major languages in each group and use them as bridge languages . We mine for all bitexts among these 27 bridge languages. The motivation for this bridge language approach is to connect the languages of the various groups, but sill avoid mining the full matrix. Additional details are given in the Appendix. The margin threshold used to mine parallel sentences impacts the quality of mined bitexts. A higher threshold leads to better aligned sentences, and thus higher quality bitexts, but also to smaller datasets. Thus, there is a trade-off between size and quality. Exploratory experiments based on training different NMT models showed that a threshold around 1 . 06 gave good results. We display a representative example on Hungarian-Danish in Fig. 3. 1.050 1.055 1.060 1.065 1.070 Margin threshold 10.210.410.610.811.011.211.4 BLEU s c o r e hu-da da-hu Figure 3: BLEU scores on Hu-Da TED dev set for various margin threshold values. We mine a total of 10.8 billion parallel sentences out of which only 2913 million are aligned with English, considering a margin threshold of 1.06 for all language pairs. Table 1 gives a summary for the 54 largest languages. The full list of supported languages is given in the Appendix. In contrast to other works, such as the European ParaCrawl project, 1 we do not limit to alignments with English, but provide alignments for 1197 language pairs. This yielded unprecedented amounts of bitexts of non-English language pairs, for example 286M for Spanish-French, 24M for Arabic-French and Spanish-Chinese, and a total of 326M bitexts with Norwegian (which is not present in Europarl). Further, a variety of different Asian languages were mined, producing 7.2M pairs for Japanese-Korean, 7.8M for Indonesian-Malay, and 1.3M for Bengali-Hindi. To the best of our knowledge, this makes CCMatrix the largest collection of high-quality mined parallel texts, with coverage over a wide variety of languages. Providing multiple aligned bitexts for many languages also opens the possibility of improved training of massively multilingual NMT systems (Fan et al., 2020), as this substantially increases the amount of bitexts for low resource languages. As an example, Nepali has less than 1M bitexts with English, but 17M bitexts with multiple languages (see last column of Table 1). Table 1 gives the amount of mined bitexts for various language pairs. The general tendency is of course that mining in large monolingual corpora leads to larger extracted bitexts. This is however not systematically true. Let us consider for example Danish, a Germanic language. When aligned with Norwegian, also a Germanic language, we obtain 17.7M bitexts. The pair Danish-Italian, however, has only 14.7M bitexts although Italian has almost six times more sentences than Norwegian. One one hand, a possible explanation could be that LASER alignments are more reliable for languages which are very similar, i.e. in the same language fam-ily. On the other hand, it may also be that people which live in nearby countries have similar interests which increases the chance to find translations on the Web. Additional analysis and examples are provided in the Appendix. To assess our mined bitext, we train NMT systems only on our mined data and evaluate on several public benchmarks. We do not use any of the training data provided with these corpora, so do not use any available human translated data, and have no guarantee our bitext covers the same domain as the test sets. Nevertheless, we show on the many to many TED corpus that our mined data produces high quality translation systems, even through distant language pairs not aligned through English and low resource languages. Finally, we demonstrate that models trained on CCMatrix can surpass state of the art systems in WMT'19 and WAT'20. We examine the quality of our mined bitext across a diverse set of languages, focusing on performance of bitext pairs not aligned through English. Following Gottschalk and Demidova (2017), we evaluate on the test sets of the TED corpus (Qi et al., 2018), which contains parallel TED talk transcripts in 58 languages. This corpus is tokenized, so we deto-kenize using Moses, with the exception of pairs ISON a m e S i ze k o v i z h a f d a d e e n i s n l no s v ca e s f r g l it p t r o b e bg c s h r p l r u s k s l s r uk e l e t fi hu l t l v e u s q t r a r h e f a s w bn h i m r n e s i u r t a i d m g m s m l tl T o t a l j a Ja p a n e s e 3 929 7 . involving Chinese, Japanese and Korean as it creates artifacts. We consider 29 different languages, resulting in 778 NMT systems to train. We apply the same preprocessing and training procedure for all language pairs. We train a SentencePiece Model (Kudo and Richardson, 2018) with a vocabulary of size 50 k. The bitext were not filtered to remove sentences which may appear in the TED dev or test sets. Also, we did not try to optimize the architecture of the NMT models to to size of the bitexts for each language pair. Instead, for all the pairs, we use the same architecture, a Transformer model with six layers for both the encoder and decoder. We use a dimension of 512 and 4096 for the feed-forward. We train each model for 50 epochs with an initial learning rate of 0 . 001 . We keep the model with the best BLEU on the TED validation set. In Table 2, we report tokenized BLEU on the test sets. When translating into Chinese, we scored with sacrebleu -tok zh , and Kytea 6 was used to tokenize Japanese, respectively. The average BLEU over all pairs is 18 . 8 and 33 . 0 for pairs with English. There are 86 pairs out of 778 with BLEU above 30 , compared to 10 out of 1620 language pairs for WikiMatrix. The best WikiMatrix pair reached 37 . 3 BLEU (for Brazilian Portuguese to English), while here 25 pairs are over 37 . 3 , the best pair reaching 51 . 2 BLEU (Norwegian to English). These results show the quality of the mined bitexts and suggest that our mining strategy is robust to the noise and domain differences existing in large corpora like Common Crawl. However, since we did not optimize the NMT systems for each language pair, these BLEU score should not be considered as the best possible ones based on the CCMatrix bitexts. In particular, we anticipate that better results can be obtained when using models with more parameters for the high-resource language pairs. Further, our mined data provides a starting point for those interested in training translation systems directly between languages that currently have no available bitext training data. In particular CCMatrix bitexts have been used to train a massively multilingual NMT systems for 100  100 languages (Fan et al., 2020). Next, we focus on arguably the most competitive translation benchmark, the WMT news translation task, to compare our mined data to the best existing systems. We only consider the high resource directions, as they constitute the largest challenge  existing systems perform strongly, and previous work incorporating mined data from Paracrawl (Ott et al., 2018) only found marginal gains. We follow Ng et al. (2019) and trained systems on en-de, en-ru, en-zh, and de-fr. We used the Transformer Big architecture with FFN size 8192, System de-en en-de en-ru ru-en zh-en en-zh de-fr fr-de Single systems NT'18 WMT bitext 46.2 45.9 33.5 33.4 25.8 39.2 -NT'18 CCMatrix 49.9 50.3 35.7 36.9 30.2 40.8 -NT'19 WMT bitext 41.0 40.4 31.4 38.1 --NT'19 CCMatrix 43.3 44.5 35.5 41.8 34.8 35.6 37.9 33.5 NT'20 WMT bitext 40.3 31.9 24.0 35.5 --NT'20 CCMatrix 39.2 35.1 25.5 37.1 35.0 38.8 33.8 33.8 Ensembles + BT + Reranking NT'19 best 42.8 44.9 36.3 40.2 39.9 44.6 37.3 35.0 Table 3: BLEU scores on the Newstest'18, Newstest'19 and Newstest'20 test sets . embedding size 2048, with 9 encoder/decoder layers, with LayerDrop (Fan et al., 2019). We trained for 400k updates on 8 GPUs. Given the large amounts of mined bitext (see Table 1), we train only on data with a margin threshold at least 1.07, and perform some additional filtering, resulting in 146M for en-de, 78M for en-ru, 82M for de-fr and 31M for en-zh. For each direction, we learn joint source-target BPE (Sennrich et al., 2016) and share input/output embeddings. We tune training parameters on WMT'12-13 when available and on the WMT'19 dev set for de-fr. In Table 3 we demonstrate that the performance of a single model trained on mined data is better than the performance of the best published single models trained on WMT bitext, this can be seen as a clear indicator of the quality of the mined data. Because CCMatrix data is mined from the Web, we want to make sure there is no significant leakage of the test sets that might be available online into the training data. While there are no exact matches of test and train samples, partial overlap is still possible. Following Radford et al. (2019) and Shoeybi et al. (2019) in Table 4 we report the percentage of 8-gram BPE tokens from the test data that are also found in CCMatrix training data. Finally, in Table 3 we also report performance on Newstest'20 tests sets that were not available at the time of mining the data. We further investigate the impact of training on a combination of human translated and mined data. We examine En-De and include the WMT'19 training data. We found that this system outperforms the system trained on CCMatrix data only on average by only 0.6 BLEU, achieving BLEU score 50.9 on newstest2018 and 45.1 on newstest2019. Finally, we examine the quality of our mined data on low resource, distant language pairs. We focus on Russian-Japanese, a language direction in the 2020 Workshop on Asian Translation (WAT) (Nakazawa et al., 2020).The organizers provide a tiny amount of parallel data from the Global Voices domain for training (12k sentences), and a development (486 sentences) and test set (600 sentences) from the News Commentary domain, respectively. 7 We trained an NMT system on CCMatrix Japanese-Russian mined data only, without using other resources or texts aligned with English. We applied a threshold of 1.06 on the margin which yielded 9.5 million parallel sentences. We filtered the mined bitexts to exclude all sentences which 7 https://github.com/aizhanti/JaRuNC System ja-ru ru-ja CCMatrix dev 13.68 20.38 CCMatrix test 14.77 19.60 WAT'20 test best 14.36 18.48 Table 5: BLEU scores on WAT'20 . appear in the WAT dev or test set. We use the same NMT architecture as in Section 5.1. We report tokenized BLEU in Table 5. When translating from Russian into Japanese, to-kenization was performed with Kytea and then scored with multi-bleu.perl . We outperform the best performing system at WAT'20, 8 in particular when translating into Japanese. On one hand, the participants in WAT were constrained to only use the provided resources. But on the other hand, Russian/English and Japanese/English were included and participants were encouraged to train multilingual models, and use techniques like monolingual pre-training or back-translation. Therefore, our results are not directly comparable, but remain a positive indicator of the quality of our mined bitexts. We show that margin-based mining in a joint multilingual sentence embedding space can be scaled to monolingual texts of more than 71 billion unique sentences in 90 languages, including several low resource languages. This procedure yields 10.8 billion parallel sentences, out of which only 2.9 billions are aligned with English. We performed an extensive evaluation of the quality of the mined bitexts by training NMT systems for many language pairs. Training only on mined data, we outperform the best single NMT systems at WMT'19 for translations between German, Russian, and Chinese with English, as well as between German and French. We also achieve state-of-the-art BLEU scores for translation between Russian and Japanese at WAT'20. All mined data is freely available. 9 We hope this will enable widespread research on multilingual NMT, particularly on languages where training data is not currently available. 8 See results at http://lotus.kuee.kyoto-u.ac. jp/WAT/evaluation/index.html 9 https://github.com/facebookresearch/ LASER/tree/master/tasks/CCMatrix References Sadaf Abdul-Rauf and Holger Schwenk.
result method method method method objective objective result abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain result objective other abstain other other abstain other method other other other other objective other other other other other objective other other other method method abstain abstain method abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain method method method method method other abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method method abstain method abstain abstain other abstain abstain abstain method abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain method abstain method method method method method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain method method method abstain abstain method method abstain method method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method other abstain method abstain abstain abstain abstain abstain abstain result abstain method result result abstain abstain other other