Translated texts have been used for malicious purposes, i.e., plagiarism or fake reviews. Existing detectors have been built around a specific translator (e.g., Google) but fail to detect a translated text from a strange translator. If we use the same translator, the translated text is similar to its round-trip translation, which is when text is translated into another language and translated back into the original language. However, a round-trip translated text is significantly different from the original text or a translated text using a strange translator. Hence, we propose a detector using text similarity with round-trip translation (TSRT). TSRT achieves 86 . 9% accuracy in detecting a translated text from a strange translator. It outperforms existing detectors ( 77 . 9% ) and human recognition ( 53 . 3% ). A reader may misunderstand the original meaning of a translated text 1 . For example, Facebook translated  good morning  into  attack them , leading to an arrest 2 . Adversaries can use a translator for malicious tasks such as round-trip translation used in plagiarism (Jones and Sheridan, 2015) to avoid human recognition or in adversarial text (Iyyer et al., 2018) to fool AI. Existing work has investigated the detection of translated texts in various approaches. The parse tree approach (Chae and Nenkova, 2009; Li et al., 2015) exploits text structure. The N gram approach (Aharoni et al., 2014; Arase and Zhou, 2013) estimates text fluency. The text complexity approach uses complex words (Nguyen-1 When we mention a translated text, translation, translator, and Google, all are related to machine translation systems 2 www.theguardian.com/technology/2017/oct/24/facebookpalestine-israel-translates-good-morning-attack-them-arrest Son and Echizen, 2017) and phrases (Nguyen-Son et al., 2017). The text coherence approach is based on matching similar words on a paragraph level (Nguyen-Son et al., 2018, 2019b). A three-layer CNN (Riley et al., 2020) is trained on either one-way or round-trip translated texts. Our previous work (Nguyen-Son et al., 2019a) combined round-trip translation with BLEU scores. All these approaches fail to detect a text translated by another translator or from a different language. Motivation The first translation round induces a low similarity between the translated and original texts, whereas the extent of similarity increases in later rounds (Vanmassenhove et al., 2019). Let us consider an example in Fig. 1. We randomly selected an English text t from an English-Russian pair 3 ; the Russian text was translated into English by Google, called t  ( Go,RU  EN ) . We measured the similarity between a text and its round-trip translation using the minimum edit distance ( MED ) (Levenshtein, 1966). The translated text t  is the result of using the translator once, and the similarity between t  and its round-trip translation t  ( Go,RU  EN  RU ) is high ( MED = 1) . Otherwise, the similarity between the original text t with t ( Go,RU  EN  RU ) is low ( MED = 5) . Based on the difference in similarity, we can distinguish the original from the translated text. In reality, a translator's source language is often unknown. The similarity decreases when using another language. For example, the similarity between t  ( Go,RU  EN ) translated from Russian and its round-trip translation t  ( Go,RU  EN  DE  EN ) from German is low ( MED = 6) . It is close to the similarity in the original pair 3 This pair belongs to a Commentary News corpus (Barrault et al., 2019)  (,) : The actions of the chief banker, their every word or hint suddenly take on tremendous significance. { t, t ( Go,EN  DE  EN ) } ( MED = 4) . A change in a translator induces a similar phenomenon. We thus detected the translator and the language before detecting the translated text. Contributions We propose a novel translation detector that utilizes text similarity with round-trip translation (named TSRT). This detector can be used as a warning to prevent the risk of translated texts in a certain region where people are familiar with few languages and translators. First, we create round-trip translations from multiple configuration translator and language tuples. Second, we use each tuple's round-trip translations to train individual subclassifiers. Then, we use the tuple with the highest similarity between a suspicious text and its round-trip translation to choose a suitable subclassifier. Finally, we use the subclassifer to determine if the text is an original or translated text. Experiments demonstrate that TSRT efficiently detects different kinds of translated texts (round-trip and one-way) when the translation translator and language is changed. Training Phase First, we collect original texts T i and translated texts T  i , which are translated with a configuration tuple  i = { language  i , translator  i } (see Fig. 2). Second, we generate round-trip translations T  i i and T   i i for T i and T  i , respectively. Finally, T i and T  i are combined with T  i i and T   i i to train a subclassifier   i by fine-tuning the BERT model (Devlin et al., 2019). We repeat the procedure with other subclassifiers. In Fig. 1, t , t  , t ( Go,RU ) , and t  ( Go,RU ) belong to T , T  , T ( Go,RU ) , and T  ( Go,RU ) , respectively, with  = ( Go, RU ) . Testing Phase For a suspicious text s , we aim to determine if s is an original or a translated text. First, we generate round-trip translated texts s  i with all configuration tuples in the training phase. Next, we calculate the similarity   i between t and all s  i using the minimum edit distance ( MED ). Finally, we process s with the subclassifier associated with the best similarity  b corresponding to the lowest MED . In the case of t  in Fig. 1, two round-trip translations t  ( Go,RU ) and t  ( Go,DE ) are generated with respect to  ( Go,RU ) = 1 and  ( Go,DE ) = 6 . The subclassifier  ( Go,RU ) associated with the lower MED is chosen for classifying t  . Round-trip translation detection : We collected 11 , 748 distinct movie reviews from the Sentiment Treebank (Socher et al., 2013) (19.1 words/review). We chose 9 , 000 / 1 , 000 reviews for training/developing and used the remaining pairs for testing. This ratio is reused in further experiments. We used the original reviews to generate round-trip translations by using configuration tuples of two translators and three languages (Table 1). In addition to Google, we chose Fairseq 4 (Ng et al., 2019), the winner in the WMT'19 shared task. We compare TSRT 5 with 4 Fairseq is only supported for Russian and German, so we cannot use it for Japanese. existing methods using the accuracy metric (accu-racy and F -score are equivalent in this balanced corpus). BERT and TSRT have the same optimized hyperparameters 6 . The first four methods do not work well with this parallel corpus. The round-trip translation (Nguyen-Son et al., 2019a) based on BLEU and BERT (Devlin et al., 2019) improves by approximately 10% . TSRT provides the highest performance, as it captures round-trip information using deep learning. We analyzed the text lengths of the top three detectors on the whole (Go,RU) test set (Fig. 3). BERT surpasses round trips in only short length ranges, while TSRT outperforms the others in all ranges. Human recognition : We selected 100 random reviews from the test set for human recognition 7 . We sent them to 14 raters (6 were native English detection 6 We optimize hyperparameters with recommended values from BERT (maximum size of 128 , batch size of 32 , learning rate of 2 e 5 , and epoch of 3 ). Since the development accuracy is equivalent to the test accuracy, we use the test accuracy for further experiments. 7 The survey is available at https://forms.gle/ L8EkZxXuEH9Co3UB7 . speakers), who decided whether each review was an original or a translated text. The average accuracy was 53 . 3% ( 55 . 0% for the native speakers and 52 . 0% for the nonnative speakers), which was close to random. The low Fleiss'  = 0 . 13 implied slight agreement in the native speakers' ratings. For nonnative speakers,  was even lower (  =  0 . 07 ). This indicates that the translated texts were indistinguishable by humans. One-way translation detection : We collected parallel sentences from the Commentary News corpus (Barrault et al., 2019). We randomly selected 11 , 748 pairs with 21 . 9 words on average per sentence (same as the movie reviews). We experimented with two languages (Russian and German) and two translators (Google and Fairseq) (see Fig. 4). Since one-way translation is more challenging to detect, the accuracy is decreased for all methods. In the top three detectors, while BERT and round-trip translation yield unstable results, TSRT remains consistent. Comparison : Humans are familiar with limited languages and translators. Normally, they use their mother tongue and English (international language) and translate by choosing a popular translator such as Google or an open-source translator such as Fairseq. Table 2 presents the translation detection with translator and language changes. While the existing methods are trained with (Go,DE) or (Fa,RU), TSRT is trained on (Go,DE)+(Go,RU) or (Fa,RU)+(Go,RU), respectively. We tested all of them in (Go,RU). Our results showed that the existing methods were significantly downgraded in terms of accuracy, but TSRT remained stable. Ablation Studies : We trained TSRT on various configuration tuples and tested it on (Go,RU) (Ta-ble 3). Training TSRT on the combination with the correct configuration tuple (Go,RU) boosts the performance. Configuration identification : We identify the translator and language on round-trip translation detection while the one-way approach obtains similar results. For translator change (Table 4's second col-umn), we used (Go,RU) and (Fa,RU). For the language change (the third column), we used (Go,RU) and (Go,DE). All were tested on (Go,RU). We used BERT as the identification baseline. We replaced MED with BLEU in TSRT. All the metric-based approaches outperformed the baseline. The trans-Training data Acc(Red.) (Go,RU) 90.2(-00.0) (Fa,RU) 70.2(-20.0) (Fa,RU) 70.2(-20.0) (Go,DE) 73.4(-16.8) (Fa,DE) 66.6(-23.6) (Go,RU)+(Fa,RU) 86.9(-03.3) (Go,RU)+(Go,DE) 86.6(-03.6) (Go,RU)+(Fa,RU)+(Go,DE)+(Fa,DE) 81.5(-08.7) Table 3: TSRT's results with individuals and combinations of configuration tuples of translators and languages. lator detection outperformed language detection. While a specific translator often uses the same architecture for all languages, various translators have different architectures. Therefore, a translator change was more apparent than a language change. MED (designed for structure similarity) was better than BLEU (designed for corpus levels). This paper proposed a one-way and round-trip translation detection mechanism using text similarity with round-trip translation (TSRT), which is robust to language and translator changes. First, we trained subclassifiers on specific lan-Method Translator Language BERT 63 . guages/translators using round-trip translation. Then, we identified the language and translator using the highest similarity between the suspicious and round-trip translation texts. Finally, we chose the corresponding subclassifier for translation detection. The evaluation results show that TSRT outperforms other methods, with an accuracy of up to 90 . 2% . Moreover, TSRT could also identify the original translator and translation language with 93 . 3% and 85 . 6% of accuracy, respectively. In future work, we will exploit saturation after repeatedly using the same AI system to detect other artificial texts such as fake COVID-19 news. We would like to thank you very much for the anonymous reviewers to provide useful comments.
abstain abstain method abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective method method method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain method method other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain method method abstain other abstain abstain abstain abstain objective objective abstain method method abstain abstain abstain abstain abstain method other