We propose a novel data-augmentation technique for neural machine translation based on ROTk ciphertexts. ROTk is a simple letter substitution cipher that replaces a letter in the plaintext with the k th letter after it in the alphabet. We first generate multiple ROTk ciphertexts using different values of k for the plaintext which is the source side of the parallel data. We then leverage this enciphered training data along with the original parallel data via multi-source training to improve neural machine translation. Our method, CipherDAug , uses a co-regularization-inspired training procedure, requires no external data sources other than the original training data, and uses a standard Transformer to outperform strong data augmentation techniques on several datasets by a significant margin. This technique combines easily with existing approaches to data augmentation, and yields particularly strong results in low-resource settings. 1 1 Introduction One naturally wonders if the problem of translation could conceivably be treated as a problem in cryptography. [...] frequencies of letters, letter combinations, [...] etc., [...] are to some significant degree independent of the language used (Weaver, 1949) Indeed, to a system which treats inputs as atomic identifiers, the alphabet behind these identifiers is irrelevant. Distributional properties are of sole importance, and changes in the underlying encoding should be transparent provided these properties are preserved. In light of this, a bijective cipher such as ROTk (Figure 1) is in effect invisible to modern NLP techniques: distributional features are invariant under such a cipher, guaranteeing that the meaning of an enciphered text is the same as the un-enciphered text, given the key. This work exploits this fact to develop a novel approach to data 1 Our code is available at https://github.com/ protonish/cipherdaug-nmt PLAIN abcdefghijklmnopqrstuvwxyz ROT1 bcdefghijklmnopqrstuvwxyza ROT2 cdefghijklmnopqrstuvwxyzab ROT3 defghijklmnopqrstuvwxyzabc SRC : es ist diese pyramide. Data augmentation is a simple regularization-inspired technique to improve generalization in neural machine translation (NMT) models. These models (Bahdanau et al., 2015; Vaswani et al., 2017) learn powerful representational spaces (Ra-ganato and Tiedemann, 2018; Voita et al., 2019; Kudugunta et al., 2019) which scale to large numbers of languages and massive datasets (Aharoni et al., 2019). However, in the absence of data augmentation, their complexity makes them susceptible to memorization and poor generalization. Data augmentation for NMT requires producing new, high-quality parallel training data. This is not trivial as slight modifications to a sequence can have drastic syntactic or semantic effects, and changes to a source sentence generally require corresponding changes to its translation. Existing techniques suffer various limitations: back-translation (Sennrich et al., 2016b; Edunov et al., 2018; Xia 201 et al., 2019a; Nguyen et al., 2019) can yield semantically poor results due to its use of trained models that are susceptible to errors (Edunov et al., 2018). Word replacement approaches (Gao et al., 2019; Liu et al., 2021; Takase and Kiyono, 2021; Belinkov and Bisk, 2018; Sennrich et al., 2016a; Guo et al., 2020a; Wu et al., 2021a) may ignore context cues or fracture alignments between sequences. This paper overcomes these limitations by exploiting the invariance of distributional features under ROTk ciphers. We contribute a novel data augmentation technique which creates enciphered copies of the source side of a parallel dataset. We then leverage this enciphered training data along with the original parallel data via multi-source training to improve neural machine translation. We also provide a co-regularization-inspired training procedure which exploits this enciphered data to outperform existing strong NMT data augmentation techniques across a wide range of experiments and analyses. Our technique can be flexibly combined with existing augmentation techniques, and does not rely on any external data. A ROTk cipher (Figure 1) produces a ciphertext by replacing each letter of its input ( plaintext ) with the k th letter after it in the alphabet. Past work (Dou and Knight, 2012; Dou et al., 2014) has explicitly used decipherment techniques (Kambhatla et al., 2018) to improve machine translation. We emphasize that decipherment itself is not the purpose of the present work: rather, we use ciphers simply to re-encode data while preserving its meaning. This is possible because ROTk is a 1:1 cipher where each ciphertext symbol corresponds to a unique plaintext symbol; this means it will preserve distributional features from the plaintext. This makes ROTk cryptographically weak, but suitable for use in data augmentation. Concretely, given a set of n training samples D = { ( x i , y i ) } ni =1 and a set of keys K , we use Algorithm 1 to generate | K | n new samples; giving ( | K | + 1) n samples when added to the training set. The ciphertexts produced by Algorithm 1 are guaranteed to be lexically diverse, not only from the plaintext but also from one another. Given this fact, we can naively regard each D k as a different language and formulate a multi-Algorithm 1 Cipher-Augment Training Data Training data D = { x i , y i } ni =1 Set of cipher keys K = { k 1 , k 2 , lingual training setting (Johnson et al., 2017). For a plaintext sample x i , ciphertext samples { ROT  k j ( x i ) , ..., ROT  k | K | ( x i ) } , and target sequence y i , the multi-source model is trained by minimizing the cross-entropy L iNLL =  log p  ( y i | x i )  | K | (cid:88) j log p  ( y i | ROTk j ( x i )) (1) where | K | is the number of distinct keys used to generate ciphertexts. While this yields a multilingual model, this formulation does not allow explicit interaction between a plaintext sample and the corresponding ciphertexts. To allow such interactions, we design another model that relies on inherent pivoting between sources and enciphered sources. We achieve this by adding ROTk ( source )  source as a translation direction; following Johnson et al. (2017) we prepend the appropriate target token to all source sentences and train to minimize the objective L iNLL =  log p  ( y i | x i )  | K | (cid:88) j [ log p  ( y i | ROTk j ( x i )) + log p  ( x i | ROTk j ( x i )) ] (2) We refer to (2) as the naive model. Discussion. In this setting the decoder must learn the distributions of both the true target language and the source language. This may lead to quicker saturation of the decoder and sub-optimal use of its capacity, which must now be shared between two languages; this is a notorious property of many-to-many multilingual NMT (Aharoni et al., 2019). To better leverage the equivalence between plain-and ciphertext data, we take inspiration from multiview learning (Xu et al., 2013). We rethink enciphered samples as different views of the authentic source samples which can be exploited for co-202 training (Blum and Mitchell, 1998). This is motivated by the observation that plain and enciphered samples have identical sentence length, grammar, and (most importantly) sentential semantics. Given an enciphered source cipher ( x i ) we model the loss for a plaintext sample ( x i , y i ) as L i =  1 L iNLL ( p  ( y i | x i ) ) (cid:124) (cid:123)(cid:122) (cid:125) anchor source x-entropy +  2 L iNLL ( p  ( y i | cipher ( x i )) ) (cid:124) (cid:123)(cid:122) (cid:125) cipher source x-entropy +  L idist ( p  ( y i | x i ) , p  ( y i | cipher ( x i )) ) (cid:124) (cid:123)(cid:122) (cid:125) agreement loss, see (4) (3) where the original source language sentence x i is called the anchor here since it is always paired with each enciphered version. The first two terms are conventional negative log-likelihoods, to encourage the model to generate the appropriate target for both x i and cipher ( x i ) . The third term is the agreement loss , measured as the pairwise symmetric KL divergence 2 between the output distributions for x i and cipher ( x i ) : L idist ( p  ( y i | x i ) , p  ( y i | cipher ( x i )) ) = 1 2[ D iKL ( p flat  ( y i | x i ) || p  ( y i | cipher ( x i )) ) + D iKL ( p flat  ( y i | cipher ( x i )) || p  ( y i | x i )) ] (4) This term allows explicit interactions between plainand ciphertexts by way of co-regularization . Co-regularization relies on the assumption that the target functions in each view agree on labels of most examples (Sindhwani et al., 2005) and constrains the model to consider only solutions which capture this agreement. In cases where there are many output classes and the model predictions strongly favour certain of these classes, (4) may have an outsized influence on model behaviour. As a precautionary measure, we use a softmax temperature  to flatten the model predictions, based on a similar technique in knowledge distillation (Hinton et al., 2015) and multi-view regularization (Wang et al., 2021). The flattened prediction for an ( x, y ) pair is given by p flat  ( x | y ) = exp ( z y ) / (cid:80) y j exp ( z y j ) / (5) where z y is the logit for the output label y . A higher value of  produces a softer, more even distribution over output classes. Datasets We use the widely studied IWSLT14 De  En and IWSLT17 Fr  En language pairs as our small-sized datasets. 3 For high-resource experiments, we evaluate on the standard WMT14 En  De set of 4.5M sentence pairs. 4 We also extend our experiments to the extremely low-resource pair Sk  En from the multilingual TED dataset (Qi et al., 2018) with 61k training samples, and dev and test splits of size 2271 and 2245 respectively. Ciphertext Generation and Vocabularies. We use a variant of ROTk which preserves whitespace, numerals, special characters, and punctuation. As a result, these characters appear the same in both plainand ciphertexts. For our naive approach, we encipher the German side of the IWSLT14 dataset with up to 20 keys {1,2,3,4,5, . . . ,20} . For our main experiments, we encipher the source side of every translation direction 5 with key {1} for WMT experiments and keys {1,2} for the rest. 6 We use sentencepiece (Kudo and Richardson, 2018) to tokenize text into byte-pair encodings 3 The De  En data has a train/dev/test split of about 170k/7k/7k. The Fr  En data has a 236k/890/1210 split using dev2010 and tst2015 . newstest2013 and test on newstest2014 5 In all generated ciphertexts, the source alphabet is preserved, only the distribution of characters is changed. The target side is never altered. (BPE; Sennrich et al. 2016c) by jointly learning subwords on the source, enciphered-source, and target sides. We tune the number of BPE merges as recommended by Ding et al. (2019); the resulting subword vocabulary sizes for each dataset are tabulated in Table 1. In all experiments, we set the loss weight hyperparameters  1 ,  2 to 1, and  to 5. Section 4.1 shows an ablation over  to justify this setting. We find that softmax temperature  = 1 works well for all experiments;  = 2 results in more stable training for larger datasets. Evaluation We evaluate on BLEU scores 7 (Pa-pineni et al., 2002). Following previous work (Vaswani et al., 2017; Nguyen et al., 2019; Xu et al., 2021), we compute tokenized BLEU with multi_bleu.perl 8 for IWSLT14 and TED datasets, additionally apply compound-splitting for WMT14 En-De 9 and SacreBLEU 10 (Post, 2018) for IWSLT17 datasets. For all experiments, we perform significance tests based on bootstrap resampling (Clark et al., 2011) using the compare-mt toolkit (Neubig et al., 2019). Baselines Our main baselines are strong and widely used data-augmentation techniques that do not use external data. We compare CipherDAug to back-translation-based data-diversification (Nguyen et al., 2019), word replacement techniques like SwitchOut (Wang et al., 2018), WordDrop (Sennrich et al., 2016a), and RAML (Norouzi et al., 2016), and the subword-regularization technique BPE-Dropout (Provilkov et al., 2020). See supplemental sections A.1 and A.2 for further baseline and implementation details. 7 Decoder beam size 4 and length penalty 0.6 for WMT, and 5 and 1.0 for all other experiments. 8 mosesdecoder/scripts/generic/multi-bleu.perl 9 tensorflow/tensor2tensor/utils/get_ende_bleu.sh 10 SacreBLEU signature: nrefs:1|case:mixed| eff:no|tok:13a|smooth:exp|version:2.0.0 Model De  En Transformer 34.91 + Word Dropout 34.83 + SwitchOut 34.82 + RAML 35.11 + RAML + Switchout 35.17 + RAML + WordDrop 35.47 Naive Multi-Source Equation (1) Equation (2) 2 keys 35.45 35.85 5 keys 35.65 35.98 10 keys 33.70 35.42 20 keys 32.95 34.75 5 keys + RAML + Switchout -36.17 5 keys + RAML + WordDrop -36.63 CipherDAug 1 key 36.21 CipherDAug 2 keys 37.60 Table 2: Results on the IWSLT14 De-En validation set comparing the naive approach and CipherDAug. Simply using 2 enciphered sources gives a BLEU score of 35.45, which nearly matches the performance of the best baseline, RAML+SwitchOut, at 35.47. Adding the ROTk (source)  source direction further improves the score to 35.85. Adding the ROT-k (source)  source direction consistently yields better results than the vanilla multi-source model, but increasing the number of keys has a less consistent effect. We hypothesize that more keys are generally beneficial, but that the model becomes saturated when too many are used. Based on these observations, we limit later experiments to 2 keys. We observe further gains by combining the naive method with the two best performing baselines. This emphasizes that ciphertext-based augmentation is orthogonal to other data-augmentation methods and can be seamlessly combined with these to yield greater improvements. We present our main results in Table 3. While using a single key improves significantly over the Transformer model, augmenting with 2 keys outperforms all baselines. Table 4 shows additional comparisons against approaches that introduce architectural improvements to the transformer (such as MAT; Fan et al. 2020) or that require large pre-trained models, like BiBERT (Xu et al., 2021). our method yields stronger improvements over the standard Transformer than any other data augmentation technique (Table 3). This includes strong methods such RAML+SwitchOut and data diversification, which report improvements as high as 1.8 and 1.9 BLEU points respectively. Data diversification involves training a total of 7 different models for forward and backward translation on the source and target data. By contrast, CipherDAug trains a single model, and improves the baseline transformer by 2.9 BLEU points on IWSLT14 De  En and about 2.2 BLEU points on the smaller datasets. On WMT14 En  De, our method using 1 key improves by 0.6 BLEU over the baseline transformer and significantly outperforms word replacement methods like SwitchOut and WordDropout. 12 Wu et al. 2020 introduce a new model architecture for mixing subword representations that involves a two-stage training process. CipherDAug, on the other hand, only uses a vanilla Transformer that is trained end-to-end. Low-resource setting The Sk  En dataset is uniquely challenging as it has only 61k pairs of training samples. This dataset is generally paired with a related high-resource language pair such as Cs-En (Neubig and Hu, 2018), or trained in a massively multilingual setting (Aharoni et al., 2019) with 58 other languages from the multilingual TED dataset (Qi et al., 2018). Xia et al. (2019b) introduced a generalized data augmentation technique that works in this multilingual setting and leverages over 2M monolingual sentences for each language using back-translation. Applying CipherDAug to this dataset (Table 5) yields significant improvements over these methods, achieving 32.62 BLEU on Sk  En and 24.61 on En  Sk. Discussion On the relatively larger WMT14 dataset (4.5M), despite improving significantly over the baseline Transformer, the Base model 205 |src  tgt| |vocab| D emb Emb  Train  BLEU Transformer-256 12k 12k 256 3M 37M 34.40 Transformer-512 12k 12k 512 6.1M 44M 34.64 Transformer-256 20k 20k 256 5.1M 42M 34.19 Transformer-512 20k 20k 512 10.1M 52M 34.39 CipherDAug-1key 11.8k 16k 256 4.1M 40M 36.25 CipherDAug-1key 11.8k 16k 512 8.2M 47M 36.19 CipherDAug-2keys 11.8k 20k 256 5M 42M 36.90 CipherDAug -2keys 11.8k 20k 512 10.1M 52M 37.53 Table 6: Results on IWSLT14 De  En with baseline Transformer and CipherDAug using different vocabulary sizes and embedding dimensions. (68M params) approaches saturation when  9M enciphered sentences (2 keys) are added. Upgrading to Transformer Big (218M) may be viable, but would be an unfair comparison with other models. The model capacity becomes a bottleneck with larger datasets when the model is optimised to translate each of the source sentences (4.5M plain and 9M enciphered) individually (single-source) as well as together (multi-source) through the co-regularization loss. The results indicate that our proposed approach works best in small and low resource data settings. Number of Keys Figure 2 (left) shows the effect of adding different amounts of enciphered data. We obtain the best performance using just 2 different keys. Using more or fewer degrades performance, though both cases still outperform the baseline. As noted in Section 3.2, the model may become saturated when too many keys are used. Agreement Loss Figure 2 (right) shows an ablation analysis on the agreement loss. We find that CipherDAug is sensitive to the weight  given to this term: increasing or decreasing it from our default setting  = 5 incurs a performance drop of nearly 2 BLEU. Despite the performance gains attendant to this term, it is equally clear that agreement loss cannot fully account for CipherDAug's improvements over the baseline: in the naive setting where  = 0 , CipherDAug still outperforms the baseline by approximately 1 BLEU. Learning BPE vocabularies jointly vs. separately From Table 7, we see that there is no significant impact on BLEU if we learn BPE vocabu-number of updates BLEU 32 34 36 38 20000 40000 60000 80000 100000 1 key 5 keys 2 keys Baseline KL div. laries separately for each language or enciphered language from IWSLT14 De  En. This is consistent with results from Neubig and Hu (2018) in the context of mutilingual NMT. Note that it is preferable to learn the BPEs jointly as this allows us to limit the total vocabulary size. When learned separately, we cannot control the combined vocabulary size which may result in a larger or smaller vocabulary (and therefore, a different number of embedding parameters) than intended. Disentangling the effects of increased parameters in the embedding layer CipherDAug leverages the combined vocabularies of the original parallel bitext and enciphered copies of the source text. This necessarily increases in the number of parameters in the embedding layer even though the rest of the network remains identical. To understand the effect of these extra parameters, we compare CipherDAug against the baseline Transformer model with different vocabulary and embedding sizes. Results from different settings are shown in Table 6. 13 As we reduce the embedding dimension of our best model (CipherDAug with 2 keys) from 512 to 256, we observe a small change of -0.6 BLEU in the final scores. With 1 cipher key, however, our model exhibits a slight (statistically insignificant) improvement of +0.06 BLEU. These results show that the few extra embedding parameters in CipherDAug do not have an outsized impact on model performance, but we emphasize that reducing the dimensionality of the embedding layer diminishes its expressivity and is therefore not a completely fair comparison. The attention mechanism of a model might not reflect a model's true inner reasoning (Jain and Wallace, 2019; Moradi et al., 2019, 2021). To better analyze NMT models, Lee et al. (2018) introduce the notion of hallucinations . A model hallucinates when small perturbations in its input cause drastic changes in the output, implying it is not actually attentive to this input. Using Algorithm 2 of Raunak et al. (2021), Table 8 shows the number of hallucinations on the IWSLT14 De-En test set for the baseline and CipherDAug models. We use the 50 most common subwords as perturbations. CipherDAug sees a 40% reduction in hallucinations relative to the baseline, suggesting it is more resilient against perturbations and more attentive to the content of its input. We argue that CipherDAug is effective in part because it reduces the impact of rare words. On average, the rarest subword in a ROTk enciphered 13 Note that in Table 6, the BPE vocabularies from the original source and target remain approximately same across the baseline (12k) and CipherDAug (11.8k) even though the final vocabulary sizes of our models vary with the addition of the enciphered source(s). sentence is significantly more frequent than the rarest subword in a plaintext sentence. This is apparent in an example like the following: hier ist es ntig, das, was wir unter politically correctness verstehen, immer wieder anzubringen. Figure 3 plots the frequency of each subword in this sentence and its ROTk enciphered variants. In the plaintext, we observe a series of rare subwords ically , _correct , and ness coming from the English borrowing. After encipherment, however, these are replaced by a variety of more common subwords jd , bmm , _d , and so on. The result is that the enciphered sentences have fewer rare subwords; this allows them to share more information with other sentences, and allows the more common enciphered tokens to inform the model's encoding of less common plaintext tokens. We reiterate that this trend holds across the whole corpus, and highlights the value of an augmentation scheme that allows a model to see many different segmentations of each input. This is not the only mechanism by which CipherDAug improves performance: we find improvements for tokens in every frequency bucket, not simply those which are rare (Figure 4). In Section 2.2, we argue that the agreement loss in (4) acts as a co-regularization term in a multiview learning setting. Multi-view learning works best when the different views capture distinct information. In CipherDAug, this is accomplished by 207 target word frequency f m ea s u r e 0.2 0.3 0.4 0.5 0.6 0.7 < 1 1 2 3 4 [ 5 , 10 ) [ 10 , 100 ) [ 100 , 1 k ) >= 1 k Transformer Moving Avg CipherDAug Moving Avg sentence length s e n t e n c e BLEU 15 20 25 30 35 40 < 10 [ 10 , 20 ) [ 20 , 30 ) [ 30 , 40 ) [ 40 , 50 ) [ 50 , 60 ) >= 60 Transformer CipherDAug Figure 4: CipherDAug yields improvements for tokens of all frequencies and sentences of every length. allowing enciphered inputs to receive different segmentations than plaintext inputs. As evidence that the different views capture distinct information, we note that even after training with co-regularization the model remains sensitive to the choice of input encoding, as seen in cases such as Figure 6 where the model may produce any of three distinct outputs depending on whether it is given plainor ciphertext as input. If all of the input views captured identical information we should expect no such variation, especially after training with an explicit co-regularization term. To further analyze CipherDAug, we turn to canonical correlation analysis (CCA; Hardoon et al. 2004; Raghu et al. 2017), which finds a linear transform to maximize correlation between values in two high dimensional datasets. As detailed in Raghu et al. 2017, it is useful for measuring correlations between activations from different networks. For each IWSLT14 De-En test sentence, we save the activations from each layer of our baseline and CipherDAug models. For the CipherDAug model, we save activations on plaintext and enciphered inputs. For every pair of layers, we compute the projection weighted 14 CCA (PWCCA) between activations from those layers. If this value is high (rel-ative to a random baseline), this means that there is a linear transformation under which the activations from those layers are linearly correlated, implying that the layers capture similar information. Figure 5 plots the PWCCA between encoder states from the baseline and CipherDAug models, and between CipherDAug encoder states with dif-14 See Raghu et al. 2017 for an explanation of CCA variants including PWCCA. ferent input encodings. It is immediately clear that CipherDAug learns similar, but not identical, representations for plainand ciphertext inputs: the state of a layer in the de  en setting is generally predictive of the state of that same layer in the ROT-1(de)  en and ROT-2(de)  en settings. We emphasize, however, that representations for plainand ciphertexts are not identical, as can be seen by comparing against the baseline model. Here, some layers in one model show a moderate correlation to every layer of the other model; other layers show a strong correlation with a different layer from the other model. This implies that, while the two models extract some of the same information, they do so at different depths in the encoder. Moreover, CipherDAug states from enciphered inputs present an entirely different pattern of correlations than plaintext inputs. This implies that CipherDAug not only learns different information than the baseline, but that these differences are distinct for plaintexts and ciphertexts. These results strengthen Section 4.4's claim that plain-and ciphertexts capture distinct information. Data-augmentation (Sennrich et al., 2016b) can be broadly categorized into back-translation based methods and those which perturb or change the input (Wang et al., 2018). Back-translation (Sennrich et al., 2016b) is arguably the de-facto data augmentation method for NMT. Besides back-translating 208 Model De  En Transformer 34.71 CipherDAug-2keys de  en 37.53 ROT-1(de)  en 37.41 ROT-2(de)  en 37.35 Source: sein onkel floh mit ihrer heiligkeit in die diaspora, die leute nach nepal brachte. external monolingual data (Edunov et al., 2018), Li et al. (2019) forward-translate the source (Zhang and Zong, 2016) and/or backward-translate the target side (Sennrich et al., 2016a) of the original (in-domain) parallel data. Our technique produces lexically diverse samples using only the original source data, rather than relying on model predictions which may be of limited quality. Belinkov and Bisk (2018) showed that NMT models can be sensitive to orthographic variation, and that training with noise improves their robustness (Khayrallah and Koehn, 2018). Common noising techniques include token dropping (Zhang et al., 2020), word replacement (Xie et al., 2017; Wu et al., 2021a), Word-Dropout (randomly zeroing out word embeddings; Sennrich et al. 2016a; Gal and Ghahramani 2016) and adding synthetic noise by swapping random characters or replacing words with common typos (Karpukhin et al., 2019). Adding enciphered data is distinct from noising as the ciphertexts are generated deterministically and follow the same distribution as the underlying natural language, simply using shifted letters of the same alphabet. 15 To extend the support of the empirical data distribution, Norouzi et al. (2016) introduced RAML on the target side; Wang et al. (2018) proposed SwitchOut as a more general method which they applied to the source side. Special cases of SwitchOut include Word-Dropout and sequence-mixing (Guo et al., 2020a), which exchanges words between similar source sentences to encourage compositional behaviour. Such methods generate several different samples for each sentence because of the large vocabulary to choose replacements from; they often give poor coverage despite this. In contrast, CipherDAug guarantees lexically diverse examples with semantic equivalence to the source sentences without having to choose specific replacements. Adversarial techniques (Gao et al., 2019) perform soft perturbations of tokens or spans 15 CipherDAug can also apply to non-alphabetic scripts (e.g. Mandarin, Japanese) by incrementing Unicode codepoints modulo the size of the block containing the script in question. (Takase and Kiyono 2021, Karpukhin et al. 2019). An advantage of soft replacements over hard ones is that they take into account the context of the tokens being replaced (Liu et al., 2021; Mohiuddin et al., 2021). These methods require architectural changes to a model whereas CipherDAug does not. Ciphertext-based augmentation is orthogonal to most other data-augmentation methods and can be seamlessly combined with these to jointly improve neural machine translation. We introduce CipherDAug, a novel technique for augmenting translation data using ROTk enciphered copies of the source corpus. This technique requires no external data, and significantly outperforms a variety of strong existing data augmentation techniques. We have shown that an agreement loss term, which minimizes divergence between representations of plainand ciphertext inputs, is crucial to the performance of this model, and we have explained the function of this loss term with reference to co-regularization techniques from multi-view learning. We have also demonstrated other means by which enciphered data can improve model performance, such as by reducing the impact of rare words. Overall, CipherDAug shows promise as a simple, out-of-the-box approach to data augmentation which improves on and combines easily with existing techniques, and which yields particularly strong results in low-resource settings. We would like to thank the anonymous reviewers for their helpful comments and Kumar Abhishek for the numerous discussions that helped shape this paper. The research was partially supported by the Natural Sciences and Engineering Research Council of Canada grants NSERC RGPIN-2018-06437 and RGPAS-2018-522574 and a Department of National Defence (DND) and NSERC grant DGDND-2018-00025 to the third author.
objective abstain objective result result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain objective result result method abstain abstain abstain abstain abstain abstain abstain method abstain other abstain method method abstain abstain abstain method method abstain method abstain other abstain abstain method abstain abstain method method method abstain method abstain method method method abstain abstain abstain abstain method method abstain method method method method method method abstain abstain other abstain abstain abstain method method method abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other method other other other other other other other other other other other other objective abstain result objective abstain other other