We propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating (path-specific) counterfactual aligned phrases. We generate these by sampling new source phrases from a masked language model, then sampling an aligned counterfactual target phrase by noting that a translation language model can be interpreted as a Gumbel-Max Structural Causal Model (Oberst and Sontag, 2019). Compared to previous work, our method takes both context and alignment into account to maintain the symmetry between source and target sequences. Experiments on IWSLT'15 English  Vietnamese, WMT'17 English  German, WMT'18 English  Turkish, and WMT'19 robust English  French show that the method can improve the performance of translation, backtranslation and translation robustness. Neural machine translation (NMT) models (Kalch-brenner and Blunsom, 2013; Bahdanau et al., 2014; Vaswani et al., 2017) have reached state-of-the-art performance on various benchmarks. However, these models frequently rely on large-scale parallel corpora for training, exhibiting degraded performance on low-resource languages (Zoph et al., 2016). Further, modern NMT systems are often brittle, as noises (e.g. grammatical errors) can cause significant mistranslations (Sakaguchi et al., 2017; Michel and Neubig, 2018). Data augmentation is a promising direction to overcome these issues. It works by enlarging the number of data points for training without manually collecting new data. It is widely used to improve diversity and robustness and to avoid overfitting on small datasets. Even though data augmentation (e.g. image flipping, cropping and blurring) has GX 1 <latexit sha1_base64="k9WkuKGgiaiqCUB1ED1VEkollNs=">AAACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiRV0GXBhS4r2Ae0sUymN+3QySTMTIQSAn6HH+BWP8GduHXnF/gbTtIsbOuBgXPPuZd753gRo1LZ9rdRWlldW98ob1a2tnd298z9g7YMY0GgRUIWiq6HJTDKoaWoYtCNBODAY9DxJteZ33kEIWnI79U0AjfAI059SrDS0sA86gdYjT0/uUkHiZM+5CXBLOmmA7Nq1+wc1jJxClJFBZoD86c/DEkcAFeEYSl7jh0pN8FCUcIgrfRjCREmEzyCnqYcByDdJP9Dap1qZWj5odCPKytX/04kOJByGni6MztRLnqZ+J/Xi5V/5SaUR7ECTmaL/JhZKrSyQKwhFUAUm2qCiaD6VouMscBE6djmtkidCwhgWTLOYg7LpF2vOee1+t1FtWEXGZXRMTpBZ8hBl6iBblETtRBBT+gFvaI349l4Nz6Mz1lryShmDtEcjK9fey2hUA==</latexit> GX 2 <latexit sha1_base64="rhCGUP7ylxcMUtAWVv78y5IyK28=">AAACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiRV0GXBhS4r2Ae0sUymN+3QySTMTIQSAn6HH+BWP8GduHXnF/gbTtIsbOuBgXPPuZd753gRo1LZ9rdRWlldW98ob1a2tnd298z9g7YMY0GgRUIWiq6HJTDKoaWoYtCNBODAY9DxJteZ33kEIWnI79U0AjfAI059SrDS0sA86gdYjT0/uUkHST19yEuCWdJNB2bVrtk5rGXiFKSKCjQH5k9/GJI4AK4Iw1L2HDtSboKFooRBWunHEiJMJngEPU05DkC6Sf6H1DrVytDyQ6EfV1au/p1IcCDlNPB0Z3aiXPQy8T+vFyv/yk0oj2IFnMwW+TGzVGhlgVhDKoAoNtUEE0H1rRYZY4GJ0rHNbZE6FxDAsmScxRyWSbtec85r9buLasMuMiqjY3SCzpCDLlED3aImaiGCntALekVvxrPxbnwYn7PWklHMHKI5GF+/fNOhUQ==</latexit> GX 3 <latexit sha1_base64="GGmnjJWgBubbJgjXJttnNZcxV6g=">AAACGHicbVDLSsNAFJ34rPUVdaebYBFclaQVdFlwocsK9gFtLJPpTTt0MgkzE6GEgN/hB7jVT3Anbt35Bf6GkzQL23pg4Nxz7uXeOV7EqFS2/W2srK6tb2yWtsrbO7t7++bBYVuGsSDQIiELRdfDEhjl0FJUMehGAnDgMeh4k+vM7zyCkDTk92oagRvgEac+JVhpaWAe9wOsxp6f3KSDpJ4+5CXBLOmmA7NiV+0c1jJxClJBBZoD86c/DEkcAFeEYSl7jh0pN8FCUcIgLfdjCREmEzyCnqYcByDdJP9Dap1pZWj5odCPKytX/04kOJByGni6MztRLnqZ+J/Xi5V/5SaUR7ECTmaL/JhZKrSyQKwhFUAUm2qCiaD6VouMscBE6djmtkidCwhgWTLOYg7LpF2rOvVq7e6i0rCLjEroBJ2ic+SgS9RAt6iJWoigJ/SCXtGb8Wy8Gx/G56x1xShmjtAcjK9ffnmhUg==</latexit> GX 4 <latexit sha1_base64="RQRWpR4wEM/zCK6MyWqydogiQ+w=">AAACGHicbVDLSsNAFJ34rPUVdaebYBFclaQWdFlwocsK9gFtLJPpTTt0MgkzE6GEgN/hB7jVT3Anbt35Bf6GkzQL23pg4Nxz7uXeOV7EqFS2/W2srK6tb2yWtsrbO7t7++bBYVuGsSDQIiELRdfDEhjl0FJUMehGAnDgMeh4k+vM7zyCkDTk92oagRvgEac+JVhpaWAe9wOsxp6f3KSDpJ4+5CXBLOmmA7NiV+0c1jJxClJBBZoD86c/DEkcAFeEYSl7jh0pN8FCUcIgLfdjCREmEzyCnqYcByDdJP9Dap1pZWj5odCPKytX/04kOJByGni6MztRLnqZ+J/Xi5V/5SaUR7ECTmaL/JhZKrSyQKwhFUAUm2qCiaD6VouMscBE6djmtkidCwhgWTLOYg7LpF2rOhfV2l290rCLjEroBJ2ic+SgS9RAt6iJWoigJ/SCXtGb8Wy8Gx/G56x1xShmjtAcjK9fgB+hUw==</latexit> i would love a sandwich ich wrde ein sandwich lieben GY 1 <latexit sha1_base64="z1XM2OhhTgc51FaxMG+3wlSz3zs=">AAACGHicbVDLSsNAFJ34rPUVdaebwSK4KkkVdFlwocsK9iFtDJPpTTt08mBmIpQQ8Dv8ALf6Ce7ErTu/wN9w0mZhWw8MnHvOvdw7x4s5k8qyvo2l5ZXVtfXSRnlza3tn19zbb8koERSaNOKR6HhEAmchNBVTHDqxABJ4HNre6Cr3248gJIvCOzWOwQnIIGQ+o0RpyTUPewFRQ89PrzM3tbOHSUkJT+8z16xYVWsCvEjsglRQgYZr/vT6EU0CCBXlRMqubcXKSYlQjHLIyr1EQkzoiAygq2lIApBOOvlDhk+00sd+JPQLFZ6ofydSEkg5DjzdmZ8o571c/M/rJsq/dFIWxomCkE4X+QnHKsJ5ILjPBFDFx5oQKpi+FdMhEYQqHdvMFqlzAQE8T8aez2GRtGpV+6xauz2v1K0ioxI6QsfoFNnoAtXRDWqgJqLoCb2gV/RmPBvvxofxOW1dMoqZAzQD4+sXfMehUQ==</latexit> GY 3 <latexit sha1_base64="WCQxaEeNXu9Aufbl2yA+fYO1yGg=">AAACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiStoMuCC11WsA9pY5hMb9qhkwczE6GEgN/hB7jVT3Anbt35Bf6GkzQL23pg4Nxz7uXeOW7EqJCm+a2VVlbX1jfKm5Wt7Z3dPX3/oCPCmBNok5CFvOdiAYwG0JZUMuhFHLDvMui6k6vM7z4CFzQM7uQ0AtvHo4B6lGCpJEc/GvhYjl0vuU6dpJE+5CXBLLlPHb1q1swcxjKxClJFBVqO/jMYhiT2IZCEYSH6lhlJO8FcUsIgrQxiAREmEzyCvqIB9kHYSf6H1DhVytDwQq5eII1c/TuRYF+Iqe+qzuxEsehl4n9eP5bepZ3QIIolBGS2yIuZIUMjC8QYUg5EsqkimHCqbjXIGHNMpIptbotQuQAHliVjLeawTDr1mtWo1W/Pq02zyKiMjtEJOkMWukBNdINaqI0IekIv6BW9ac/au/ahfc5aS1oxc4jmoH39AoAToVM=</latexit> GY 4 <latexit sha1_base64="V4NqcQT+xKxlby5wn7lpUAxZg8Q=">AAACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiS1oMuCC11WsA9pY5hMb9qhkwczE6GEgN/hB7jVT3Anbt35Bf6GkzQL23pg4Nxz7uXeOW7EqJCm+a2VVlbX1jfKm5Wt7Z3dPX3/oCPCmBNok5CFvOdiAYwG0JZUMuhFHLDvMui6k6vM7z4CFzQM7uQ0AtvHo4B6lGCpJEc/GvhYjl0vuU6dpJE+5CXBLLlPHb1q1swcxjKxClJFBVqO/jMYhiT2IZCEYSH6lhlJO8FcUsIgrQxiAREmEzyCvqIB9kHYSf6H1DhVytDwQq5eII1c/TuRYF+Iqe+qzuxEsehl4n9eP5bepZ3QIIolBGS2yIuZIUMjC8QYUg5EsqkimHCqbjXIGHNMpIptbotQuQAHliVjLeawTDr1mnVeq982qk2zyKiMjtEJOkMWukBNdINaqI0IekIv6BW9ac/au/ahfc5aS1oxc4jmoH39AoG5oVQ=</latexit> GY 5 <latexit sha1_base64="V7US2RzZJ4vJ3zHJ61FCV1eBSc4=">AAACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiRV0WXBhS4r2Ie0MUymN+3QyYOZiVBCwO/wA9zqJ7gTt+78An/DSZqFbT0wcO4593LvHDdiVEjT/NZKS8srq2vl9crG5tb2jr671xZhzAm0SMhC3nWxAEYDaEkqGXQjDth3GXTc8VXmdx6BCxoGd3ISge3jYUA9SrBUkqMf9H0sR66XXKdOcp4+5CXBLLlPHb1q1swcxiKxClJFBZqO/tMfhCT2IZCEYSF6lhlJO8FcUsIgrfRjAREmYzyEnqIB9kHYSf6H1DhWysDwQq5eII1c/TuRYF+Iie+qzuxEMe9l4n9eL5bepZ3QIIolBGS6yIuZIUMjC8QYUA5EsokimHCqbjXICHNMpIptZotQuQAHliVjzeewSNr1mnVaq9+eVRtmkVEZHaIjdIIsdIEa6AY1UQsR9IRe0Ct60561d+1D+5y2lrRiZh/NQPv6BYNfoVU=</latexit> GX 5 <latexit sha1_base64="/6YisMcD2nY5Ql5gU77jQTisf+Y=">AAACGHicbVDLSsNAFJ34rPUVdaebYBFclaQquiy40GUF+4A2lsn0ph06mYSZiVBCwO/wA9zqJ7gTt+78An/DSZqFbT0wcO4593LvHC9iVCrb/jaWlldW19ZLG+XNre2dXXNvvyXDWBBokpCFouNhCYxyaCqqGHQiATjwGLS98XXmtx9BSBryezWJwA3wkFOfEqy01DcPewFWI89PbtJ+cpE+5CXBLOmkfbNiV+0c1iJxClJBBRp986c3CEkcAFeEYSm7jh0pN8FCUcIgLfdiCREmYzyErqYcByDdJP9Dap1oZWD5odCPKytX/04kOJByEni6MztRznuZ+J/XjZV/5SaUR7ECTqaL/JhZKrSyQKwBFUAUm2iCiaD6VouMsMBE6dhmtkidCwhgWTLOfA6LpFWrOmfV2t15pW4XGZXQETpGp8hBl6iOblEDNRFBT+gFvaI349l4Nz6Mz2nrklHMHKAZGF+/gcWhVA==</latexit> X <latexit sha1_base64="DSqt/3Qa7/Ug0Q+grni0IN8OAF8=">AAACB3icbVDLSgMxFM34rPVVdekmWARXZaYKuiy6cVnBPmA6lEx6pw3NJEOSEcrQD/AD3OonuBO3foZf4G+YaWdhWw8EDufcyz05YcKZNq777aytb2xubZd2yrt7+weHlaPjtpapotCikkvVDYkGzgS0DDMcuokCEoccOuH4Lvc7T6A0k+LRTBIIYjIULGKUGCv5vZiYESU86077lapbc2fAq8QrSBUVaPYrP72BpGkMwlBOtPY9NzFBRpRhlMO03Es1JISOyRB8SwWJQQfZLPIUn1tlgCOp7BMGz9S/GxmJtZ7EoZ3MI+plLxf/8/zURDdBxkSSGhB0fihKOTYS5//HA6aAGj6xhFDFbFZMR0QRamxLC1e07QUU8LwZb7mHVdKu17zLWv3hqtq4LToqoVN0hi6Qh65RA92jJmohiiR6Qa/ozXl23p0P53M+uuYUOydoAc7XL+rvmqg=</latexit> Y <latexit sha1_base64="wVsefuj4A71dBp3jPzQ+93dXKxs=">AAACB3icbVDLSgMxFM3UV62vqks3wSK4KjNV0GXRjcsK9iHtUDLpnTY0kwxJRihDP8APcKuf4E7c+hl+gb9hpp2FbT0QOJxzL/fkBDFn2rjut1NYW9/Y3Cpul3Z29/YPyodHLS0TRaFJJZeqExANnAloGmY4dGIFJAo4tIPxbea3n0BpJsWDmcTgR2QoWMgoMVbq9iJiRpTw9HHaL1fcqjsDXiVeTiooR6Nf/ukNJE0iEIZyonXXc2Pjp0QZRjlMS71EQ0zomAyha6kgEWg/nUWe4jOrDHAolX3C4Jn6dyMlkdaTKLCTWUS97GXif143MeG1nzIRJwYEnR8KE46NxNn/8YApoIZPLCFUMZsV0xFRhBrb0sIVbXsBBTxrxlvuYZW0alXvolq7v6zUb/KOiugEnaJz5KErVEd3qIGaiCKJXtArenOenXfnw/mcjxacfOcYLcD5+gXsiZqp</latexit> GY 2 <latexit sha1_base64="6VoUEA0hP001E3j1j8xL/LIcIFc=">AAACGHicbVDLSsNAFJ34rPUVdaebwSK4KkkVdFlwocsK9iFtLJPpTTt08mBmIpQQ8Dv8ALf6Ce7ErTu/wN9wkmZhWw8MnHvOvdw7x404k8qyvo2l5ZXVtfXSRnlza3tn19zbb8kwFhSaNOSh6LhEAmcBNBVTHDqRAOK7HNru+Crz248gJAuDOzWJwPHJMGAeo0RpqW8e9nyiRq6XXKf9pJY+5CUlPLlP+2bFqlo58CKxC1JBBRp986c3CGnsQ6AoJ1J2bStSTkKEYpRDWu7FEiJCx2QIXU0D4oN0kvwPKT7RygB7odAvUDhX/04kxJdy4ru6MztRznuZ+J/XjZV36SQsiGIFAZ0u8mKOVYizQPCACaCKTzQhVDB9K6YjIghVOraZLVLnAgJ4low9n8MiadWq9lm1dnteqVtFRiV0hI7RKbLRBaqjG9RATUTRE3pBr+jNeDbejQ/jc9q6ZBQzB2gGxtcvfm2hUg==</latexit> Figure 1: We interpret a translation language model p ( Y j |X , Y  j ) ( Y  j means that phrase Y j has been removed from sequence Y ) as a causal model. become a standard technique in computer vision (Krizhevsky et al., 2012; Huang et al., 2017; Chen et al., 2020), it is non-trivial to apply in machine translation since even a slight modification to a sequence can result in drastic changes in its syntax and semantics. Indeed there is relatively little work in this direction due to these difficulties (Sennrich et al., 2016; Fadaee et al., 2017; Wang et al., 2018; Gao et al., 2019; Xia et al., 2019; Kobayashi, 2018). Further, work based on word replacement either ignores the contexts of replaced words or breaks the alignment between source and target sequences, both detrimental for generating high-quality data. In this paper we observe that a translation language model can be interpreted as a causal model, as described in Figure 1. Doing so allows us to ask counterfactual questions of the form: Given source and target sequences, if a phrase in the source sequence is changed, how would the target sequence change? We propose a data augmentation method for machine translation that generates counterfactual parallel translation data. To ensure these counterfactuals are close to the original data we sample a new source phrase from a masked language model. We then consider the (path-specific) counterfactual target phrase that is aligned to that source phrase (given by an unsupervised phrasal alignment method). The idea is that this augmentation procedure exposes inductive biases in existing language models that enables new translation models to learn more efficiently and exhibit more robust generalisation. Specifically, our augmentation procedure performs the following three steps: 1. We utilize unsupervised phrasal alignment (e.g. Neubig et al. (2011) and Dyer et al. (2013)) to obtain correspondences between source and target phrases. 2. A source phrase is removed and then resampled according to a trained masked language model (Devlin et al., 2018; Raffel et al., 2019). 3. We perform (path-specific) counterfactual inference on the causal model given by a trained translation language model (Lample and Conneau, 2019) to resample only the aligned target phrase, given the changed source phrase. Different from prior work, our approach takes advantage of both source/target context and alignment for data augmentation. Experiments on IWSLT'15 English  Vietnamese, WMT'17 English  German, and WMT'18 English  Turkish show that our method improves the translation performance on both high-resource and low-resource datasets. We additionally demonstrate that our method complements existing approaches such as backtranslation (Sennrich et al., 2015a). Finally, we demonstrate that our method improves translation robustness (we evaluate this on the WMT'19 English  French robustness dataset). Neural machine translation. Given a set of parallel sequences, S = { ( X i , Y i ) } Ni =1 , NMT maximizes the log-likelihood of Y given X , assuming each ( X i , Y i ) pair is independently and identically distributed: max  (cid:88) ( X i , Y i ) S log p  ( Y i |X i ) . However, paired sequences are usually expensive to collect, as it requires an expert to translate sequences X i into another language Y i . Data augmentation aims to generate new parallel sequences (  X i ,  Y i ) without manually collecting new data. Phrasal alignment. Phrasal alignment identifies the translation relationships among phrases in parallel sequences. Given a parallel sequence ( X , Y ) , where X = ( X 1 = x 1 , X 2 = x 2 , ..., X |X| = x |X| ) and Y = ( Y 1 = y 1 , Y 2 = y 2 , ..., Y |Y| = y |Y| ) ( X/Y and x/y denote a phrase and its value, re-spectively), phrasal alignment h learns a mapping that projects each position i of X to a position j of Y , i.e. j = h ( i ) . In this paper, we use pialign (Neubig et al., 2011) to obtain alignments. Causal modelling. We formulate causality using the structural causal model (SCM) framework of Pearl (2003). Each SCM is a set of structural equations represented by a graph. The edges of this graph specify the inputs and outputs of the structural equations. Specifically, a variable V i is caused by a set of observable parent variables pa ( V i ) and unobserved variables U i if there exists a (determin-istic or stochastic) structural equation f i : V i = f i ( pa ( V i ) , U i ) . If the structural equations f are identified, it is possible to compute a causal quantity called counterfactuals . Counterfactuals are questions that, given the current state of the world, ask what would have changed if some variable V had been different. For example, Would a person have been able to obtain a visa if they had been born in a different country?. Formally we denote the counterfactual value of a variable V i , had another variable W  pa ( V i ) been  w (i.e., compared to its observed value w ) as V i,W   w . To compute counterfactuals we can follow a three-step procedure (for more details see Chapter 4 of Pearl et al. (2016)): 1. Abduction : Given a prior distribution on unobserved variables p ( U i ) , compute the posterior given all observed variables V = v : p ( U i | V = v ) ; 2. Action : Modify the structural equation for V i , so that W is fixed to the counterfactual value  w (the modified equation is denoted as f i,  w ); 3. Prediction : Compute the distribution p ( V i,W   w | V = v ) using p ( U i | V = v ) , the observed variables v , and the modified structural equation f i,  w . 3 Method Our goal is to take an input sequence pair ( X , Y ) and create augmented data from it. We aim to do so by removing phrases, resampling them in the source sequence, and computing the counterfactual effect of doing so in the target sequence. We argue that for any such augmentation method for NMT, it is crucial to leverage both contextual and alignment information, for the following reasons. (1) Context : As contextual information is widely used to disambiguate words (Peters et al., 2018) and generate realistic-looking sequences (Zellers et al., 2019), it is critical to utilize contextual information to obtain grammatically-correct and semantically-sound sequences. (2) Alignment : Phrasal alignment plays a critical role in statistical machine translation (Brown et al., 1993; Vogel et al., 1996). As phrasal alignment provides information about which phrase in the source sequence produces a phrase in the target sequence, a data augmentation algorithm which disregards alignment risks breaking the symmetry between source and target sequences. To this end, in Section 3.1, we introduce a technique called Translation-Counterfactual Word Replacement (TCWR) for leveraging both context and alignment to replace phrases in source and target sequences. In Section 3.2, we propose a new data augmentation algorithm based on this replacement technique. In Section 3.3, we describe the architectures used to parameterize the models. Consider the sequence pair ( X , Y ) in Figure 2. A translation language model that learns p ( Y j |X , Y  j ) (where Y  j indicates the sequence Y with Y j removed) for all j  { 1 , ..., |Y|} induces a causal graph on this pair. Specifically it is fully connected, in the following way: (a) all phrases in X cause all phrases in Y , (b) all phrases in Y cause all other phrases in Y (these connections are signified by the wide gray arrow in Figure 2). Additionally, there are unobserved variables GX i , GY i that cause each individual phrase (more on this be-low). We choose this fully connected structure to take contexts of each phrase into account. Note that this graph is cyclic, yet the counterfactual distribution we care about is identifiable given the posterior of the unobserved variables (which we describe below) and the known equations of the causal model (i.e., the translation language model). Consider that we have an alignment between X and Y , which singles-out the causal effects shown with black arrows in Figure 2. Our idea is to derive a new sequence pair (  X ,  Y ) by computing a counterfactual. We propose to calculate the counterfactual corresponding to a single alignment, i.e. a path-specific counterfactual: What would Y j have looked like, had X i =  x i instead of x i , given that Y j is aligned to X i , and all other phrases X  i , Y  j had been held constant ?. This allows us to consider 1. Context : By holding all other phrases constant we control for the specific context around the changed phrases  x i ,  y j 1 ; 2. Alignment : The derived counterfactual is based on the direct effect of X i on Y j , where this singled-out link is identified from an alignment. We now outline the three steps to calculate the counterfactual. The example in Figure 2 is used for illustration. The goal is to sample from the following counterfactual distribution: p ( Y 2 , { X 3   x 3 , X  3  x  3 , Y  2  y  2 } |X , Y ) with the translation language model, which describes What would Y 2 have looked like, had X 3 =  x 3 instead of x 3 , given that Y 2 is aligned to X 3 , and all other phrases X  3 , Y  2 had been held constant ?. For ease of illustration, we assume both X 3 and Y 2 1 Further, the posterior of unobserved random variables GY j given X , Y will encode additional context w.r.t. the original sequence pair X , Y . contain one token after Byte Pair Encoding (BPE) segmentation (Sennrich et al., 2015b). In Section 3.3, we explain how we use a sequence-to-sequence model to generate phrases containing multiple tokens after segmentation. 1. Abduction. The goal of the abduction step is to estimate any unobserved variables that im-pact the counterfactual. As our translation language model specifies a categorical distribution p ( Y 2 |X , Y  2 ) , this unobserved randomness, i.e. the prior of GY 2 , takes the form of a Gumbel random vector. This is due to the fact that random sampling from a categorical distribution can be done via a procedure called the Gumbel-Max Trick (Maddison et al., 2014b). Definition 3.1 (Gumbel-Max Trick) . Two steps are required to sample from a categorical distribution p ( Y ) with K categories: 1. Sample g 1 , . . . , g K  Gumbel (0 , 1) . Each g k can be computed as g k =  log(  log u k ) where u k  Uniform (0 , 1) ; 2. Compute y = arg max k =1 ,...,K log p ( Y = k ) + g k . y 2 = arg max k =1 ,..., | V | log p ( Y 2 = k |X , Y  2 ) + g k , s.t. g k  Gumbel (0 , 1) . The abduction step samples from the posterior distribution over these Gumbel random variables, given the observed pair ( X , Y ) , i.e., p ( GY 2 |X , Y ) . Fortunately, sampling from the posterior is straightforward to do in two steps (Maddison et al., 2014a; Maddison and Tarlow, 2017): 1. Let y 2 = k  . Sample  g k   Gumbel (0 , 1) ; 2. For the remaining k , compute the probabilities from the model p ( Y 2 = k |X , Y  2 ) , and sample from the distribution  g k  Gumbel (log p ( Y 2 = k |X , Y  2 ) , 1) truncated within the range (  ,  g k  ) . The resulting samples [ g 1 , . . . ,  g | V | ] are from the posterior p ( GY 2 |X , Y ) . We describe these steps in more detail in Algorithm 1. 2. Action. In this step, we replace a phrase x 3 in the source sequence with a substitute phrase  x 3 . While any replacement leads to a valid counterfactual, we propose to sample  x 3 as  x 3  p ( X 3 |X  3 ) , where p ( X 3 |X  3 ) is given by a trained masked language model. By sampling from a distribution Algorithm 1: Gumbel Posterior Sampling Input : The observed phrase y j = k  Probabilities p ( Y j = k |X , Y  j ) for k = 1 , . . . , | V | Output : Sampled Gumbel values g  p ( GY j |X , Y ) Sample  g k   Gumbel(0 , 1) for k  1 to | V | do if k (cid:54) = k  then // Sample from truncated Gumbel Sample h k  Gumbel (0 , 1) u k = h k + log p ( Y j = k |X , Y  j )  g k =  log( e  u k + e   g k  ) conditioned on the remaining phrases in X , we sample a realistic replacement word for X 3 . In Figure 2, we sample  x 3 = apple' in place of x 3 = house'. 3. Prediction. Given the posterior samples [ g 1 , . . . ,  g | V | ]  p ( GY 2 |X , Y ) and the substitute phrase  x 3 , we can compute the counterfactual distribution of interest p ( Y 2 , { X 3   x 3 , X  3  x  3 , Y  2  y  2 } |X , Y ) , via the trained translation language model. We do so by computing:  y 2 = arg max k =1 ,..., | V | log p ( Y 2 = k |  x 3 , X  3 , Y  2 ) +  g k . The sample  y 2 from the counterfactual distribution is based on the direct effect of X 3 on Y 2 . We remark that the causal model we consider was first introduced by Oberst and Sontag (2019) and called the Gumbel-Max Structural Causal Model . Our insight here is that counterfactuals from this model can be used as an effective data augmentation method for machine translation. Given the above procedure to replace phrases, we propose a new data augmentation method, shown in Algorithm 2. The algorithm takes an input pair of sequences ( X , Y ) and loops through every phrase X i  X . At each iteration with probability c it replaces the phrase pair ( x i , y j ) with ( x i ,  y j ) . We introduce a special [MASK] token (Devlin et al., 2018) to represent a removed phrase for parameterizing both p ( X i |X  i ) and p ( Y j |X , Y  j ) as: Eq. 2 only requires monolingual datasets, which are abundant. On the other hand, Eq. 3 requires parallel corpora to train. We parameterize Eq. 3 using a variant of the translation language model (Lample and Conneau, 2019). The main difference is that only phrases in target sequences are masked, whereas Lample and Conneau (2019) mask both source and target tokens, with the goal of learning bilingual relations. Another difference is that a phrase with consecutive tokens is masked, while masked tokens in Lample and Conneau (2019) are not necessarily consecutive. To better tackle unknown and rare tokens, we adopt BPE to segment phrases into tokens. As the number of tokens is undetermined during generation, we use a sequence-to-sequence Transformer model (Vaswani et al., 2017) to encode inputs and decode tokens one by one until a special end-of-sequence symbol is encountered. More specifically, given a sequence of N tokens ( t 1 , ..., t N ) , the sequence contains a special [MASK] token signifying a masked phrase. Each token t i is first projected into its embedding e t i , which is a sum of its token embedding, position embedding, and language embedding, inspired by XLM (Lample and Conneau, 2019). Then, a Transformer encoder is applied to encode the tokens into their hidden representations H  RN  o (where o denotes the hidden size), i.e. H = Encoder ( e t 1 , ..., e t N ) . The hidden representation of [MASK], h [MASK]  R o , is fed into a Transformer decoder to predict the tokens of the masked phrase. We learn our models p  1 ( X i |X  i ) , p  2 ( Y j |X , Y  j ) by maximizing the following objectives: E XD (cid:2) E i  Uniform (1 ,..., |X| ) [log p  1 ( X i |X  i )] (cid:3) and E ( X , Y ) S (cid:2) E j  Uniform (1 ,..., |Y| ) [log p  2 ( Y j |X , Y  j )] (cid:3) . We categorize previous work on data augmentation for NMT into two classes, word replacement and backtranslation . Word replacement. WordDropout (Sennrich et al., 2016) randomly zeros out word embeddings in order to introduce noises. BPEDropout (Provilkov et al., 2020) stochastically corrupts the segmentation procedure of BPE, leading to different subword segmentations with the same BPE vocabulary. RAML (Norouzi et al., 2016) applies a reward-augmented maximum likelihood objective, which essentially augments target sequences with sequences sampled based on metrics, such as edit distance and BLEU score (Wang et al., 2018). SwitchOut (Wang et al., 2018) extends RAML, augmenting both source and target sequences by randomly replacing words with noisy words sampled from a uniform distribution. These works do not take context and alignment into account. TDA (Fadaee et al., 2017) first uses two uni-directional language models to replace a word in the source sequence, before replacing the corresponding word based on a bilingual lexicon. TDA does not consider contexts in target sequences and relies on a high-quality bilingual lexicon. SCDA (Gao et al., 2019) uses a soft augmentation approach, where Dataset # Sequences # Words # Chars News Commentary 0.46M 10.05M 63.96M News Crawl 2010 6.8M 0.14B 0.83B Table 1: The statistics of the monolingual datasets. the one-hot representation of a word is replaced by a soft distribution of words given by a language model. DADA (Cheng et al., 2019) uses gradient information to generate adversarial sequences for more robust NMT. AdvAug (Cheng et al., 2020) extends DADA, where embeddings of virtual sequences are sampled from an adversarial distribution for augmentation. SCDA and AdvAug ignore the alignment information, thereby breaking the symmetry of source and target sequences. While DADA takes both context and alignment into account, it replaces multiple words in source and target sequences simultaneously, which risks generating unnatural sequences. In this paper, we utilize both alignment and contextual information to sequentially replace aligned phrases for better performance. Backtranslation. The idea of backtranslation dates back to statistical machine translation (Goutte et al., 2009; Bojar and Tamchyna, 2011). Senrich et al. (2016) use backtranslation, where monolingual sequences in the target language are translated into the source language, and obtain substantial improvements on the WMT and IWSLT tasks. Currey et al. (2017) apply backtranslation to low-resource languages, finding that even low-quality translations due to limited parallel corpora are beneficial. He et al. (2016) propose a dual learning framework, where the primal task (source-to-target translation) and the dual task (target-to-source translation) teach each other through a reinforcement learning process until convergence. Edunov et al. (2018) scale backtranslation to millions of monolingual data and obtain state-of-the-art performance on WMT'14 English  German. Xia et al. (2019) use a two-step pivoting method for improving backtranslation on low-resource languages. We show that TCWR can be used together with backtranslation and obtain further improvements. We now describe the improvements with the data augmentation based on TCWR. We use the monolingual training data, including News Commentary and News Crawl 2010, provided by WMT'18, for Eq. 2, while the training set of each language pair is used for Eq. 3. The statistics of the monolingual and parallel corpora are summarized in Table 1 and 2, respectively. To reduce memory overhead, we train a shared language model for Eq. 2 and 3, i.e.  1 and  2 are tied. A language model is trained for each language pair to avoid performing multilingual NMT for a fair comparison with baselines, as jointly training a single model for several language pairs has been shown to be effective for both low-resource and high-resource languages (Aharoni et al., 2019). Therefore, we pre-train four models for En-Tr, EnDe, En-Vi and En-Fr, respectively. The encoder and decoder are composed of six layers. The encoder is initialized with XLM (Lample and Conneau, 2019) pre-trained with the masked language model, while the decoder is randomly initialized. The input-output embeddings are tied for reducing the size of the model (Press and Wolf, 2016). To achieve faster convergence, we apply PreNorm (Nguyen and Salazar, 2019) for getting rid of the warm-up stage of Transformer. The learning rate is set to 1e-5 and is linearly decayed with more training steps. The hidden size o is set to 1024. Same as BERT, the maximum sequence size is set to 512. We use LAMB (You et al., 2019) as the optimizer. GELU (Hendrycks and Gimpel, 2016) is used as the activation function. 16 sequences are used at each pre-training step. We train the masked language model for 50% of the time and the left time is used for training the translation language model. After pre-training, we use the pre-trained models to perform data augmentation on training data. Then, the augmented data are combined with training data for training NMT models. We use fairseq 2 to implement the NMT models. The vocabularty size is 37K. Six encoder and decoder layers are applied. The hidden size is set to 1024. 16 self-attention heads are employed. We use Adam as the optimizer. The learning rate is initially set to 1e-7 and is gradually increased to 5e-4 with 4K warm-up steps, before applying linear decay. Dropout is set to 0.3. Label smoothing with the smoothing factor 0.1 is used. For decoding, we use beam search, and the beam size is set to 12. SacreBLEU (Post, 2018) is used as the metric. We study the effect of pre-training steps on machine translation quality. We use the language models at different pre-training steps and evaluate these models on the development set of the WMT'18 English  Turkish task. The results are shown in Figure 3. The BLEU score improves with more pre-training steps and peaks at around 110K steps. We do not observe better performance with more pre-training steps, as the models become more overfitted on the training sets. We plot the learning curves of the language model for En-Tr with/without XLM initialization. As shown in Figure 4, the model with the XLM initialization converges faster and better compared to the model with the random initialization. As XLM is trained using the masked language model objective on large-scale monolingual data, we draw the conclusion that large-scale pre-training can improve downstream language model pre-training tasks. We further evaluate the models on the development set of the WMT'18 English  Turkish task. The model with the XLM initialization also performs better (17.49 BLEU) compared to its counterpart (16.68 BLEU). Thus, the model with the XLM initialization can also generate better data for improving NMT. As shown in Figure 5, we vary the sampling probability in Algorithm 2 and evaluate on the development set of the WMT'18 English  Turkish task. We observe that the BLEU score is maximized with sampling probability 0.2. The BLEU scores decrease with larger sampling probabilities. components. We randomly choose a phrase with uniform distribution to replace the original phrase for ablating source and target contexts. For ablating alignment, we randomly choose a position in the target sequence instead of following the alignment given by pialign. We also study removing  g k in Eq. 1. Since  g k comes from the abduction step, which encodes the information from the original pair ( X , Y ) , Eq. 1 encourages the model to sample a new pair that is similar to the original pair. Therefore, the model without  g k collapses to a probabilistic approach that directly samples phrases from the translation language model, disregarding the information from the original pair. The results are shown in Table 3. We observe that ablating the source context, target context and alignment are negative for translation quality, demonstrating the necessity of considering all these components for data augmentation. The result of ablating  g k shows the effectiveness of incorporating the original information from ( X , Y ) . We evaluate the algorithms on WMT'17 English  German (En-De), WMT'18 English  Turkish (En-Tr) and IWSLT'15 English  Vietnamese (En-Vi). As shown in Table 2, En-Tr and En-Vi are two low-resource language pairs, while En-De is a high-resource language pair. For En-Tr, we use newstest17 for validation and newstest18 for testing. For En-De, we use new-stest16 for validation and newstest17 for testing. For En-Vi, we use the TED tst2012 for validation and the TED tst2013 for testing. We compare TCWR with six baselines, WordDropout, BPEDropout, SwitchOut, SCDA, TDA and DADA. For WordDropout and BPEDropout, we perform a range search on its dropout probability from 0 to 1 and select the best one on de-Method En  Tr En  De En  Vi Baseline 15.35 27.54 31.66 +TCWR 17.38 29.37 33.76 +BT 19.24 29.19 33.38 +BT +TCWR 20.19 30.26 35.72 Table 5: The BLEU scores on the testing sets with backtranslation and TCWR. velopment sets. Similarly, we choose the temperature with the highest score on development sets for SwitchOut. For SCDA, we search the replacing probability and set it to 0.15. We follow the official implementation 3 of TDA. We reuse the hyperpa-rameters from Cheng et al. (2019) for DADA. The results on three language pairs are shown in Table 4. Compared to the baseline with no data augmentation, TCWR yields improvements of 2.03, 1.63 and 1.79 BLEU for En-Tr, En-De and En-Vi, respectively. TCWR also outperforms the other augmentation methods, which further con-firms the effectiveness of considering source context, target context, and alignment for NMT data augmentation. Besides, these results demonstrate that TCWR brings consistent improvements to both low-resource and high-resource language pairs. As backtranslation is a widely-used data augmentation method by utilizing monolingual data to generate new parallel pairs, we show how TCWR can be used with backtranslation. To perform backtranslation, we use the monolingual sequences from News Crawl 2017, News Crawl 2010 and VNTC 4 for EnTr, En-De and En-Vi, respectively. Then we perform data augmentation on both training data and backtranslated data. As shown in Table 5, TCWR improves upon backtranslation, demonstrating that TCWR and backtranslation are not mutually exclusive, and TCWR can enhance the performance of backtranslation. Noisy or non-standard input text (e.g. text with spelling errors and code switching) can cause significant degradation in most NMT systems. We use the WMT'19 English  French robustness dataset for evaluating translation robustness. As the parallel pairs are scarce for this task, we com-3 https://github.com/marziehf/ DataAugmentationNMT 4 https://github.com/duyvuleo/VNTC En: Kosovo is taking a hard look at its privatisation process in light of recurring [complaints / problems ]. bine its training data with the English  French pairs from Europarl-v7. The models are validated on the development set of the MTNT dataset and tested on the released test set of the WMT'19 robustness task. The results are shown in Table 7. We observe that TCWR outperforms the baseline without any data augmentation or with the other methods. If we regard the task as adapting from the source dataset with clean text (Europarl-v7) to the target dataset with noisy text (WMT'19 robustness dataset), TCWR helps this adaptation via enlarging training examples with language models trained using noisy and non-standard text. We thereby conclude that TCWR can improve NMT robustness. As shown in Table 6, we perform a case study of TCWR. We observe that TCWR can reasonably substitute words in source sequences based on contexts and modify corresponding target words, which demonstrates the benefits of considering both context and alignment for augmentation. We proposed a data augmentation method for NMT, which introduces a causal inductive bias that takes both context and alignment into account. The method was shown to improve the performance of translation, backtranslation and translation robustness on four NMT benchmarks. We would like to thank the anonymous reviewers for their insightful comments. We also thank Chris Dyer and Jiatao Gu for helpful discussions.
objective abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain result objective objective method abstain result objective method method result abstain objective other other other other other other other abstain other method other other other other other other method other objective objective method other other other method objective method other other other other other method abstain objective objective method abstain other other method other method other other method other other other other other other other other other abstain method objective method method other method method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain other objective other other other other other other other other other other other other other other method other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain result result abstain objective abstain other other