In this work, we propose a flow-adapter architecture for unsupervised NMT. It leverages normalizing flows to explicitly model the distributions of sentence-level latent representations, which are subsequently used in conjunction with the attention mechanism for the translation task. The primary novelties of our model are: (a) capturing language-specific sentence representations separately for each language using normalizing flows and (b) using a simple transformation of these latent representations for translating from one language to another. This architecture allows for unsupervised training of each language independently. While there is prior work on latent variables for supervised MT, to the best of our knowledge, this is the first work that uses latent variables and normalizing flows for unsupervised MT. We obtain competitive results on several unsupervised MT benchmarks. Recent advances in deep learning have boosted the development of neural machine translation (NMT). Typical NMT models leverage an encoder-decoder framework (Cho et al., 2014; Sutskever et al., 2014). However, NMT models have been shown to be data-hungry, as the number of parallel sentences significantly influences the performance (Zoph et al., 2016). Unfortunately, large-scale bilingual corpora are limited to a relatively small subset of languages (Al-Onaizan et al., 2002). In contrast to bilingual corpora, monolingual corpora are much easier to obtain. Unsupervised NMT, compared with supervised NMT, aims to train a model without parallel data. Some early works (Irvine and Callison-Burch, 2016; Sennrich et al., 2016b; Cheng et al., 2016) used monolingual corpora to boost performance when parallel data is not abundant. Lample et al. (2018a) and Artetxe et al. (2018) explored the possibility of training a model relying only on mono-Figure 1: Inference pipeline of proposed flow-adapter based model for source-to-target translation. The decoder also uses the attentional input (shown as the gray arrow between the encoder and the decoder). lingual corpora. They both leveraged a shared-encoder architecture in order to generate universal representations, trained with techniques such as initial word-by-word translation through bilingual dictionaries (Lample et al., 2018b; Artetxe et al., 2017), denoising auto-encoding (DAE) (Vincent et al., 2008) and iterative back-translation (BT) (Hoang et al., 2018). However, Yang et al. (2018) argued that it is a bottleneck in such shared-encoder models to use a shared encoder that maps pairs of sentences of different languages to the same shared latent space. They proposed to use two independent encoders sharing part of their weights and achieved better results. But all of those aforementioned approaches trained the translation models almost from scratch (with only some prior knowledge in the pre-trained embeddings) and therefore it is hard to further advance their performance. models (Peters et al., 2018; Devlin et al., 2019), researchers have begun to explore the possibility of using pre-trained models for unsupervised NMT. Conneau and Lample (2019) extended the pre-training from a single language to multiple languages, referred to as cross-lingual pre-training. By using pre-trained cross-language models (XLMs) to initialize encoder and decoder, they achieved good unsupervised MT performance on multiple language pairs. In related work, Song et al. (2019) proposed masked sequence to sequence pre-training (MASS), which directly pre-trains a whole encoder-decoder model. stn et al. (2021) proposed a language-specific denoising-adapter architecture to increase the multilingual modeling capacity of the pre-trained model mBART (Liu et al., 2020) and used these adapters for multilingual unsupervised NMT. Although these adapters are trained with monolingual data only, the fine-tuning step relies on parallel data. Current NMT frameworks rely heavily on the attention mechanism (Bahdanau et al., 2015; Vaswani et al., 2017) to capture alignments. However, attention-based context vectors can fail to extract sufficiently accurate sentence-level semantics and thus result in incorrect translations or translation ambiguity (Tu et al., 2016; Zhang et al., 2016). To tackle this issue, several variational frameworks for modeling the translation process have been proposed (Zhang et al., 2016; Eikema and Aziz, 2019; Setiawan et al., 2020). These approaches incorporate sentence-level latent representations into NMT. A latent representation, in the context of this paper, is a fixed-size continuous vector from an unknown distribution that captures the semantics of a source sentence. The target sentence is then generated from this latent representation using a simple transformation along with the attention mechanism commonly found in transformer architectures. In this way, when the attention mechanism learns incorrect alignments, the latent representation plays a complementary role in guiding the translation. Prior work in this vein has only been conducted in supervised NMT. In this paper, we propose a flow-adapter architecture for unsupervised NMT. Similar to variational methods, we model the distribution of sentence-level representations. However, unlike variational methods, which model the distribution in an implicit way, we use a pair of normalizing flows to explicitly model the distributions of source and target languages. Secondly, different from some previous unsupervised NMT models that assume that the representations of source and target sentences share a common semantic space, we assume the representations are different because of language-specific characteristics. Hence they are modeled separately for each language. Subsequently a simple transformation converts source representations into target representations. This makes it possible to better capture sentence semantics in a language-specific manner. Lastly, instead of minimizing KL loss, the flows are directly trained by maximum likelihood estimation (MLE) of sentence-level latent representations. This gives the latent representations more flexibility. Our main contributions: (1) We propose a novel flow-adapter architecture. It uses normalizing flows to explicitly model the distributions of sentence-level representations and performs a latent representation transformation from source to target. To the best of our knowledge, this is the first work that uses latent variables and normalizing flows for unsupervised NMT. (2) Experiments show the validity and effectiveness of our flow-adapter architecture. It performs very well in unsupervised NMT on several language pairs on the Multi30K dataset. When additionally using pre-trained models, we achieve results competitive with the state of the art on WMT datasets, especially for en-fr (WMT'14) and en-ro (WMT'16). Normalizing flows (NFs) are a special type of deep generative model. Different from generative adversarial networks (GAN) (Goodfellow et al., 2014) and variational auto-encoding (VAE) (Kingma and Welling, 2014), NFs allow for not only sampling but also exact density estimation. Due to such desirable properties, in recent years, they have been successfully applied to fields such as image (Ho et al., 2019; Kingma and Dhariwal, 2018), audio (Esling et al., 2019; van den Oord et al., 2018) and video generation (Kumar et al., 2019). In addition to significant achievements in modeling continuous data, NFs have also been used for modeling discrete data, either by directly modeling the data in discrete space (Tran et al., 2019; Hesselink and Aziz, 2020) or by transforming the discrete data into continuous space (Ziegler and Rush, 2019; Tang et al., 2021). NFs transform between two distributions based on the following change-of-variables formula (we follow the introduction of (Dinh et al., 2015, 2017)): where z  p z ( z ) and x  p x ( x ) denote two vectors from a simple latent distribution p z ( z ) and a complex distribution of the observed data p x ( x ) , f  is an invertible and differentiable function (neu-ral network with parameters  ), f  ( z ) = x and det f  ( z )  z denotes the determinant of the Jacobian matrix of f  . The idea of NFs is to learn an f  such that f  and f  1  transform between the latent space p z ( z ) and the observed space p x ( x ) . Constructing a single arbitrarily complex invertible and differentiable function is usually cumbersome. Therefore, a generally adopted approach is to stack multiple transformations f i together, i.e., x = f  ( z ) = f K      f 1 ( z ) . Similarly, for the reverse direction we have z = f  1  ( x ) = f  1 1      f  1 K ( x ) , whose Jacobian matrix is efficient to compute. Here K denotes the number of sequential flows (e.g., K = 3 in Table 1). Normalizing flows are usually optimized by MLE of the parameters  , i.e., log p ( D|  ) = (cid:80) Nn =1 log p x ( x ( n ) |  ) , where N is the data size. By applying a variant of the change-of-variable formula in Equation (1), i.e., log p x ( x ) = log p z ( f  1  ( x )) + log (cid:12)(cid:12)(cid:12)(cid:12) det f  1  ( x )  x (cid:12)(cid:12)(cid:12)(cid:12) , the MLE objective can be reformulated as follows: log p ( D|  ) = N (cid:88) n =1 log p z ( f  1  ( x ( n ) ) |  ) + log (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) det f  1  ( x ( n ) )  x ( n ) |  (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) (2) 2.2 Latent-variable (variational) NMT Compared with standard encoder-decoder based NMT models, latent-variable (variational) approaches (Zhang et al., 2016; Eikema and Aziz, 2019; Ma et al., 2019; Calixto et al., 2019; Setiawan et al., 2020; Shu et al., 2020) additionally leverage latent random variables. Let x be a sentence from the source language and y be its translation in the target language. Then, the variational NMT framework introduces a continuous random latent variable z for the translation modeling, i.e., p ( y | z , x ) . With the introduction of z , the conditional probability p ( y | x ) can then be reformulated as follows: p ( y | x ) = (cid:90) z p ( y | z , x ) p ( z | x ) d z (3) In this way, z serves as a global semantic signal that is helpful to counteract incorrect alignments the model has learned and uses through attention. However, the integration of z poses challenges for inference. To address this problem, variational NMT adopts techniques from VAE (Kingma and Welling, 2014; Rezende et al., 2014), namely, neural approximation and the reparameterization trick. Neural approximation leverages a neural network to approximate the posterior distribution p ( z | x , y ) with q  ( z | x , y ) , where  denotes the parameters of the neural network. In most works, q  ( z | x , y ) is designed as a diagonal Gaussian N (  , diag (  2 )) , where the mean  and the variance  2 are parameterized with neural networks. Reparameterization means that the latent random variable z is parameterized as a function of the mean  and the variance  2 . In this way, the gradient with respect to the parameters  and  2 can be computed. The reparameterization of z is often carried out in a location-scale manner: z =  +    where   N (0 , 1) With these two techniques, the learning objective of variational NMT is the evidence lower-bound or ELBO of the conditional probability p ( y | x ) : L ( ,  ; x , y ) =  KL ( q  ( z | x , y ) || p  ( z | x )) + E q  ( z | x , y ) [log p  ( y | z , x )] (4) where p  ( z | x ) is the prior distribution modeled by a neural network and p  ( y | z , x ) is modeled by the decoder given the input source sentence x and the latent variable z . The KL term minimizes the discrepancy between the prior p  ( z | x ) and the posterior q  ( z | x , y ) . In the inference step, z can therefore be sampled from the prior, which only requires x instead of the posterior that requires both x and y . Although this variational framework leverages latent variables, which are helpful for translation, it still has some flaws: 1) training a variational NMT framework requires parallel corpora to construct the posterior q  ( z | x , y ) and such parallel corpora are not available for unsupervised MT; 2) the distribution family of the latent variables, e.g., p  ( z | x ) , is pre-defined, e.g., a Gaussian, which might restricts the advantage of using a complex posterior; 3) as variational NMT leverages z 1255 sampled from p  ( z | x ) for inference, an underlying assumption is that z should be the same whether only x is considered or both x and y are considered. In other words, this framework assumes z is language-agnostic, which might not be true since language-specific characteristics can influence the generation of z . In this work, we want to reap the benefits of introducing latent variables into unsupervised MT while at the same time avoiding the flaws of variational NMT we just discussed. Therefore, we propose a flow-adapter based framework that uses two NFs to explicitly model the distribution of the sentence-level latent representations of the source and target sentences. In this way, we can take account of multilinguality in unsupervised MT and make use of language-specific sentence-level representations. During the translation process, a latent code transformation is performed to transform the source-language representation into the target-language representation so that the decoder can leverage them to generate a better target-language sentence. We will first introduce the sentence-level representation as well as the latent code transformation in Section 3.1, followed by the description of the flow-adapter based framework for unsupervised MT in Section 3.2. As previously mentioned, variational methods such as (Zhang et al., 2016; Setiawan et al., 2020) assume that the semantics of the source sentence x and target sentence y are the same and thus the generated latent variable z is the same regardless of whether we only consider x or consider both x and y . Unsupervised NMT methods such as (Lample et al., 2018a; Conneau and Lample, 2019) similarly assume that a shared encoder maps source and target sentences into a shared latent space. In this work, however, we diverge from this assumption and follow Yang et al. (2018) in adopting the desideratum that the unique and internal characteristics of each language be respected. One could think that the semantics of a pair of sentences should theoretically be the same; but in reality, because of language-specific characteristics, the latent representations z obtained by an encoder can be different for source and target sentences. Differences in vocabulary, pragmatics and other linguistic properties all influence the generation of the latent representations. Therefore, we consider the latent representations from a different perspective as follows. We can view z x and z y as expressions of the sentence-level representations in two distinct languages based on the same semantics  where  is truly language-agnostic. z x and z y are obtained by applying parameter-free techniques such as pooling to the output of the encoder fed with source and target languages (see Section 3.2 for details). Modeling by NFs. For our unsupervised scenario, we propose to explicitly model the distributions of the sentence-level representations of both source and target sentences  i.e., p z x ( z x ) and p z y ( z y )  using NFs with K sequential flows: p z x ( z x ) = p  (  ) K (cid:89) i =1 (cid:12)(cid:12)(cid:12) (cid:12)(cid:12) det f ( i ) x ( z ( i ) )  z ( i ) (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)  1 (5) p z y ( z y ) = p  (  ) K (cid:89) i =1 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) det f ( i ) y ( z ( i ) )  z ( i ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)  1 (6) where p  (  ) is a base distribution, e.g., the standard normal distribution; f ( i ) x and f ( i ) y are the i th transformations for the source and target languages, respectively; and z ( i ) is the intermediate variable where we define z (1) =  and z ( K ) = z x or z y for notational convenience. The base distribution can be viewed as the true underlying semantic space, abstracting away from language specifics. Our transformation to the sentence-level representations is similar to (Li et al., 2020). They argued that BERT induces a non-smooth anisotropic semantic space of sentences, which can harm its accurate representation of semantic similarity. Therefore, they also used NFs to transform the anisotropic BERT sentence-level distribution to a standard Gaussian distribution that is smooth and isotropic and reported better performance on some sentence-level similarity tasks. By using this type of sentence-level representation, the semantics of sentences from different languages can therefore be aligned in a simple common space in an unsupervised way, which we show is effective for unsupervised MT. For simplicity, we denote the NFs for transforming the distributions of source and target sentence-level representations to the base distribution as mappings G ( z x   ) and G ( z y   ) . Because of the 1256 Figure 2: Top two diagrams: denoising auto-encoding for source and target sentences. invertibility property of NFs, these mappings are also invertible, and we have G (   z x ) = G  1 ( z x   ) and G (   z y ) = G  1 ( z y   ) . Latent Code Transformation. Inspired by AlignFlow (Grover et al., 2020), we consider the cross-domain transformation between z x and z y . In this way, we can formulate a language-specific latent code for the decoder. We formalize the cross-language latent code transformation from the source to the target language as follows: G ( z x  z y ) = G (   z y )  G ( z x   ) (7) The target-to-source latent code transformation is then the composition of G (   z x ) and G ( z y   ) . As G (   z y ) and G (   z x ) are the inverse mappings of G ( z y   ) and G ( z y   ) , we can easily obtain them with normalizing flows, such as realNVP (Dinh et al., 2017) and Glow (Kingma and Dhariwal, 2018). We also note that G ( z x  z y ) and G ( z y  z x ) are both invertible since they are compositions of two invertible mappings. Moreover, G ( z x  z y ) is the inverse of G ( z y  z x ) and vice versa (see Appendix A.1 for details). The general architecture is shown in Figure 1. The transformer architecture (Vaswani et al., 2017) is used for both encoder and decoder. We use source encoder/decoder to denote the encoder/decoder for encoding/generating the source-language sentence. Similarly, target encoder/decoder refer to the encoder/decoder encoding/generating the target-language sentence. The decoders work in an autoregressive way. Source flow and target flow are NFs for modeling the sentence-level latent representations of the source and target language, respectively, as introduced in Section 3.1. Encoding. The source encoder and the target encoder work in the same way; for brevity, we only describe the procedure of encoding the source sentence and how z x is generated. The source encoder takes the source sentence x = { x 0 ,    , x S } as input and generates the hidden representations { h 0 ,    , h S } . These hidden representations will be used as encoder-decoder attentional inputs. In addition, we use the hidden representations to generate a sentence-level representation for the source sentence by applying max-pooling and mean-pooling to the token-level representations. After that, we sum up the results with the CLS representation h 0 , which usually encodes some global information. Finally, we use a projection matrix W to project the resulting vector to a latent space. The output is referred to as z x , i.e., the sentence-level representation of the source sentence (see Appendix A.2 for equation and illustration). Cross-lingual Translation. We hypothesize that the decoder can better leverage language-specific latent representations (i.e., z x for the source decoder and z y for the target decoder) than indiscriminately using the same representational space for source and target, e.g., z x for the target decoder. Therefore, we propose to perform a latent code transformation for cross-language translation as shown in Figure 1. If the model is performing the translation in the source-to-target direction, the source flow first transforms the source latent representation z x into  , which is a vector in the 1257 semantic base space. Then the target flow transforms  back into z y , which is in the target latent representation space. Then z y is used in the target decoder for generating the target sentence. Denoising Auto-Encoding (DAE) and Back Translation (BT) Processes. The DAE reconstructs a sentence from its noised version. For inducing noise, we use the same strategy which is used by (Lample et al., 2018a) (For more details, please refer to Appendix A.3). Since we train the DAEs separately for source and target languages, hence we don't need a latent code transformation there. For BT, however, such a latent code transformation is performed twice; taking BT for the source language as an example: first in the source-to-target direction, then in the target-to-source direction as shown in Figure 2. Decoding. To enable the decoder to capture the global semantics and mitigate improper alignments, we use the procedure outlined in (Setiawan et al., 2020), and incorporate the latent representation z into the output of the last layer of the decoder { s 0 ,    , s T } : o i = (1  g i )  s i + g i  z (8) where g i =  ([ s i ; z ]) ,  (  ) is the sigmoid function,  denotes Hadamard product between two vectors, and o i is the logit vector used to generate a prediction at the i th position. The values in g i control the contribution of z to o i . In case the dimension of the latent representation does not match the dimension of the decoder output, a linear projection maps z to the desired dimension. Training. Our flow-adapter framework has three learning objectives: DAE, BT and MLE of the sentence-level representations. The description of DAE and BT is omitted here as they are well known from related work (Lample et al., 2018a; Artetxe et al., 2018). A single training iteration consists of a DAE step followed by a BT step as shown in Figure 2. MLE computation is integrated into the DAE step to calculate the likelihood of the sentence-level representations. Our MLE learning objective for the source monolingual dataset can be formulated as follows (similar for the target dataset, omitted): LMLE ( G ( z x   ) ) = E z  p z x [log p z x ( z )] (9) where p z x ( z ) = p  ( G ( z x   ) ( z )) (cid:12)(cid:12)(cid:12)(cid:12) det G ( z x   )  z x (cid:12)(cid:12)(cid:12)(cid:12) z x = z (10) by definition of the source NFs in Equation 5. E z  p z x is approximated via mini-batches of sentence-level latent representations generated by the encoder in the training process. By training the source flow and the target flow with this MLE loss, the flows can therefore transform between the language-specific latent space of the representations and the base semantic space. In this way, the latent code transformations, i.e., G ( z x  z y ) and G ( z y  z x ) can be constructed. Multi30K task1 dataset (Elliott et al., 2016, 2017). 1 This is a multi-modal dataset that has 30,000 images annotated with captions in English, German and French. Similar to (Lample et al., 2018a), we only use the caption of each image. The officially provided train, validation and test sets are used. We use this dataset as a small-scale test for validating the effectiveness of our methods. WMT datasets. 2 Our experiments are run with the settings that were used for XLM (Conneau and Lample, 2019). XLM uses the monolingual data from the WMT News Crawl datasets 3 . We report results on newstest2014 en-fr , newstest2016 en-de and newstest2016 en-ro . Preprocessing. We tokenize the sentences with the Moses script (Koehn et al., 2007). For the Multi30K dataset, we process it similar to Lample et al. (2018a). Specifically, the sentences are randomly divided into two parts. The source-language monolingual dataset is built from the source-language sentences in the first part and the target-language dataset from the second part. In this way, there will be no exact translations of any sentences in the datasets. For the WMT datasets, we use the preprocessing methods from (Conneau and Lample, 2019). For the English-Romanian dataset, we remove the diacritics as done by Sennrich et al. (2016a) to avoid their inconsistent usage in the Romanian part of the dataset. Metric & Performance. We use BLEU as metric (Papineni et al., 2002) for all our experiments. Although Artetxe et al. (2020) recommended to use 1 https://github.com/multi30k/dataset 2 http://www.statmt.org/ 3 https://github.com/facebookresearch/XLM/blob/main/get-data-nmt.sh 1258 Models en-de de-en en-fr fr-en de-fr fr-de baseline 11.87 19.31 16.52 19.24 11.03 8.36 3-scf 12.25 19.83 16.98 20.12 11.67 8.98 3-glow 11.91 20.14 16.86 19.55 11.49 8.61 Table 1: BLEU of our flow-adapter model for multilingual translation on Multi30K. unsupervised validation criteria for systematic tuning, we follow the setting of (Conneau and Lample, 2019; Song et al., 2019) and use the provided parallel validation sets for tuning hyperparameters. We report the results on the test sets of the models that achieve best performance on the validation sets. Pre-trained Embeddings & Models. We use the pre-trained MUSE 4 (Lample et al., 2018b) embeddings for the multilingual unsupervised MT experiment (Table 1). We also leverage pre-trained cross-lingual models in the experiment of shared & separate decoder(s) (Table 2). Specifically, XLM models from HuggingFace 5 (Wolf et al., 2020) are used to initialize the encoder. Moreover, we also incorporate our flow-adapter architecture directly into the codebase of the original implementation of XLM 6 for the WMT dataset experiment (Table 3). In this case, the encoder and decoder are both initialized with pre-trained models. Details of these models can be found in Appendix A.3. As Multi30K provides parallel test data for English, French and German, we first conduct experiments to show the multilingual translation ability of our flow-adapter models. The results are shown in Table 1. The baseline model (without flow-adapter architecture) is trained with only DAE loss, while the flow-adapter based models (3-scf and 3-glow) are additionally trained with MLE loss for the NFs. 3-scf (resp. 3-glow) is the baseline model with two realNVP NF models (Dinh et al., 2017) (resp. Glow NF models (Kingma and Dhariwal, 2018)) , each of which consists of 3 sequential flows. Each NF model is used to model the sentence-level represen-4 https://github.com/facebookresearch/MUSE 5 https://github.com/huggingface 6 https://github.com/facebookresearch/XLM tations of one specific language, and two NF models then construct a flow-adapter for that translation direction (as shown in Figure 1). The flow-adapter based models additionally perform the latent code transformation to generate a language-specific representation while the baseline model does not perform such a transformation. For this experiment, we use the pre-trained cross-lingual word embeddings (MUSE embeddings) and randomly initialize a shared encoder and a shared decoder for all three languages. It is worth noting that the training objective does not contain the iterative back-translation. For further research where there are far more languages accommodated, random online back-translation (ROBT) proposed by Zhang et al. (2020) could be considered. Table 1 shows improvements over all six translation directions by using the flow-adapter architecture. Notably, our 3-scf and 3-glow models achieve 19.83 and 20.14 BLEU scores, respectively, on de-en , which is 0.52 and 0.83 higher than the baseline model. Similar improvements can also be seen on other translation directions. Our 3-scf model has BLEU scores that are 0.46 to 0.88 higher than the baselines while our 3-glow model has BLEU scores that are 0.04 to 0.83 higher than the baselines. The overall improvements show that the flow-adapter can generate more suitable sentence-level representations by performing the latent code transformation, which is helpful for the decoder to capture the semantics and generate more suitable translations. We also find that the translation performance is closely related to the language pair and the translation direction for both the baseline models and flow-adapter models. Our models obtain very good performance on en-fr , with performances in both the en-fr or fr-en directions better by 16 BLEU points. For other language pairs (including en-fr ), there is always one direction showing better performance than the other. Specifically, de-en achieves more than 19 BLEU points compared with 12 points for en-de , and de-fr achieves more than 11 BLEU points compared with 8.5 for fr-de . We present the performance of our flow-adapter models under the shared-decoder and separate-decoder settings on Multi30K. For this experiment, the encoder is initialized with the pre-trained XLM model and fixed; the decoder parameters are ran-1259 Models en-de de-en en-fr fr-en baseline (shared decoder) 0.25 0.17 0.13 0.11 3-scf (shared decoder) 25.80 28.92 39.26 36.84 3-glow (shared decoder) 26.09 29.48 39.21 36.66 baseline (separate decoders) 27.54 28.97 39.17 36.27 3-scf (separate decoders) 28.24 30.63 39.64 36.45 3-glow (separate decoders) 28.79 30.45 39.31 36.29 UNMT (Lample et al., 2018a) 22.74 26.26 32.76 32.07 Table 2: BLEU of the flow-adapter models and unsupervised SOTA model, i.e., UNMT (Lample et al., 2018a), on Multi30K. domly initialized and then trained. We also report the performance of a previous SOTA model, i.e., UNMT (Lample et al., 2018a). 7 The results are shown in Table 2. First, we notice that the shared-decoder baseline model obtains very low BLEU scores. By checking the translation generated, we find the model only copies the input as translation. This phenomenon shows that this baseline, which does not perform the latent code transformation, cannot model two languages simultaneously well, and thus cannot generate translations in the desired language domains. However, by incorporating the flow-adapter, the models will no longer have this limitation. Both shared-decoder models, i.e., 3-scf and 3-glow, achieve very good performance on all translation pairs. For example, the 3-scf model obtains BLEU scores of 25.80, 28.92, 39.26 and 36.84 on en-de , de-en , en-fr and fr-en , which are much higher than the baseline. Compared with the shared-decoder scenario, the models under the separate-decoder setting do not suffer from the copying problem, because different decoders are used to specifically model and generate sentences in distinct language domains. The downside, however, is that using multiple decoders at the same time can substantially increase the number of trainable parameters. Within the separate-decoder models, the flow-adapter models generally perform better than the baseline model, with about 1 BLEU increase on en-de and de-en directions and relatively smaller improvements on en-fr and fr-en . Those improvements demonstrate that the model can benefit from the flow-adapter architectures as language-specific latent representations are used, thus advancing the translation quality. els generally perform better than the shared-decoder models. The separate-decoder baseline is much better than its counterpart as it avoids the copying problem. For the 3-scf flow-adapter models, we find that the separate-decoder model outperforms the shared-decoder model by 2.44, 1.71, 0.38 on en-de , de-en and en-fr . However, on fr-en , the shared-decoder model achieves a BLEU socre that is by 0.39 BLEU points better. A similar phenomenon can also be seen for the 3-glow model. We conjecture this is due to the similarity between languages. As English and French share common vocabulary, some common features can therefore be captured by a shared decoder, thus improving its generalization. Lastly, when compared with UNMT, our models show superiority, improving performance by more than 4 BLEU points in each direction. We attribute the improvements to the usage of the pre-trained model and incorporation of language-specific sentence-level representations obtained by our latent code transformation. We further integrate our flow-adapter architecture into the original implementation of XLM (Con-neau and Lample, 2019) and conduct experiments on the WMT datasets. To fully leverage the pre-trained models, we initialize both the encoder and decoder with XLM models and set them trainable. In contrast to the experiment in Section 4.4, a single shared decoder is used for this experiment, since the decoder is also initialized with the pre-trained model and has far more parameters compared with the randomly initialized transformer decoder we use in Section 4.4. We report the performance of the flow-adapter based models (5-scf and 5-1260 Models en-de de-en en-ro ro-en en-fr fr-en XLM (EMD + EMD) (Conneau and Lample, 2019) 21.30 27.30 27.50 26.60 29.40 29.40 XLM (MLM + MLM) (Conneau and Lample, 2019) 26.40 34.30 33.30 31.80 33.40 33.30 5-scf 26.50 32.63 34.11 31.69 35.77 33.72 5-glow 26.43 32.04 33.87 31.32 35.25 33.12 MASS (Song et al., 2019) 28.30 35.20 35.20 33.10 37.50 34.90 CSP and fine-tuning (Yang et al., 2020) 28.70 35.70 -37.90 34.50 Table 3: BLEU of the flow-adapter models (5-scf and 5-glow) and SOTA models on WMT datasets. glow 8 ) as well as the performance of the SOTA models, namely XLM, MASS and CSP. 9 The results are shown in Table 3. Noticeably, both of our flow-adapter models achieve remarkable performance on all language pairs. Compared with the results of XLM (EMD + EMD), which uses the pre-trained cross-lingual embeddings instead of pre-trained models, both 5-scf and 5-glow achieve overall better performance. For example, 3-scf achieves BLEU scores higher by 5.20, 5.33, 6.61, 5.09, 6.37 and 4.32 on en-de , de-en , en-ro , ro-en , en-fr and fr-en , respectively. While not being as good as 5-scf, 5-glow still shows superiority over XLM (EMD + EMD). These improvements can be contributed to (1) the usage of pre-trained models and (2) the introduction of the flow-adapter. We further compare our flow-adapter based models with XLM (MLM + MLM), which is also initialized with pre-trained models. We find the performance of x-en directions is consistently lower than en-x directions for our models except for ende . This pattern is not limited to our architecture but is consistently present in prior work. We, again, speculate this is relating to the complexity of languages as well as similarity between languages. We leave this finding for future investigation. Our flow-adapter based models, though achieving similar or relatively worse BLEU scores on de-en and ro-en compared with XLM (MLM + MLM), obtain higher scores on other directions, i.e., en-de and en-ro , suggesting that our models might be more helpful on specific translation directions, as the flow-adapter generates language-specific rep-8 Preliminary experiments showed that using 5 flows provides slightly better results than 3 flows for WMT as WMT has many more sentences than Multi30K and therefore more powerful generative models (by adding more intermediate flows) are needed to model the sentence-level representations. 9 We follow prior convention and compare directly with MASS and CSP even though dataset processing for MASS and CSP (e.g., filtering, sampling) are not strictly the same as for XLM. But the difference is small and results would not be much different as Yang et al. (2020) mentions. resentations. Lastly, 5-scf achieves scores by 2.37 and 0.42 better than XLM (MLM + MLM) on en-fr and fr-en . As in the other experiments, the improvement due to flow adapters seems to be related to the languages involved in that language pair and the translation directions. We would like to investigate this phenomenon in future research. Finally, out models are competitve with MASS and CSP, with only small differences in BLEU. In general, the experiments shows the validity and effectiveness of our flow-adapter architecture integrated into pre-trained models. In this work, we propose a novel flow-adapter architecture for unsupervised NMT. The flow-adapter employs a pair of NFs to explicitly model the distributions of the sentence-level representations. A latent code transformation is performed in translation, which enables the decoder to better capture the semantics of sentences in certain language domains. Through extensive experiments, we show the flow-adapter can improve multilingual translation ability. Moreover, it can alleviate the copying problem. By integrating the flow-adapter into pre-trained XLM models, we achieve results competitive to state-of-the-art models on WMT datasets. In the future, we would like to explore the possibility of pre-training the flow-adapter simultaneously when pre-training the language models so that the flows can learn more information. Moreover, we would like to extend normalizing flows to language generation. By using different flows for different languages, multilingual language generation of the same semantics can be performed. We are grateful to Alex Fraser and Alexandra Chronopoulou for their insightful input. This work was funded by the European Research Council (ERC #740516).
objective abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method method abstain abstain abstain abstain abstain objective abstain objective result abstain result other other other other method other other other other method other other other other other other other other other other other other other other other other other abstain abstain method abstain abstain method abstain method abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method method abstain abstain abstain method method abstain method other method method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result objective abstain abstain result abstain result objective objective abstain other other