This work exploits translation data as a source of semantically relevant learning signal for models of word representation. In particular, we exploit equivalence through translation as a form of distributional context and jointly learn how to embed and align with a deep generative model. Our EMBEDALIGN model embeds words in their complete observed context and learns by marginalisation of latent lexical alignments. Besides, it embeds words as posterior probability densities, rather than point estimates, which allows us to compare words in context using a measure of overlap between distributions (e.g. KL divergence). We investigate our model's performance on a range of lexical semantics tasks achieving competitive results on several standard benchmarks including natural language inference, paraphrasing, and text similarity. Natural language processing applications often count on the availability of word representations trained on large textual data as a means to alleviate problems such as data sparsity and lack of linguistic resources (Collobert et al., 2011; Socher et al., 2011; Tu et al., 2017; Bowman et al., 2015). Traditional approaches to inducing word representations circumvent the need for explicit semantic annotation by capitalising on some form of indirect semantic supervision. A typical example is to fit a binary classifier to detect whether or not a target word is likely to co-occur with neighbouring words (Mikolov et al., 2013). If the binary classifier represents a word as a continuous vector, that vector will be trained to be discriminative of the contexts it co-occurs with, and thus words in similar contexts will have similar representations. The underlying assumption is that context (e.g. neighbouring words) stands for the meaning of the target word (Harris, 1954; Firth, 1957). The success of this distributional hypothesis hinges on the definition of context and different models are based on different definitions. Importantly, the nature of the context determines the range of linguistic properties the representations may capture (Levy and Goldberg, 2014b). For example, Levy and Goldberg (2014a) propose to use syntactic context derived from dependency parses. They show that their representations are much more discriminative of syntactic function than models based on immediate neighbourhood (Mikolov et al., 2013). In this work, we take lexical translation as indirect semantic supervision (Diab and Resnik, 2002). Effectively we make two assumptions. First, that every word has a foreign equivalent that stands for its meaning. Second, that we can find this equivalent in translation data through lexical alignments. 1 For that we induce both a latent mapping between words in a bilingual sentence pair and distributions over latent word representations. To summarise our contributions:  we model a joint distribution over sentence pairs that generates data from latent word representations and latent lexical alignments;  we embed words in context mining positive correlations from translation data;  we find that foreign observations are necessary for generative training, but test time predictions can be made monolingually;  we apply our model to a range of semantic natural language processing tasks showing its usefulness. 1 These assumptions are not new to the community, but in this work they lead to a novel model which reaches more applications. 4 expands on the relation to other uses of bilingual data for word representation. In a nutshell, we model a distribution over pairs of sentences expressed in two languages, namely, a language of interest L1 , and an auxiliary language L2 which our model uses to mine some learning signal. Our model, EMBEDALIGN , is governed by a simple generative story: 1. sample a length m for a sentence in L1 and a length n for a sentence in L2 ; 2. generate a sequence z 1 , . . . , z m of d dimensional random embeddings by sampling independently from a standard Gaussian prior; 3. generate a word observation x i in the vocabulary of L1 conditioned on the random embedding z i ; 4. generate a sequence a i , . . . , a n of n random alignmentseach maps from a position a j in x m 1 to a position j in the L2 sentence; 5. finally, generate an observation y j in the vocabulary of L2 conditioned on the random embedding z a j that stands for x a j . The model is parameterised by neural networks and parameters are estimated to maximise a lowerbound on log-likelihood of joint observations. In the following, we present the model formally (  2.1), discuss efficient training (  2.2), and concrete architectures (  2.3). Notation We use block capitals (e.g. X ) for random variables, lowercase letters (e.g. x ) for assignments, and the shorthand X m 1 for a sequence X 1 , . . . , X m . Boldface letters are reserved for deterministic vectors (e.g. v ) and matrices (e.g. W ). Finally, E [ f ( Z );  ] denotes the expected value of f ( z ) under a density q ( z |  ) . We model a joint distribution over bilingual parallel data, i.e., L1  L2 sentence pairs. An observation is a pair of random sequences h X m 1 , Y n 1 i , where a random variable X ( Y ) takes on values in the vocabulary of L1 ( L2 ). For ease of exposition, the length m ( n ) of each sequence is assumed observed throughout. The L1 sentence is generated one word at a time from a random sequence of latent embeddings Z m 1 , each Z taking on values in R d . The L2 sentence is generated one word at a time given a random sequence of latent alignments A n 1 , where A j  { 1 , . . . , m } is the position in the L1 sentence to which y j aligns. 2 For i  { 1 , . . . , m } and j  { 1 , . . . , n } the generative story is Z i  N ( 0 , I ) (1a) X i | z i  Cat( f ( z i ;  )) (1b) A j | m  U (1 /m ) (1c) Y j | z m 1 , a j  Cat( g ( z a j ;  )) (1d) and Figure 1 is a graphical depiction of our model. We map from latent embeddings to categorical distributions over either vocabulary using a neural network whose parameters are deterministic and collectively denote by  (the generative parameters ). The marginal likelihood of a sentence pair is shown in Equation (2). P  ( x m 1 , y n 1 | m, n ) = Z p ( z m 1 ) m Y i =1 P  ( x i | z i )  n Y j =1 m X a j =1 P ( a j | m ) P  ( y j | z a j )d z m 1 (2) Due to the conditional independences of our model, it is trivial to marginalise lexical alignments for any given latent embeddings z m 1 , but marginalising the embeddings themselves is intractable. Thus, we employ amortised mean field variational inference using the inference model q  ( z m 1 | x m 1 ) , m Y i =1 N ( z i | u i , diag( s i (cid:12) s i )) (3) where each factor is a diagonal Gaussian. We map from x m 1 to a sequence u m 1 of independent posterior 2 We pad L1 sentences with NULL to account for untranslatable L2 words (Brown et al., 1993). Instead, Schulz et al. (2016) generate untranslatable words from L2 contextan alternative we leave for future work. mean (or location) vectors, where u i ,  ( h i ;  ) , as well as a sequence s m 1 of independent standard deviation (or scale) vectors, where s i ,  ( h i ;  ) , and h m 1 = enc( x m 1 ;  ) is a deterministic encoding of the L1 sequence (we discuss concrete architectures in  2.3). All mappings are realised by neural networks whose parameters are collectively denoted by  (the variational parameters ). Note that we choose to approximate the posterior without conditioning on y n 1 . This allows us to use the inference model for monolingual prediction in absence of L2 data. Variational  and generative  parameters are jointly point-estimated to attain a local optimum of the evidence lowerbound (Jordan et al., 1999): log P  ( x m 1 , y n 1 | m, n )  m X i =1 E [log P  ( x i | Z i ); u i , s i ] + n X j =1 E   log m X a j =1 P ( a j | m ) P  ( y j | Z a j ); u m 1 , s m 1    m X i =1 KL [ N ( u i , diag( s i (cid:12) s i )) ||N ( 0 , I )] . The variational family is location-scale, thus we can rely on stochastic optimisation (Robbins and Monro, 1951) and automatic differentiation (Bay-din et al., 2015) with reparameterised gradient estimates (Kingma and Welling, 2014; Rezende et al., 2014; Titsias and Lazaro-Gredilla, 2014). Moreover, because the Gaussian density is an exponential family, the KL terms in (4) are available in closed-form (Kingma and Welling, 2014, Appendix B). The likelihood terms in the ELBO (4) require evaluating two softmax layers over rather large vocabularies. This makes training prohibitively slow and calls for efficient approximation. We employ an approximation proposed by Botev et al. (2017) termed complementary sum sampling (CSS), which we review in this section. Consider the likelihood term log P ( X = x | z ) that scores an observation x given a sampled embedding z we use serif font x to distinguish a particular observation from an arbitrary event x  X in the support. The exact class probability P ( X = x | z ) = exp( u ( z, x )) P x X exp( u ( z, x )) (5) requires a normalisation over the complete support. CSS works by splitting the support into two sets, a set C that is explicitly summed over and must include the positive class x , and another set N that is a subset of the complement set X \ C . We obtain an estimate for the normaliser X x C exp( u ( z, x )) + X x N  ( x ) exp( u ( z, x )) (6) by importanceor Bernoulli-sampling from the support using a proposal distribution Q ( X ) , where  ( x ) corrects for bias as N tends to the entire complement set. In this paper, we design C and N per training mini-batch: we take C to consist of all unique words in a mini-batch of training samples and N to consist of 10 3 negative classes uniformly sampled from the complement set X \ C , in which case  ( x ) = 10  3 |X \ C| . 3 CSS makes it particularly easy to approximate likelihood terms such as those with respect to L2 in Equation (4). Because those terms depend on a marginalisation over alignments, an approximation must give support to all words in the sequence y n 1 . With CSS this is extremely simple, we just need to make sure all unique words in y n 1 are in the set C which our mini-batch procedure does guarantee. Botev et al. (2017) show that CSS is rather stable and superior to the most popular softmax approximations. Besides being simple to implement, CSS also addresses a few problems with other approximations. To name a few: unlike importance sampling approximations, CSS converges to the exact softmax with bounded computation (it takes as many samples as there are classes). Unlike hierarchical softmax , CSS only affects training, that is, at test time we simply use the entire support instead of the approximation. Without a softmax approximation, inference for our model would take time proportional to O ( m  v x + m  v y + m  n ) where v x ( v y ) corresponds to the size of the vocabulary of L1 ( L2 ). The first term ( m  v x ) corresponds to projecting from m latent embeddings to m categorical distributions over the vocabulary of L1 . The second term ( m  v y ) corresponds to projecting the same m latent embeddings to m categorical distributions over the vocabulary of L2 . Finally, the third term ( m  n ) is due to marginalisation of alignments. 3 We sample uniformly from the complement set until we have 10 3 unique classes. We realise this operation outside the computation graph providing C and N as inputs to each training iteration, but a GPU-based solution is also possible. Note, however, that with the CSS approximation we drop the dependency on vocabulary sizes (as the combined sizes of C and N is an independent constant). Moreover, if inference is performed on GPU, the squared term ( m  n  m 2 ) is amortised due to parallelism. Thus, while training our model is somewhat slower than monolingual models of word representation, which typically run in O ( m ) , it is not at all impracticably slower. Here we present the neural network architectures that parameterise the different generative and variational components of  2.1. Refer to Appendix B for an illustration. Generative model We have two generative components, namely, a categorical distribution over the vocabulary of L1 and another over the vocabulary of L2 . We predict the parameter (event probabilities) of each distribution with an affine transformation of a latent embedding followed by the softmax nonlinearity to ensure normalisation: f ( z i ;  ) = softmax ( W 1 z i + b 1 ) (7a) g ( z a j ;  ) = softmax (cid:0) W 2 z a j + b 2 (cid:1) (7b) where W 1  R v x  d , b 1  R v x , W 2  R v y  d , b 2  R v y , and v x ( v y ) is the size of the vocabulary of L1 ( L2 ). With the approximation of  2.2, we replace the L1 softmax layer (7a) by exp (cid:0) z > i c x + b x (cid:1) normalised by the CSS estimate (6) at training, and similarly for the L2 softmax layer (7b). In that case, we have parameters for c x , c y  R d deterministic embeddings for x and y , respectivelyas well as bias terms b x , b y  R . Inference model We predict approximate posterior parameters using two independent transformations u i = M 1 h i + d 1 (8a) s i = softplus( M 2 h i + d 2 ) (8b) of a shared representation h i  R d x of the i th word in the L1 sequence x m 1 where M 1 , M 2  R d  d x are projection matrices, d 1 , d 2  R d are bias vectors, and the softplus nonlinearity ensures that standard deviations are non-negative. To obtain the deterministic encoding h m 1 , we employ two different architectures: (1) a bag-of-words (BOW) encoder, where h i is a deterministic projection of x i onto R d x ; and (2) a bidirectional (BIRNN) encoder, where h i is the element-wise sum of two LSTM hidden states ( i th step) that process the sequence in opposite directions. We use 128 units for deterministic embeddings, and 100 units for LSTMs (Hochreiter and Schmidhuber, 1997) and latent representations (i.e. d = 100 ). We start the section describing the data used to estimate our model's parameters as well as details about the optimiser. The remainder of the section presents results on various benchmarks. Training data We train our model on bilingual parallel data. In particular, we use parliament proceedings (Europarl-v7) (Koehn, 2005) from two language pairs: English-French and English-German. 4 We employed very minimal preprocessing, namely, tokenisation and lowercasing using scripts from MOSES (Koehn et al., 2007), and have discarded sentences longer than 50 tokens. Table 1 lists more information about the training data, including the English-French Giga web corpus (Bojar et al., 2014) which we use in  3.4. 5 Corpus Sentence pairs Tokens Europarl EN-FR 1 . Optimiser For all architectures, we use the Adam optimiser (Kingma and Ba, 2014) with a learning rate of 10  3 . Except where explicitly indicated, we  train our models for 30 epochs using mini batches of 100 sentence pairs;  use validation alignment error rate for model selection;  train every model 10 times with random Glorot initialisation (Glorot and Bengio, 2010) and report mean and standard deviation;  anneal the KL terms using the following schedule: we use a scalar  from 0 to 1 with additive steps of size 10  3 every 500 updates. This means that at the beginning of the training, we allow the model to overfit to the likelihood terms, but towards the end we are optimising the true ELBO (Bowman et al., 2016). It is also important to highlight that we do not employ regularisation techniques (such as batch normalisation, dropout, or L 2 penalty) for they did not seem to yield consistent results. Since our model leverages learning signal from parallel data by marginalising latent lexical alignments, we use alignment error rate to double check whether the model learns sensible word correspondences. Intrinsic assessment of word alignment quality requires manual annotation. For English-French, we use the NAACL English-French hand-aligned data ( 37 sentence pairs for validation and 447 for test) (Mihalcea and Pedersen, 2003). For English-German, we use the data by Pado and La-pata (2006) ( 98 sentence pairs for validation and 987 for test). Alignment quality is then measured in terms of alignment error rate (AER) (Och and Ney, 2000)an F-measure over predicted alignment links. For prediction we condition on the posterior means E [ Z m 1 ] which is just the predicted variational means u m 1 and select the L1 position for which P ( y j , a j | u m 1 ) is maximum (a form of approximate Viterbi alignment). We start by analysing validation results and selecting amongst a few variants of EMBEDALIGN . We investigate the use of annealing and the use of a bidirectional encoder in the variational approximation. Table 2 (3) lists  AER for EN-FR (EN-DE ) as well as accuracy of word prediction. It is clear that both annealing (systems decorated with subscript  ) and bidirectional representations improve the results across the board. In the rest of the paper we still investigate whether or not recurrent encoders help, but we always report results based on annealing. In order to establish baselines for our models we report IBM models 1 and 2 (Brown et al., 1993). In a nutshell, IBM models 1 and 2 both estimate the conditional P ( y j | x m 1 ) = P ma j =1 P ( a j | m ) P ( y j | x a j ) by marginalisation of latent lexical alignments. The only difference between the two models is the prior over alignments, which is uniform for IBM1 and categorical for IBM2. An important difference between IBM models and EMBEDALIGN concerns the lexical distribution. IBM models are parameterised with independent categorical parameters, while our model instead is parameterised by a neural network. IBM models condition on a single categorical event x a j , namely, the word aligned to. Our model instead conditions on the latent embedding z a j that stands for the word aligned to. In order to establish even stronger conditional alignment models, we embed the conditioning words and replace IBM1's independent parameters by a neural network (single hidden layer MLP). We call this model a neural IBM1 (or NIBM for short). Note that in an IBM model, the sequence x m 1 is never modelled, therefore we can condition on it without restrictions. For that reason, we also experiment with a bidirectional LSTM encoder and condition lexical distributions on its hidden states. Table 4 shows AER for test predictions. First observe that neural models outperform classic IBM1 by far, some of them even approach IBM2's performance. Next, observe that bidirectional encodings make NIBM much stronger at inducing good word-1015 to-word correspondences. EMBEDALIGN cannot catch up with NIBM, but that is not necessarily surprising. Note that NIBM is a conditional model, thus it can use all of its capacity to better explain L2 data. EMBEDALIGN , on the other hand, has to find a compromise between generating both streams of the data. To make that point a bit more obvious, Table 5 (6) lists accuracy of word prediction for EN-FR (EN-DE ). Note that, without sacrificing L2 accuracy, and sometimes even improving it, EMBEDALIGN achieves very high L1 accuracy. This still does not imply that induced representations have captured aspects of lexical semantics such as word senses. All this means is that we have induced features that are jointly good at reconstructing both streams of the data one word at time. Of course it is tempting to conclude that our models must be capturing some useful generalisations. For that, the next sections will investigate a range of semantic NLP tasks. The English lexical substitution task (LST) consists in selecting a substitute word for a target word in context (McCarthy and Navigli, 2009). In the most traditional variant of the task, systems are presented with a list of potential candidates and this list must be sorted by relatedness. Dataset The LST dataset includes 201 target words present in 10 sentences/contexts each, along with a manually annotated list of potential replacements. The data are split in 300 instances for validation and 1 , 710 for test. Systems are evaluated by Model cos KL RANDOM 30 . comparing the predicted ranking to the manual one in terms of generalised average precision (GAP) (Melamud et al., 2015). Prediction We use EMBEDALIGN to encode each candidate (in context) as a posterior Gaussian density. Note that this task dispenses with inferences about L2 . Each candidate is compared to the target word in context through a measure of overlap between their inferred densitieswe take KL divergence. We then rank candidates using this measure. Table 7 lists GAP scores for variants of EMBEDALIN (bottom section) as well as some baselines and other established methods (top section). For comparison, we also compute GAP by sorting candidates in terms of cosine similarity, in which case we take the Gaussian mean as a summary of the density. The top section of the table contains systems reported by Melamud et al. (2015) (RANDOM and SKIPGRAM ) and by Brazinskas et al. (2017) (BSG). Note that both SKIPGRAM (Mikolov et al., 2013) and BSG were trained on the very large ukWaC English corpus (Ferraresi et al., 2008). SKIPGRAM is known to perform remarkably well regardless of its apparent insensitivity to context (in terms of design). BSG is a close relative of our model which gives SKIPGRAM a Bayesian treatment (also by means of amortised variational inference) and is by design sensitive to context in a manner similar to EMBEDALIGN , that is, through its inferred posteriors. Our first observation is that cosine seems to outperform KL slightly. Others have shown that KL can be used to predict directional entailment (Vil-nis and McCallum, 2014; Brazinskas et al., 2017), since LST is closer to paraphrasing than to entailment directionality may be a distractor, but we 1016 Model MR CR SUBJ MPQA SST TREC MRPC SICK-R SICK-E SST14 W 2 VEC 77.7 79.8 90.9 88.3 79.7 83.6 72.5/81.4 0.80 78.7 0.65/0.64 NMT 64.7 70.1 84.9 81.5 -82.8 -/--0.43/0.42 EN 57.6 66.2 70.9 71.8 58.0 62.9 70.3/80.1 0.62 73.7 0.54/0.55 EN-FR 63.5 71.5 78.9 82.3 65.1 62.1 71.4/80.5 0.69 75.9 0.69/0.59 EN-DE 64.0 68.9 77.9 81.8 65.1 59.5 71.2/80.5 0.69 74.8 0.62/0.61 COMBO 66.7 73.1 82.4 84.8 69.2 67.7 71.8/80.7 0.73 77.4 0.62/0.61 Table 8: English sentence evaluation results: the last four rows correspond to the mean of 10 runs with EMBEDALIGN models. leave it as a rather speculative point. One additional point worth highlighting: the middle section of Table 7. EN BoW and EN BiRNN show what happens when we do not give EMBEDALIGN L2 supervision at training. That is, imagine the model of Figure 1 without the bottom plate. In that case, the model representations overfit for L1 word-by-word prediction. Without the need to predict any notion of context (monolingual or otherwise), the representations drift away from semantic-driven generalisations and fail at lexical substitution. Conneau et al. (2017) developed a framework to evaluate unsupervised sentence level representations trained on large amounts of data on a range of supervised NLP tasks. We assess our induced representations using their framework on the following benchmarks evaluated on classification  accuracy (MRPC is further evaluated on  F1) MR classification of positive or negative movie reviews; SST fined-grained labelling of movie reviews from the Stanford sentiment treebank (SST); TREC classification of questions into k -classes; CR classification of positive or negative product reviews; SUBJ classification of a sentence into subjective or objective; MPQA classification of opinion polarity; SICK-E textual entailment classification; MRPC paraphrase identification in the Microsoft paraphrase corpus; as well as the following benchmarks evaluated on the indicated correlation metric(s) SICK-R semantic relatedness between two sentences (  Pearson); SST-14 semantic textual similarity (  Pearson/Spearman). Prediction We use EMBEDALIGN to annotate every word in the training set of the benchmarks above with the posterior mean embedding in context. We then average embeddings in a sentence and give that as features to a logistic regression classifier trained with 5 -fold cross validation. 6 For comparison, we report a SKIPGRAM model (here indicated as W 2 VEC ) as well as a model that uses the encoder of a neural machine translation system (NMT) trained on English-French Europarl data. In both cases, we report results by Conneau et al. (2017). Table 8 shows the results for all benchmarks. 7 We report EMBEDALIGN trained on either EN-FR or EN-DE . The last line (COMBO ) shows what happens if we train logistic regression on the concatenation of embeddings inferred by both EMBEDALIGN models, that is, EN-FR and EN-DE . Note that these two systems perform sometimes better sometimes worse depending on the benchmark. There is no clear pattern, but differences may well come from some qualitative difference in the induced latent space. It is a known fact that different languages realise lexical ambiguities differently, thus representations induced towards different languages are likely to capture different generalisations. 8 As COMBO results show, the representations induced from different corpora are somewhat complementary. That same observation has guided paraphrasing models based on pivoting (Bannard and Callison-Burch, 2005). Once more we report a monolingual variant of EMBEDALIGN (indicated by EN ) in an attempt to illustrate how crucial the 6 http://scikit-learn.org/stable/ 7 In Appendix A we provide bar plots marked with error bars ( 2 standard deviations). 8 We also acknowledge that our treatment of German is likely suboptimal due to the lack of subword features, as it can also be seen in AER results. Word similarity benchmarks are composed of word pairs which are manually ranked out of context. For completeness, we also tried evaluating our embeddings in such benchmarks despite our work being focussed on applications where context matters. Prediction To assign an embedding for a word type, we infer Gaussian posteriors for all training instances of that type in context and aggregate the posterior means through an average (effectively collapsing all instances). To cover the vocabulary of the typical benchmark, we have to use a much larger bilingual collection than Europarl. Based on the results of  3.1, we decided to proceed with English-French only recall that models based on that pair performed better in terms of AER. Results in this section are based on EMBEDALIGN (with bidirectional variational encoder) trained on the Giga web corpus (see Table 1 for statistics). Due to the scale of the experiment, we report on a single run. We trained on Giga with the same hyperparam-eters that we trained on Europarl, however, for 3 epochs instead of 30 (with this dataset an epoch amounts to 183 , 000 updates). Again, we performed model selection on AER. Table 9 shows the results for several datasets using the framework of Faruqui and Dyer (2014a). Note that EMBEDALIGN was designed to make use of context information, thus this evaluation setup is a bit unnatural for our model. Still, it outperforms SKIPGRAM on 5 out of 13 benchmarks, in particular, on SIMLEX-999, whose relevance has been argued by Upadhyay et al. (2016). We also remark that this model achieves 0 . 25 test AER and 45 . 16 test GAP on lexical substitutiona considerable improvement compared to models trained on Europarl and reported in Tables 4 (AER) and 7 (GAP). Our model is inspired by lexical alignment models such as IBM1 (Brown et al., 1993), however, we generate words y n 1 from a latent vector representation z m 1 of x m 1 , rather than directly from the observation x m 1 . IBM1 takes L1 sequences as conditioning context and does not model their distribution. Instead, we propose a joint model, where L1 sentences are generated from latent embeddings. There is a vast literature on exploiting multilingual context to strengthen the notion of synonymy captured by monolingual models. Roughly, the literature splits into two groups, namely, approaches that derive additional features and/or training objectives based on pre-trained alignments (Klementiev et al., 2012; Faruqui and Dyer, 2014b; Luong et al., 2015; Suster et al., 2016), and approaches that promote a joint embedding space by working with sentence level representations that dispense with explicit alignments (Hermann and Blunsom, 2014; AP et al., 2014; Gouws et al., 2015; Hill et al., 2014). The work of Kocisky et al. (2014) is closer to ours in that they also learn embeddings by marginalising alignments, however, their model is conditionalmuch like IBM modelsand their embeddings are not part of the probabilistic model, but rather part of the architecture design. The joint formulation allows our latent embeddings to har-vest learning signal from L2 while still being driven by the learning signal from L1 in a conditional model the representations can become specific to alignment deviating from the purpose of well representing the original language. In  3 we show substantial evidence that our model performs better when using both learning signals. Vilnis and McCallum (2014) first propose to map words into Gaussian densities instead of point estimates for better word representation. For example, a distribution can capture asymmetric relations that 1018 a point estimate cannot. Brazinskas et al. (2017) recast the skip-gram model as a conditional variational auto-encoder. They induce a Gaussian density for each occurrence of a word in context, and for that their model is the closest to ours. Additionally, they estimate a Gaussian prior per word type thus representing both types and occurrences. Unlike our model, the Bayesian skip-gram is not trained generatively by reconstructing the data, but rather discriminatively by prediction of overlapping sets of neighbouring words. We have presented a generative model of word representation that learns from positive correlations implicitly expressed in translation data. In order to make these correlations surface, we induce and marginalise latent lexical alignments. Embedding models such as CBOW and skip-gram (Mikolov et al., 2013) are essentially speaking supervised classifiers. This means they depend on somewhat artificial strategies to derive labelled data from monolingual corporawords far from the central word still have co-occurred with it even though they are taken as negative evidence. Training our proposed model does not require a heuristic notion of negative training data. However, the model is also based on a somewhat artificial assumption: L1 words do not necessarily need to have an L2 equivalent and, even when they do, this equivalent need not be realised as a single word. We have shown with extensive experiments that our model can induce representations useful to several tasks including but not limited to alignment (the task it most obviously relates to). We observed interesting results on semantic natural language processing benchmarks such as natural language inference, lexical substitution, paraphrasing, and sentiment classification. We are currently expanding the notion of distributional context to multiple auxiliary foreign languages at once. This seems to only require minor changes to the generative story and could increase the model's disambiguation power dramatically. Another direction worth exploring is to extend the model's hierarchy with respect to how parallel sentences are generated. For example, modelling sentence level latent variables may capture global constraints and expose additional correlations to the model. We thank Philip Schulz for comments on an earlier version of this paper as well as the anonymous NAACL reviewers. One of the Titan Xp cards used for this research was donated by the NVIDIA Corporation. This work was supported by the Dutch Organization for Scientific Research (NWO) VICI Grant nr. 277-89-002.
abstain method result method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain result abstain objective objective abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain other method abstain other method abstain method other abstain method method other abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method method method abstain method method abstain abstain method method abstain method other method abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain other abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain other result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain method other objective other other other objective abstain other other other other other method method abstain abstain abstain abstain abstain result result abstain abstain abstain abstain other other other