We propose a generative model for a sentence that uses two latent variables, with one intended to represent the syntax of the sentence and the other to represent its semantics. We show we can achieve better disentanglement between semantic and syntactic representations by training with multiple losses, including losses that exploit aligned paraphrastic sentences and word-order information. We also investigate the effect of moving from bag-of-words to recurrent neural network modules. We evaluate our models as well as several popular pretrained embeddings on standard semantic similarity tasks and novel syntactic similarity tasks. Empirically, we find that the model with the best performing syntactic and semantic representations also gives rise to the most disentangled representations. 1 1 Introduction As generative latent variable models, especially of the continuous variety (Kingma and Welling, 2014; Goodfellow et al., 2014), have become increasingly important in natural language processing (Bowman et al., 2016; Gulrajani et al., 2017), there has been increased interest in learning models where the latent representations are disentangled (Hu et al., 2017). Much of the recent NLP work on learning disentangled representations of text has focused on disentangling the representation of attributes such as sentiment from the representation of content, typically in an effort to better control text generation (Shen et al., 2017; Zhao et al., 2017; Fu et al., 2018). In this work, we instead focus on learning sentence representations that disentangle the syntax and the semantics of a sentence. We are moreover interested in disentangling these representa-1 Code and data are available at github.com/ mingdachen/disentangle-semantics-syntax tions not for the purpose of controlling generation, but for the purpose of calculating semantic or syntactic similarity between sentences (but not both). To this end, we propose a generative model of a sentence which makes use of both semantic and syntactic latent variables, and we evaluate the induced representations on both standard semantic similarity tasks and on several novel syntactic similarity tasks. We use a deep generative model consisting of von Mises Fisher (vMF) and Gaussian priors on the semantic and syntactic latent variables (respec-tively) and a deep bag-of-words decoder that conditions on these latent variables. Following much recent work, we learn this model by optimizing the ELBO with a VAE-like (Kingma and Welling, 2014; Rezende et al., 2014) approach. Our learned semantic representations are evaluated on the SemEval semantic textual similarity (STS) tasks (Agirre et al., 2012; Cer et al., 2017). Because there has been less work on evaluating syntactic representations of sentences, we propose several new syntactic evaluation tasks, which involve predicting the syntactic analysis of an unseen sentence to be the syntactic analysis of its nearest neighbor (as determined by the latent syntactic representation) in a large set of annotated sentences. In order to improve the quality and disentanglement of the learned representations, we incorporate simple additional losses in our training, which are designed to force the latent representations to capture different information. In particular, our semantic multi-task losses make use of aligned paraphrase data, whereas our syntactic multi-task loss makes use of word-order information. Additionally, we explore different encoder and decoder architectures for learning better syntactic representations. Experimentally, we find that by training in this way we are able to force the learned representations to capture different information (as measured by the performance gap between the latent representations on each task). Moreover, we find that we achieve the best performance on all tasks when the learned representations are most disentangled. There is a growing amount of work on learning interpretable or disentangled latent representations both in machine learning (Tenenbaum and Freeman, 2000; Reed et al., 2014; Makhzani et al., 2015; Mathieu et al., 2016; Higgins et al., 2016; Chen et al., 2016; Hsu et al., 2017) and in various NLP applications, including sentence sentiment and style transfer (Hu et al., 2017; Shen et al., 2017; Fu et al., 2018; Zhao et al., 2018, inter alia ), morphological reinflection (Zhou and Neubig, 2017), semantic parsing (Yin et al., 2018), text generation (Wiseman et al., 2018), and sequence labeling (Chen et al., 2018). Another related thread of work is text-based variational au-toencoders (Miao et al., 2016; Bowman et al., 2016; Serban et al., 2017; Xu and Durrett, 2018). In terms of syntax and semantics in particular, there is a rich history of work in analyzing their interplay in sentences (Jurafsky, 1988; van Valin, Jr., 2005). We do not intend to claim that the two can be entirely disentangled in distinct representations. Rather, our goal is to propose modica of knowledge via particular multi-task losses and measure the extent to which this knowledge leads learned representations to favor syntactic or semantic information from a sentence. There has been prior work with similar goals for representations of words (Mitchell and Steedman, 2015) and bilexical dependencies (Mitchell, 2016), finding that decomposing syntactic and semantic information can lead to improved performance on semantic tasks. We find similar trends in our results, but at the level of sentence representations. A similar idea has been explored for text generation (Iyyer et al., 2018), where adversarial examples are generated by controlling syntax. Some of our losses use sentential paraphrases, relating them to work in paraphrase modeling (Wi-eting et al., 2016; Wieting and Gimpel, 2018). Deudon (2018) recently proposed a variational framework for modeling paraphrastic sentences, but our focus here is on learning disentangled representations. As part of our evaluation, we develop novel syntactic similarity tasks for sentence representations learned without any syntactic supervision. These evaluations relate to the broad range of work in unsupervised parsing (Klein and Manning, 2004) and part-of-speech tagging (Christodoulopoulos et al., 2010). However, our evaluations differ from previous evaluations in that we employ k -nearest-neighbor syntactic analyzers using our syntactic representations to choose nearest neighbors. There is a great deal of work on applying multitask learning to various NLP tasks (Plank et al., 2016; Rei, 2017; Augenstein and Sgaard, 2017; Bollmann et al., 2018, inter alia ) and, recently, as a way of improving the quality or disentanglement of learned representations (Zhao et al., 2017; Goyal et al., 2017; Du et al., 2018; John et al., 2018). Our goal is to extract the disentangled semantic and syntactic information from sentence representations. To achieve this, we introduce the vMF-Gaussian Variational Autoencoder (VGVAE). As shown in Figure 1, VGVAE assumes a sentence is generated by conditioning on two independent variables: semantic variable y and syntactic variable z . In particular, our model gives rise to the following joint likelihood p  ( x, y, z ) = p  ( y ) p  ( z ) p  ( x | y, z ) = p  ( y ) p  ( z ) TY t =1 p ( x t | y, z ) , where x t is the t th word of x , T is the sentence length, and p ( x t | y, z ) is given by a softmax over a vocabulary of size V . Further details on the parameterization are given below. To perform inference, we assume a factored posterior q  ( y, z | x ) = q  ( y | x ) q  ( z | x ) , as has been used in prior work ( Zhou and Neubig, 2017; Chen et al., 2018). Learning of VGVAE maximizes a lower bound on marginal log-likelihood: log p  ( x )  E y  q  ( y | x ) z  q  ( z | x ) [log p  ( x | z, y )  log q  ( z | x ) p  ( z )  log q  ( y | x ) p  ( y ) ] = E y  q  ( y | x ) z  q  ( z | x ) [log p  ( x | z, y )]  KL ( q  ( z | x ) k p  ( z ))  KL ( q  ( y | x ) k p  ( y )) def == ELBO (1) 3.1 Parameterizations VGVAE uses two distribution families in defining the posterior over latent variables, namely, the von Mises-Fisher (vMF) distribution and the Gaussian distribution. vMF Distribution. vMF can be regarded as a Gaussian distribution on a hypersphere with two parameters:  and  . R m is a normalized vector (i.e. k  k 2 = 1 ) defining the mean direction. R  0 is often referred to as a concentration parameter analogous to the variance in a Gaussian distribution. vMF has been used for modeling similarity between two sentences (Guu et al., 2018), which is particularly suited to our purpose here, since we will evaluate our semantic representations in the context of modeling paraphrases (See Sections 4.1 and 4.2 for more details). Therefore, we assume q  ( y | x ) follows vMF (   ( x ) ,   ( x )) and the prior p  ( y ) follows the uniform distribution vMF (  , 0) . With this choice of prior and posterior distribution, the KL ( q  ( y | x ) k p  ( y )) appearing in the ELBO can be computed in closed-form:   I m/ 2 (   ) I m/ 2  1 (   ) + ( m/ 2  1) log    ( m/ 2) log(2  )  log I m/ 2  1 (   )+ m 2 log  + log 2  log ( m 2 ) , (2) where I v is the modified Bessel function of the first kind at order v and (  ) is the Gamma function. We follow Davidson et al. (2018) and use an acceptance-rejection scheme to sample from vMF. N (   ( x ) , diag (   ( x ))) and that the prior p  ( z ) is N (0 , I d ) , where I d is an d  d identity matrix. Since we only consider a diagonal covariance matrix, the KL divergence term KL ( q  ( z | x ) k p  ( z )) can also be computed efficiently: 1 2(  X i log  i + X i  i + X i  2 i  d ) (3) Inference and Generative Models. The inference models q  ( y | x ) and q  ( z | x ) are two independent word averaging encoders with additional linear feedforward neural networks for producing  ( x ) and  ( x ) (or  ( x ) ). The generative model p  ( x | y, z ) is a feedforward neural network g  with the output being a bag of words. In particular, the expected output log-probability (the first term in Eq. 1) is computed as follows: E y  q  ( y | x ) z  q  ( z | x ) [log p  ( x | y, z )] = E y  q  ( y | x ) z  q  ( z | x ) " TX t =1 log exp g  ([ y ; z ]) x t P Vj =1 exp g  ([ y ; z ]) j # Where V is the vocabulary size, [; ] indicates concatenation, T is the sentence length and x t is the index of the t 'th word's word type. Recurrent Neural Networks. To facilitate better learning of syntax, we also consider replacing both the generative and inference models with RNN-based sequence models, rather than bag-of-words models. In this setting, the generative model p  ( x | y, z ) is a unidirectional long-short term memory network (LSTM; Hochreiter and Schmidhuber, 1997) and a linear feedforward neural network for predicting the word tokens (shown in Figure 2). The expected output log-probability is computed as follows: E y  q  ( y | x ) z  q  ( z | x ) [log p  ( x | y, z )] = E y  q  ( y | x ) z  q  ( z | x ) " TX t =1 log p  ( x t | y, z, x 1: t  1 ) # Where V is the vocabulary size, T is the sentence length and x t is the index of the t 'th word's word type. The inference model q  ( y | x ) is still a word averaging encoder, but q  ( z | x ) is parameterized by encoder encoder Figure 2: Diagram showing LSTM decoder that uses the semantic variable y and the syntactic variable z . a bidirectional LSTM, where we concatenate the forward and backward hidden states and then take the average. The output of the LSTM is then used as input to a feedforward network with one hidden layer for producing  ( x ) and  ( x ) (or  ( x ) ). In the following sections, we will introduce several losses that will be added into the training of our base model, which empirically shows the ability of further disentangling the functionality between the semantic variable y and the syntactic variable z . We attempt to improve the quality and disentanglement of our semantic and syntactic representations by introducing additional losses, which encourage y to capture semantic information and z to capture syntactic information. We elaborate on these losses below. Our first loss is a paraphrase reconstruction loss (PRL). The key assumption underlying the PRL is that for a paraphrase pair x 1 , x 2 , the semantic information is equivalent between the two sentences and only the syntactic information varies. To impose such constraints, PRL is defined as E y 2  q  ( y | x 2 ) z 1  q  ( z | x 1 ) [  log p  ( x 1 | y 2 , z 1 )]+ E y 1  q  ( y | x 1 ) z 2  q  ( z | x 2 ) [  log p  ( x 2 | y 1 , z 2 )] (4) That is, we swap the semantic variables, keep the syntactic variables, and attempt to reconstruct the sentences (shown in Figure 3). While instead of using a multi-task objective we could directly model paraphrases x 1 and x 2 as being generated by the same y (which naturally suggests a product-of-experts style posterior, as in Wu and Goodman (2018)), we found that for the purposes of our downstream tasks training with the multi-task loss gave superior results. Our second loss is a discriminative paraphrase loss (DPL). The DPL explicitly encourages the similarity of paraphrases x 1 , x 2 to be scored higher than the dissimilar sentences n 1 , n 2 (i.e., negative samples; see Sec. 5 for more details) by a given margin  . As shown in Figure 3, the similarity function in this loss only uses the semantic variables in the sentences. The loss is defined as max(0 ,   d ( x 1 , x 2 ) + d ( x 1 , n 1 ))+ max(0 ,   d ( x 1 , x 2 ) + d ( x 2 , n 2 )) (5) The similarity function we choose is the cosine similarity between the mean directions of the semantic variables from the two sentences: d ( x 1 , x 2 ) = cosine(   ( x 1 ) ,   ( x 2 )) (6) 4.3 Word Position Loss It has been observed in previous work that word order typically contributes little to the modelling of semantic similarity (Wieting et al., 2016). We interpret this as evidence that word position information is more relevant to syntax than semantics, at least as evaluated by STS tasks. To guide the syntactic variable to represent word order, we introduce a word position loss (WPL). Although our word averaging encoders only have access to the bag of words of the input, using this loss can be viewed as a denoising autoencoder where we have maximal input noise (i.e., an orderless representation of the input) and the encoders need to learn to reconstruct the ordering. For both word averaging encoders and LSTM encoders, WPL is parameterized by a three-layer feedforward neural network f (  ) with input from the concatenation of the samples of the syntactic variable z and the embedding vector e i at input position i ; we then attempt to predict a one-hot vector representing the position i . More specifically, we define WPL def == E z  q  ( z | x ) "  X i log softmax ( f ([ e i ; z ])) i # where softmax (  ) i indicates the probability at position i . KL Weight. Following previous work on VAEs (Higgins et al., 2016; Alemi et al., 2016), we attach a weight to the KL divergence and tune it based on development set performance. Negative Samples. When applying DPL, we select negative samples based on maximizing cosine similarity to sentences from a subset of the data. In particular, we accumulate k mini-batches during training, yielding a mega-batch S (Wieting and Gimpel, 2018). Then the negative samples are selected based on the following criterion: n 1 = argmax n S n 6 = x 2 cosine(   ( x 1 ) ,   ( n )) where x 1 , x 2 forms the paraphrase pair and the mega-batch size is fixed to k = 20 for all of our experiments. Since all of our models are trained from scratch, we observed some instabilities with DPL during the initial stages of training. We suspect that this is because the negative samples at these initial stages are of low quality. To overcome this issue, DPL is included starting at the second epoch of training so that the models can have a warm start. We subsampled half a million paraphrase pairs from ParaNMT-50M (Wieting and Gimpel, 2018) as our training set. We use SemEval semantic textual similarity (STS) task 2017 (Cer et al., 2017) as a development set. For semantic similarity evaluation, we use the STS tasks from 2012 to 2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016) and the STS benchmark test set (Cer et al., 2017). For evaluating syntactic similarity, we propose several evaluations. One uses the gold parse trees from the Penn Treebank (Marcus et al., 1993), and the others are based on automatically tagging and parsing five million paraphrases from ParaNMT-50M; we describe these tasks in detail below. For hyperparameters, the dimensions of the latent variables are 50. The dimensions of word embeddings are 50. We use cosine similarity as similarity metric for all of our experiments. We tune the weights for PRL and reconstruction loss from 0.1 to 1 in increments of 0.1 based on the development set performance. We use one sample from each latent variable during training. When evaluating VGVAE based models on STS tasks, we use the mean direction of the semantic variable y , while for syntactic similarity tasks, we use the mean vector of the syntactic variable z . Our baselines are a simple word averaging (WORDAVG ) model and bidirectional LSTM averaging (BLSTMAVG ) model, both of which have been shown to be very competitive for modeling semantic similarity when trained on paraphrases (Wieting and Gimpel, 2018). Specifically, WORDAVG takes the average over the word embeddings in the input sequence to obtain the sentence representation. BLSTMAVG uses the averaged hidden states of a bidirectional LSTM as the sentence representation, where forward and backward hidden states are concatenated. These models use 50 dimensional word embeddings and 50 dimensional LSTM hidden vectors per direction. These baselines are trained with DPL only. Additionally, we scramble the input sentence for BLSTMAVG since it has been reported benefi-cial for its performance in semantic similarity tasks (Wieting and Gimpel, 2017). We also benchmark several pretrained embeddings on both semantic similarity and syntactic similarity datasets, including GloVe (Pennington et al., 2014), 3 SkipThought (Kiros et al., 2015), 4 3 We use 300 dimensional Common Crawl embeddings available at nlp.stanford.edu/projects/glove 4 github.com/ryankiros/skip-thoughts semantic var. InferSent (Conneau et al., 2017), 5 ELMo (Peters et al., 2018), 6 and BERT (Devlin et al., 2018). 7 For GloVe, we average word embeddings to form sentence embeddings. For ELMo, we average the hidden states from three layers and then average the hidden states across time steps. For BERT, we use the averaged hidden states from the last attention block. As shown in Table 1, the semantic and syntactic variables of our base VGVAE model show similar performance on the STS test sets. As we begin adding multi-task losses, however, the performance of these two variables gradually diverges, indicating that different information is being captured in the two variables. More interestingly, note that when any of the three losses is added to the base VGVAE model (even the WPL loss which makes no use of paraphrases), the performance of the semantic variable increases and the performance of the syntactic variable decreases; 5 We use model V1 available at github.com/ facebookresearch/InferSent 6 We use the original model available at allennlp. org/elmo 7 We use bert-large-uncased available at github.com/ huggingface/pytorch-pretrained-BERT this suggests that each loss is useful in encouraging the latent variables to learn complementary information. Indeed, the trend of additional losses both increasing semantic performance and decreasing syntactic performance holds even as we use more than two losses, except for the single case of VGVAE + PRL + DPL, where the syntactic performance increases slightly. Finally, we see that when the bag-of-words VGVAE model is used with all of the multi-task losses (ALL), we observe a large gap between the performance of the semantic and syntactic latent variables, as well as strong performance on the STS tasks that outperforms all baselines. Using LSTM modules further strengthens the disentanglement between the two variables and leads to even better semantic performance. While using an LSTM encoder and a bag-of-words decoder is difficult to justify from a generative modeling perspective, we include results with this con-figuration to separate out the contributions of the LSTM encoder and decoder. So far, we have only confirmed empirically that the syntactic variable has learned to not capture semantic information. To investigate what the syntactic variable has captured, we propose several syntactic similarity tasks. In particular, we consider using the syntactic latent variable in calculating nearest neighbors for a 1-nearest-neighbor syntactic parser or part-of-speech tagger. We use our latent variables to define the similarity function in these settings and evaluate the quality of the output parses and tag sequences using several metrics. Our first evaluation involves constituency parsing, and we use the standard training and test splits from the Penn Treebank. We predict a parse tree for each sentence in the test set by finding its nearest neighbor in the training set based on the cosine similarity of the mean vectors for the syntactic variables. The parse tree of the nearest neighbor will then be treated as our prediction for the test sentence. Since the train and test sentences may differ in length, standard parse evaluation metrics are not applicable, so we use tree edit distance ( Zhang and Shasha, 1989) 8 to compute the distance between two parse tree without consider-8 github.com/timtadh/zhang-shasha Constituent Parsing (TED,  ) Constituent Parsing ( F 1 ,  ) POS Tagging (%Acc.,  ) GloVe 120.8 27.3 23.9 SkipThought 99.5 30.9 29.6 InferSent 138.9 28.0 25.1 ELMo 103.8 30.4 27.8 BERT 101.7 28.6 25.4 Random baseline 121.4 19.2 12.9 Upper bound performance 51.6 71.1 62.3 WORDAVG 107.0 25.5 21.4 BLSTMAVG 106.8 25.7 21.6 semantic var. To better understand the difficulty of this task, we introduce two baselines. The first randomly selects a training sentence. We calculate its performance by running it ten times and then reporting the average. We also report the upper bound performance given the training set. Since computing tree edit distance is time consuming, we subsample 100 test instances and compute the minimum tree edit distance for each sampled instance. Thus, this number can be seen as the approximated upper bound performance for this task given the training set. To use a more standard metric for these syntactic similarity tasks, we must be able to retrieve training examples with the same number of words as the sentence we are trying to parse. We accordingly parse and tag the five million paraphrase subset of the ParaNMT training data using Stanford CoreNLP (Manning et al., 2014). To form a test set, we group sentences in terms of sentence length and subsample 300 sentences for each sentence length. After removing the paraphrases of the sentences in the test set, we use the rest of the training set as candidate sentences for nearest neighbor search, and we restrict nearest neighbors to have the same sentence length as the sentence we are attempting to parse or tag, which allows us to use standard metrics like labeled F 1 score and tagging accuracy for evaluation. As shown in Table 2, the syntactic variables and semantic variables demonstrate similar trends across these three syntactic tasks. Interestingly, both DPL and PRL help to improve the performance of the syntactic variables, even though these two losses are only imposed on the semantic variables. We saw an analogous pattern in Table 1, which again suggests that by pushing the semantic variables to learn information shared by paraphrastic sentences, we also encourage the syntactic variables to capture complementary syntactic information. We also find that adding WPL brings the largest improvement to the syntactic variable, and keeps the syntactic information carried by the semantic variables at a relatively low level. Finally, when adding all three losses, the syntactic variable shows the strongest performance across the three tasks. In addition, we observe that the use of the LSTM encoder improves syntactic performance by a large margin and the LSTM decoder improves further, which suggests that the use of the LSTM decoder contributes to the amount of syntactic information represented in the syntactic variable. Among pretrained representations, SkipThought shows the strongest performance overall and ELMo has the second best performance in the last two columns. While InferSent performs worst in the first column, it gives reasonable performance for the other two. BERT performs 5 10 15 20 25 20 40 60 80 100 Constituent Parsing BestRandomALLALL + LSTM enc. To investigate the performance gap between the bag-of-words VGVAE and VGVAE with LSTM modules, in Figure 4 we plot the performance of our models and baselines as the length of the target sentence increases. We see that performance in all settings degrades as the sentences get longer. This may be due to the fact that the data is much sparser as sentence length increases (leaving fewer candidate nearest neighbors for prediction). We also see that above 4 words or so the performance gap between the bag-of-words VGVAE and VGVAE with LSTM modules becomes more and more obvious. This may be because the bag-of-words encoder has a harder time capturing syntactic information as sentence length increases. In addition, there is a slight improvement from using an LSTM decoder when the sentence length increases beyond 12 or so, which suggests that a bag-of-words decoder may struggle to capture certain parts of the syntactic information in the sentence, even when using an LSTM encoder. To qualitatively evaluate our latent variables, we find (via cosine similarity) nearest neighbor sentences to test set examples in terms of both the semantic and syntactic representations. We also find nearest neighbors of words (which we view as single-word sentences). We discuss the results of this analysis below. that the most similar words found by the syntactic variable share the same part-of-speech tags with the query words. For example, starting is close to getting and taking, even though these words are not semantically similar. Words retrieved according to the semantic variable, however, are more similar semantically, e.g., begin and starts. As another example, times is similar to words that are either related to descriptions of frequency (e.g., twice and often) or related to numbers (e.g., thousand, seven). As shown in Table 4, sentences that are similar in terms of their semantic variables tend to have similar semantics. However, sentences that are similar in terms of their syntactic variables are mostly semantically unrelated but have similar surface forms. For example, you 're gon na save her life . has the same meaning as you will save her . while having a similar syntactic structure to you 're gon na give a speech . (despite having very different meanings). As another example, although the semantic variable does not find a good match for i have much more colours at home ., which can be attributed to the limited size of candidate sentences, the nearest syntactic neighbor (you have a beautiful view from here .) has a very similar syntactic structure to the query sentence. In this paper we explored simple methods to disentangle syntax and semantics in latent representations of sentences. One goal was to measure the impact of simple decisions on the disentanglement of both the semantic and syntactic variables, even when restricting ourselves to simplified bag-of-words encoders. Due to the constrained nature of these bag-of-words models, we found that it was important to use different word embedding spaces for the semantic and syntactic encoders. In preliminary experiments, we experimented with the use of the same word embedding space but distinct feed-forward layers in the two latent variable encoders. However, this setting proved extremely difficult to achieve a disentanglement between syntax and semantics. Hence an important component of disentanglement with these bag-of-words encoders is the use of different word embedding spaces. starting syntactic: getting heading sitting chasing taking require trying sharing bothering pushing paying semantic: begin start stopping forward rising wake initial starts goes started again getting beginning area syntactic: engines certificate guests bottle responsibility lesson pieces suit bags vessel applications semantic: sector location zone fields rooms field places yard warehouse seats coordinates territory considered syntactic: stable limited odd scary classified concerned awful purple impressive embarrassing jealous semantic: thought assumed regard reasons wished understood purposes seemed expect guessed meant jokes syntactic: gentlemen photos finding baby missile dna parent shop murder science recognition sheriff semantic: funny humor prize stars cookie paradise dessert worthy smile happiness thrilled ideal kidding times syntactic: princess officer wounds plan gang ships feelings user liar elements coincidence degrees pattern semantic: twice later thousand pages seven every once often decade forgotten series four eight day time Table 3: Examples of the most similar words to particular query words using syntactic variable (first row) or semantic variable (second row). We also conducted experiments using LSTM encoders and decoders as recurrent neural networks are a natural way to capture syntactic information in a sentence. We found this approach to give us additional benefits for both disentangling semantics and syntax and achieving better results overall. Nonetheless, we find it encouraging that even when using bag-of-words encoders, our multi-task losses are able to achieve a separation as measured by our semantic and syntactic similarity tasks. We proposed a generative model and several losses for disentangling syntax and semantics in sentence representations. We also proposed syntactic similarity tasks for measuring the amount of disentanglement between semantic and syntactic representations. We characterized the effects of the losses as well as the use of LSTM modules on both semantic tasks and syntactic tasks. Our models achieve the best performance across both sets of similarity tasks when the latent representations are most disentangled. We would like to thank the anonymous reviewers, NVIDIA for donating GPUs used in this research, and Google for a faculty research award to K. Gimpel that partially supported this research.
objective result objective objective result abstain abstain method objective objective method method method objective result abstain objective result result other other other abstain objective other abstain other method abstain objective other objective other method method abstain method abstain method other abstain abstain abstain abstain method method abstain abstain method abstain method abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain other abstain abstain abstain abstain other method abstain method method method abstain method abstain method abstain method abstain abstain abstain method method method method method method abstain abstain method method method method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain method abstain method method method method abstain abstain result result abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method abstain abstain other method result result objective objective abstain result other