Negation and uncertainty modeling are longstanding tasks in natural language processing. Linguistic theory postulates that expressions of negation and uncertainty are semantically independent from each other and the content they modify. However, previous works on representation learning do not explicitly model this independence. We therefore attempt to disentangle the representations of negation, uncertainty, and content using a Variational Autoencoder 1 . We find that simply supervising the latent representations results in good disentanglement, but auxiliary objectives based on adversarial learning and mutual information minimization can provide additional disentanglement gains. In formal semantics, negation and uncertainty are operators whose semantic functions are independent of the propositional content they modify (Cann, 1993a,b) 2 . That is, it is possible to form fluent statements by varying only one of these aspects while leaving the others the same. Negation, uncertainty, and content can thus be viewed as disentangled generative factors of knowledge and belief statements (see Figure 1). Disentangled representation learning (DRL) of factors of variation can improve the robustness of representations and their applicability across tasks (Bengio et al., 2013). Specifically, negation and uncertainty are important for downstream NLP tasks such as sentiment analysis (Benamara et al., 2012; Wiegand et al., 2010), question answering (Yatskar, 2019; Yang et al., 2016), and information extraction (Stenetorp et al., 2012). Disentangling nega-1 We make our implementation and data available at https://github.com/jvasilakes/ disentanglement-vae 2 Specifically, the propositional content can be represented by a variable, such as  p . tion and uncertainty can therefore provide robust representations for these tasks, and disentangling them from content can assist tasks that rely on core content preservation such as controlled generation (Logeswaran et al., 2018) and abstractive summarization (Maynez et al., 2020). Still, no previous work has tested whether negation, uncertainty, and content can be disentangled, as linguistic theory suggests, although previous works have disentangled attributes such as syntax, semantics, and style (Balasubramanian et al., 2021; John et al., 2019; Cheng et al., 2020b; Bao et al., 2019; Hu et al., 2017; Colombo et al., 2021). To fill this gap, we aim to answer the following research questions: RQ1: Is it possible to estimate a model of statements that upholds the proposed statistical independence between negation, uncertainty, and content? RQ2: A number of existing disentanglement objectives have been explored for text, all giving promising results. How do these objectives compare for enforcing disentanglement on this task? In addressing these research questions, we make the following contributions: 1. Generative Model: We propose a generative model of statements in which negation, uncertainty, and content are independent latent variables. Following previous works, we estimate this model using a Variational Autoencoder (VAE) (Kingma and Welling, 2014; Bowman et al., 2016) and compare existing auxiliary objectives for enforcing disentanglement via a suite of evaluation metrics. 2. Simple Latent Representations: We note that negation and uncertainty have a binary function (positive or negative, certain or uncer-tain). We therefore attempt to learn corresponding 1-dimensional latent representations for these variables, with a clear separation between each value. 3. Data Augmentation: Datasets containing negation and uncertainty annotations are relatively small (Farkas et al., 2010; Vincze et al., 2008; Jimnez-Zafra et al., 2018), resulting in poor sentence reconstructions according to our preliminary experiments. To address this, we generate weak labels for a large number of Amazon 3 and Yelp 4 reviews using a simple nave Bayes classifier with bag-of-words features trained on a smaller dataset of English reviews annotated for negation and uncertainty (Konstantinova et al., 2012) and use this to estimate our model. Details are given in Section 4.1.1. We note that, in contrast to other works on negation and uncertainty modeling, which focus on token-level tasks of negation and uncertainty cue and scope detection, this work aims to learn statement -level representations of our target factors, in line with previous work on text DRL. We here provide relevant background on negation and uncertainty processing, disentangled representation learning in NLP, as well as discussion of how this study fits in with previous work. Negation and uncertainty help determine the asserted veracity of statements and events in text (Saur and Pustejovsky, 2009; Thompson et al., 2017; Kilicoglu et al., 2017), which is crucial for downstream NLP tasks that deal with knowledge and belief. For example, negation detection has been shown to provide strong cues for sentiment analysis (Barnes et al., 2021; Ribeiro et al., 2020) and uncertainty detection assists with fake news detection (Choy and Chong, 2018). Previous works on negation and uncertainty processing focus on the classification tasks of cue identification and scope detection (Farkas et al., 2010) using sequence models such as conditional random fields (CRFs) (Jimnez-Zafra et al., 2020; Li and Lu, 2018), convolutional and recurrent neural networks (CNNs and RNNs) (Qian et al., 2016; Adel and Schtze, 2017; Ren et al., 2018), LSTMs (Fancellu et al., 2016; Lazib et al., 2019), and, most recently, transformer architectures (Khandelwal and Sawant, 2020; Lin et al., 2020; Zhao and Bethard, 2020). While these works focus mostly on learning local representations of negation and uncertainty within a sentence, we attempt to learn global representations that encode high-level information regarding the negation and uncertainty status of statements. There is currently no agreed-upon definition of disentanglement. Early works on DRL attempt to learn a single vector space in which each dimension is independent of the others and represents one ground-truth generative factor of the object being modeled (Higgins et al., 2016). Higgins et al. (2018) give a group-theoretic definition, according to which generative factors are mapped to independent vector spaces. This definition relaxes the earlier assumption that representations ought to be single-dimensional and formalizes the notion of disentanglement according to the notion of invariance. Shu et al. (2019) decompose the invariance requirement into consistency and restrictiveness, which describe specific ideal properties of the in-variances between representations and generative factors. In addition to independence and invariance, interpretability is an important criterion for disentanglement. Higgins et al. (2016) point out that while methods such as PCA are able to learn independent latent representations, because these are not representative of interpretable factors of variation, they are not disentangled . We therefore want our learned representations to be predictive of meaningful factors of variation. We adopt the 8381 x t n c u T Figure 2: Graph of the generative model. Previous works on DRL for text all use some form of supervision to enforce informativeness of the latent representations. Hu et al. (2017), John et al. (2019), Cheng et al. (2020b), and Bao et al. (2019) all use gold-standard labels of the generative factors, while other works employ similarity metrics (Chen and Batmanghelich, 2020; Balasubramanian et al., 2021). In contrast, our approach uses weak labels for negation and uncertainty generated using a classifier trained on a small set of gold-standard data. These previous works on text DRL all use a similar architecture: a sequence VAE (Kingma and Welling, 2014; Bowman et al., 2016) maps inputs to L distinct vector spaces, each of which are constrained to represent a different target generative factor via a supervision signal. We also employ this overall architecture for model estimation and use it as a basis for experimenting with existing disentanglement objectives based on adversarial learning (John et al., 2019; Bao et al., 2019) and mutual information minimization (Cheng et al., 2020b), described in Section 3.4. However, unlike these previous works, which learn high-dimensional representations of all the latent factors, we aim to learn 1-dimensional representations of the negation and uncertainty variables in accordance with their binary function. We describe our overall model in Section 3.1. Section 3.2 enumerates three specific desiderata for disentangled representations, and sections 3.3 and 3.4 describe how we aim to satisfy these desiderata. We propose a generative model of statements according to which negation, uncertainty, and content are independent latent variables. A diagram of our Figure 3: The proposed architecture corresponding to the LELBO + LINF objective (see Section 3.4). are given in Appendix A. We use a sequence VAE to estimate this model (Kingma and Welling, 2014; Bowman et al., 2016). Unlike a standard autoencoder, the VAE imposes a prior distribution on the latent representation space Z (usually a standard Gaussian) and replaces the deterministic encoder with a learned approximation of the posterior q  ( z | x ) parameterized by a neural network. In addition to minimizing the loss between the input and reconstruction, as in a standard AE, the VAE uses an additional KL divergence term to keep the approximate posterior close to the prior distribution. In our implementation, three linear layers map the final hidden state of a BiLSTM encoder to three sets of Gaussian distribution parameters (  ,  ), which parameterize the negation, uncertainty, and content latent distributions   { n, u, c } , respectively. Because we map each input to three distinct latent spaces, we include three KL divergence terms in the Evidence Lower BOund (ELBO) training objective, given in Equation (1). LELBO ( ,  ) =  E q  ( z | x ) (cid:104) log p  ( x | z ) (cid:105) + (cid:88)  { n,u,c }   KL (cid:104) q (  )  ( z (  ) | x ) || p ( z (  ) ) (cid:105) (1) where  denotes the encoder's parameters,  the 8382 decoder's parameters, p ( z (  ) ) is a standard Gaussian prior, and the   hyper-parameters weight the KL divergence term for each latent space   L . The latent representations z  are sampled from normal distributions defined by these parameters using the reparameterization trick (Kingma and Welling, 2014), i.e., z (  ) =  (  )   (  ) +   N ( 0 , I ) . The latent representations are then concatenated z = [ z ( n ) ; z ( u ) ; z ( c ) ] and used to initialize an LSTM decoder, which aims to reconstruct the input. A visualization of our architecture is given in Figure 3 and implementation details are given in Appendix E. We use 1-dimensional negation and uncertainty spaces and a 62-dimensional content space for a total latent size of 64. Notably, we do not supervise the content space, unlike previous works (John et al., 2019; Cheng et al., 2020b), which supervise it by predicting the bag of words of the input. Such a supervision technique would hinder disentanglement by encouraging the content space to be predictive of the negation and uncertainty cues. Therefore, in our model we define three latent spaces   { n, u, c } but use signals from only 2 target generative factors k  { n, u } . We aim to satisfy the following desiderata of disentangled representations put forth by previous works. 1. Informativeness : the representations should be predictive of the ground-truth generative factors (Higgins et al., 2016; Eastwood and Williams, 2018); 2. Independence : the representations for each generative factor in question should lie in independent vector spaces (Higgins et al., 2018); 3. Invariance : the mapping from the data to the representations should be invariant to changes in other generative factors (Higgins et al., 2018; Shu et al., 2019); The following sections detail how our model enforces these desiderata. Following Eastwood and Williams (2018), we measure the informativeness of a representation by its ability to predict the corresponding generative factor. Similar to previous works on DRL for text (John et al., 2019; Cheng et al., 2020b), we train supervised linear classifiers 5 on each latent space and back-propagate the prediction error. Thus, in addition to the ELBO objective in Equation (1), we define informativeness objectives for negation and uncertainty. where y ( k ) is the true label for factor k ,  y ( k ) is the classifier's prediction,  ( k ) are the parameters of this classifier, and BCE is the binary cross-entropy loss. 1. Informativeness (INF): This is based on the hypothesis that if negation, uncertainty, and content are independent generative factors, the informativeness objective described in Section 3.3 will be sufficient to drive independence and invariance. This approach was found to yield good results on disentangling style from content by Balasubramanian et al. (2021). 2. Adversarial (ADV): The latent representations should be predictive of their target generative factor only. Therefore, inspired by John et al. (2019), we train additional adversarial classifiers on each latent space that try to predict the values of the non-target generative factors, while the model attempts to structure the latent spaces such that the predictive distribution of these classifiers is a non-predictive as possible. 3. Mutual-information minimization (MIN): A natural measure of independence between two variables is mutual information (MI). Therefore, this objective minimizes an upper-bound estimate of the MI between each pair of latent spaces, following (Cheng et al., 2020a,b; Colombo et al., 2021). 5 Implemented as single-layer feed-forward neural networks with sigmoid activation. Adversarial Objective. The adversarial objective (ADV) consists of two parts: 1) adversarial classifiers which attempt to predict the value of all non-target factors from each latent space; 2) a loss that aims to maximize the entropy of the predicted distribution of the adversarial classifiers. For a given latent space  , a set of linear classifiers predict the value of each non-target factor k  =  , respectively, and we compute the binary cross-entropy loss for each. Where  ( ,k ) are the parameters of the adversarial classifier predicting factor k from latent space  and  y ( ,k ) is the corresponding prediction. For example, we introduce two such classifiers for the content space  = c , one to predict negation and one to predict uncertainty, k  { n, u } . Importantly, the prediction errors of these classifiers are not back-propagated to the rest of the VAE. We impose an additional objective for each adversarial classifier, which aims to make it's predicted distribution as close to uniform as possible. We do this by maximizing the entropy of the predicted distribution (Equation (4)) and back-propagating the error, following John et al. (2019) and Fu et al. (2018). As the objective is to maximize this quantity, the total adversarial objective is The ADV objective aims to make the latent representations as uninformative as possible for nontarget factors. Together with the informativeness objective, it pushes the representations to specialize in their target generative factors. MI Minimization Objective. The MI minimization (MIN) objective focuses on making the distributions of each latent space as dissimilar as possible. We minimize the MI between each pair of latent spaces according to Equation (6). where  ICLUB (  i ;  j ) is the Contrastive Learning Upper-Bound (CLUB) estimate of the MI (Cheng et al., 2020a). Specifically, we introduce a separate neural network to approximate the conditional variational distribution p  (  i |  j ) , which is used to estimate an upper bound on the MI using samples from the latent spaces. The full model objective along with relevant hy-perparameters weights  is given in Equation (7). Our hyperparameter settings and further implementation details are given in Appendix E. L = LELBO +  INFLINF +  ADVLADV +  MINLMIN (7) In the sections that follow, we experiment with different subsets of the terms in the full objective and their effects on disentanglement. We train a model using only the ELBO objective as our disentanglement baseline. We describe our datasets, preprocessing, and data augmentation methods in Section 4.1. Section 4.2 describes our evaluation metrics and how these target the desiderata for disentanglement given in Section 3.2. We use the SFU Review Corpus (Konstantinova et al., 2012) as our primary dataset. This corpus contains 17,000 sentences from reviews of various products in English, originally intended for sentiment analysis, annotated with negation and uncertainty cues and their scopes. Many of the SFU sentences are quite long ( > 30 tokens), and preliminary experiments revealed that this results in poor reconstructions. We therefore took advantage of SFU's annotated statement conjunction tokens to split the multi-statement sentences into single-statement ones in order to reduce the complexity and increase the number of examples. Also to reduce complexity, we remove sentences > 15 tokens following previous work (Hu et al., 2017), resulting in 14,000 sentences. We convert all cue-scope annotations to statement-level annotations. Multi-level uncertainty annotations have been shown to be rather inconsistent and noisy, achieving low inter-annotator agreement compared to binary ones (Rubin, 2007). We therefore binarize the certainty labels following (Zerva, 2019). Despite the efforts above, we found the SFU corpus alone was insufficient for obtaining fluent reconstructions. We therefore generated weak negation and uncertainty labels for a large amount of additional Amazon and Yelp review data using two nave Bayes classifiers with bag-of-words (BOW) features 6 . These classifiers were trained on the SFU training split to predict sentence level negation and uncertainty, respectively. The Amazon and Yelp datasets fit the SFU data distribution well, being also comprised of user reviews, and have been used in previous works on text DRL with good results (John et al., 2019; Cheng et al., 2020b) 7 . Statistics for the combined SFU+Amazon dataset are summarized in Appendix C. In Appendix D, we provide a complementary evaluation on a combined SFU+Yelp dataset. Evaluating disentanglement of the learned representations requires complementary metrics of the desiderata given in Section 3.2: informativeness, independence, and invariance. For measuring informativeness, we report the precision, recall, and F1 score of a logistic regression model trained to predict each of the ground-truth labels from each latent space, following Eastwood and Williams (2018). We also report the MI between each latent distribution and factor, as this gives additional insight into informativeness. For measuring independence, we use the Mutual Information Gap (MIG) (Chen et al., 2018). The MIG lies in [0 , 1] , with higher values indicating a greater degree of disentanglement. Details of the MIG computation are give in Appendix E.1. We evaluate invariance by computing the Pearson's correlation coefficient between each pair of latent variables using samples from the predicted latent distributions. It also important to evaluate the ability of the models to reconstruct the input. Specifically, we target reconstruction faithfulness (i.e., how well the input and reconstruction match) and fluency. We evaluate faithfulness in terms of the ability of 6 Implementation details and evaluation of these classifiers is given in Appendix B 7 Due to computational constraints, we randomly sample 100,000 weakly annotated Amazon examples for the final dataset. Preliminary experiments with larger numbers of Amazon examples suggested that 100k is sufficient for our purposes. the model to preserve the negation, uncertainty, and content of the input. Negation and uncertainty preservation are measured by re-encoding the reconstructions, predicting the negation and uncertainty statuses from the re-encoded latent values, and computing precision, recall, and F1 score against the ground-truth labels 8 . Following previous work, we approximate a measure of content preservation in the absence of any explicit content annotations by computing the BLEU score between the input and the reconstruction (self-BLEU) (Bao et al., 2019; Cheng et al., 2020b; Balasubramanian et al., 2021). We evaluate fluency of the reconstruction by computing the perplexities (PPL) under GPT-2, a strong, general-domain language model (Radford et al., 2019). Finally, we evaluate the models' ability to flip the negation or uncertainty status of the input. For each test example, we override the value of the latent factor we want to change to represent the opposite of its ground-truth label. The ability of the model to control negation and uncertainty is measured by re-encoding the reconstructions obtained from the overridden latents, predicting from the re-encoded latent values, and computing accuracy against the opposite of the ground-truth labels. In the following, Section 5.1 reports the disentanglement results and Section 5.2 reports the faithfulness and fluency results. Section 5.3 discusses how these results address the two research questions proposed in Section 1. 5.1 Disentanglement The informativeness of each latent space with respect to each target factor is shown in Table 1 given as predictive performance and MI. The baseline ELBO objective alone fails to disentangle. It puts almost all representation power in the content space, which is nevertheless still uninformative of the negation and uncertainty factors, with low MIs and F1s. The model using the INF auxiliary objective does, however, manage to achieve good disentanglement: the negation and uncertainty spaces are highly informative of their target factors and uninformative of the non-target factors 9 . However, the content space is still slightly predictive of negation and uncertainty, with F1s of 0.684 and 0.649, respectively. This improves with 9 Experiments using LELBO + LADV or LELBO + LMIN did not improve over LELBO alone. the ADV and MIN objectives, where the content space shows near-random prediction performance of negation and uncertainty, with slightly improved prediction performance of the negation and uncertainty spaces for their target factors. These results are corroborated by the visualizations in Figure 4, which show clear separation by classes in the negation and uncertainty latent distributions but no distinction between classes in the content space. Additionally, we note the good predictive performance of the negation and uncertainty latents, despite their simple, 1-dimensional encoding. see that the INF objective alone results in decent disentanglement, with median MIG values  0 . 4 . The ADV and MI objectives give similar increases in MIG, up to  0 . 55 for both negation and uncertainty, and their combination, ADV+MIN, improves MIG further, up to  0 . 6 , suggesting that these objectives are complementary. We demonstrate the invariance of our models' negation and uncertainty representations in Table 2. While the ELBO objective alone results in highly covariant negation and uncertainty latent distributions (0.706), this drops significantly under INF (0.200) with additional reduction contributed by the ADV and MIN objectives (0.159). Table 3 reports the self-BLEU and perplexity for each disentanglement objective. Example reconstructions are given in Table 9. These results show that the models are quite consistent regarding content reconstruction on the train set, but this consistency drops on dev and test. While the ADV and MIN objectives provide disentanglement gains over INF, the BLEU scores betray a possible trade off of slightly poorer content preservation, despite better perplexities. While self-BLEU indicates the consistency of the reconstructions with respect to content, it does not necessarily indicate consistency of the reconstructions with respect to negation and uncertainty, which often differ from their opposite value counterparts by a single token. The consistency of the INF and INF+ADV+MIN models with respect to these factors is reported in Table 4. The INF objective alone is only somewhat consistent, with re-encoded F1s of 0.830 and 0.789 for negation and uncertainty respectively. The auxiliary objectives improve these considerably, to 0.914 and 0.893, + LINF + LADV + LINF + LMIN Factor Pass P R F1 P R F1 n 1 0.969 0.965 0.967 0.959 0.975 0.967 2 0.816 0.848 0.830 0.920 0.908 0.914 u 1 0.959 0.961 0.960 0.970 0.981 0.975 2 0.767 0.820 0.789 0.930 0.864 0.893 Table 4: Consistency of the decoder with the ground-truth values of negation and uncertainty evaluated on the test set. Table 5 shows the accuracies of each model on the controlled generation task, split by transfer direction. We found that for both negation and uncertainty modifying the status of the input works well in only one direction: from negated to positive, uncertain to certain. Changing a sentence from negated to positive or from uncertain to certain generally requires the removal of cue tokens (e.g., not, never, might ), while the opposite directions require their addition . Via linear regressions between the content representations and number of tokens, we found that the content space is highly informative of sentence length, which effectively bars the decoder from adding the required negation or uncertainty tokens 10 . A manual review of correctly and incorrectly modified sentences suggested that the decoder attempts to represent the negation/uncertainty status by modifying tokens in the input, rather than adding or removing them, in order to satisfy the length constraint. When removal is required, the cue token is often simply replaced by new tokens consistent with the representation. The inclusion of nega-tion/uncertainty cue tokens, however, only seems to occur when it is possible to change an existing token to a cue token. Details of the linear regressions as well as example successful/failed transfers are given in Appendix C.3. RQ1: Is it possible to learn disentangled representations of negation, uncertainty, and content? The results suggest that it is indeed possible to estimate a statistical model in which negation, uncertainty, and content are disentangled latent variables according to our three desiderata outlined in Section 3.2. Specifically, Table 1 shows high informativeness of the negation and uncertainty spaces across objectives, and the poor predictive ability of each latent space for non-target factors suggests independence. Figure 5 further suggests independence across models, with median MIG scores in the 0.4-0.6 range. Finally, the low covariances in Table 2 demonstrates the invariance of the latent representations to each other. compare for this task? Notably, the INF objective alone results in good disentanglement according to our three desiderata, suggesting that supervision alone is sufficient for disentanglement. Still, the addition of the ADV and MIN objectives resulted in slightly more informative (Table 1) and independent (Table 2) representations. While the self-BLEU scores reported in Table 3 suggest that content preservation is generally maintained across auxiliary objectives, small dips are seen in those using the MIN objective. This trend also holds for perplexity, suggesting that while the MIN objective can contribute to disentanglement gains, it may result in poorer reconstructions. Motivated by linguistic theory, we proposed a generative model of statements in which negation, uncertainty, and content are disentangled latent variables. We estimated this model using a VAE, comparing the performance of existing disentanglement objectives. Via a suite of evaluations, we showed that it is indeed possible to disentangle these factors. While objectives based on adversarial learning and MI minimization resulted in disentanglement and consistency gains, we found that a decent balance between variable disentanglement and reconstruction ability was obtained by a simple supervision of the latent representations (i.e., the INF objec-tive). Also, our 1-dimensional negation and uncertainty representations achieved high predictive performance, despite their simplicity. Future work will explore alternative latent distributions, such as discrete distributions (Jang et al., 2017; Dupont, 2018), which may better represent these operators. This work has some limitations. First, our model does not handle negation and uncertainty scope, but rather assumes that operators scope over the entire statement. Our model was estimated on relatively short, single-statement sentences to satisfy this assumption, but future work will investigate how operator disentanglement can be unified with models of operator scope in order to apply it to longer examples with multiple clauses. Second, while our models achieved high disentanglement, they fell short on the controlled generation task. We found that this was likely due to the models memorizing sentence length, constraining the reconstructions in way that is incompatible with the addition of negation and uncertainty cue tokens. (Bosc and Vincent, 2020) also noticed this tendency for sentence length memorization in VAEs and future will will explore their suggested remedies, such as encoder pretraining. This paper is based on results obtained from a project, JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO). This work was also supported by the University of Manchester President's Doctoral Scholar award, a collaboration between the University of Manchester and the Artificial Intelligence Research Center, the European Research Council (ERC StG DeepSPIN 758969), and by the Fundao para a Cincia e Tecnologia through contract UIDB/50008/2020.
abstain abstain abstain method objective abstain abstain abstain abstain abstain other abstain other objective abstain abstain objective objective abstain abstain abstain result method objective objective method other other other other method other other other other other other other abstain method other other objective other abstain objective method method method method method abstain abstain method method other abstain abstain method method abstain method method abstain abstain method method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective result objective result abstain abstain objective objective result result abstain other other