We address part-of-speech (POS) induction by maximizing the mutual information between the induced label and its context. We focus on two training objectives that are amenable to stochastic gradient descent (SGD): a novel generalization of the classical Brown clustering objective and a recently proposed variational lower bound. While both objectives are subject to noise in gradient updates, we show through analysis and experiments that the variational lower bound is robust whereas the generalized Brown objective is vulnerable. We obtain strong performance on a multitude of datasets and languages with a simple architecture that encodes morphology and context. We consider information theoretic objectives for POS induction, an important unsupervised learning problem in computational linguistics (Christodoulopoulos et al., 2010). The idea is to make the induced label syntactically informative by maximizing its mutual information with respect to local context. Mutual information has long been a workhorse in the development of NLP techniques, for instance the classical Brown clustering algorithm (Brown et al., 1992). But its role in today's deep learning paradigm is less clear and a subject of active investigation (Belghazi et al., 2018; Oord et al., 2018). We focus on fully differentiable objectives that can be plugged into an automatic differentiation system and efficiently optimized by SGD. Specifically, we investigate two training objectives. The first is a novel generalization of the Brown clustering objective obtained by relaxing the hard clustering constraint. The second is a recently proposed variational lower bound on mutual information (McAllester, 2018). A main challenge in optimizing these objectives is the difficulty of stochastic optimization. Each objective involves entropy estimation which is a nonlinear function of all data and does not decompose over individual instances. This makes the gradients estimated on minibatches inconsistent with the true gradient estimated from the entire dataset. To our surprise, in practice we are able to optimize the variational objective effectively but not the generalized Brown objective. We analyze the estimated gradients and show that the inconsistency error is only logarithmic in the former but linear in the latter. We validate our approach on POS induction by attaining strong performance on a multitude of datasets and languages. Our simple architecture that encodes morphology and context reaches up to 80.1 many-to-one accuracy on the 45-tag Penn WSJ dataset and achieves 4.7% absolute improvement to the previous best result on the universal treebank. Unlike previous works, our model does not rely on computationally expensive structured inference or hand-crafted features. Mutual information. Mutual information between two random variables measures the amount of information gained about one variable by observing the other. Unlike the Pearson correlation coefficient which only captures the degree of linear relationship, mutual information captures any nonlinear statistical dependencies (Kinney and At-wal, 2014). Formally, the mutual information between discrete random variables X, Y with a joint distribution p is the KL divergence between the joint distribution p ( x, y ) and the product distribution p ( x ) p ( y ) over X, Y : I ( X, Y ) = (cid:88) x,y p ( x, y ) log p ( x, y ) p ( x ) p ( y ) = E ( x,y )  p (cid:20) log p ( x, y ) p ( x ) p ( y ) (cid:21) (1) It is thus nonnegative and zero iff X and Y are independent. We assume that the marginals p ( x ) and p ( y ) are nonzero. It is insightful to write mutual information in terms of entropy. The entropy of X is H ( X ) =  (cid:88) x p ( x ) log p ( x ) = E x  p (cid:20) log 1 p ( x ) (cid:21) corresponding to the number of bits for encoding the behavior of X under p . 1 The entropy of X given the information that Y equals y is H ( X | Y = y ) =  (cid:88) x p ( x | y ) log p ( x | y ) Taking expectation over Y yields the conditional entropy of X given Y : H ( X | Y ) = (cid:88) y p ( y ) (cid:32)  (cid:88) x p ( x | y ) log p ( x | y ) (cid:33) = E ( x,y )  p (cid:20) log 1 p ( x | y ) (cid:21) (2) By manipulating the terms in mutual information, we can write I ( X, Y ) = E x  p (cid:20) log 1 p ( x ) (cid:21)  E ( x,y )  p (cid:20) log 1 p ( x | y ) (cid:21) = H ( X )  H ( X | Y ) which expresses the amount of information on X gained by observing Y . Switching X and Y shows that I ( X, Y ) = H ( Y )  H ( Y | X ) . 1 The paper will always assume log base 2 to accommodate the bit interpretation. Multiplying the term inside the log by p ( x ) /p ( x ) we derive H ( p, q ) = E x  p (cid:20) log 1 p ( x ) (cid:21) + E x  p (cid:20) log p ( x ) q ( x ) (cid:21) = H ( p ) + DKL ( p || q ) Thus H ( p, q )  H ( p ) with equality iff p = q . Our primary inspiration comes from Brown clustering (Brown et al., 1992), a celebrated word clustering technique that had been greatly influential in unsupervised and semi-supervised NLP long before continuous representations based on neural networks were popularized. It finds a clustering C : V  [ m ] of the vocabulary V into m classes by optimizing the mutual information between the clusters of a random bigram ( X, Y ) . Given a corpus of N words ( x 1 . . . x N ) , it assumes a uniform distribution over consecutive word pairs ( x i  1 , x i ) and optimizes the following empirical objective max C : V  [ m ] (cid:88) c,c (cid:48)  [ m ] #( c, c (cid:48) ) N log (cid:18) #( c, c (cid:48) ) N #( c )#( c (cid:48) ) (cid:19) (3) where #( c, c (cid:48) ) denotes the number of occurrences of the cluster pair ( c, c (cid:48) ) under C . While this optimization is intractable, Brown et al. (1992) derive an effective heuristic that 1. initializes m most frequent words as singleton clusters and 2. repeatedly merges a pair of clusters that yields the smallest decrease in mutual information. The resulting clusters have been useful in many applications (Koo et al., 2008; Owoputi et al., 2013) and has remained a strong baseline for POS induction decades later (Christodoulopoulos et al., 2010). But the approach is tied to highly nontrivial combinatorial optimization tailored for the spe-cific problem and difficult to scale/generalize. In the remainder of the paper, we assume discrete random variables ( X, Y )  X Y with a joint distribution D that represent naturally co-occurring observations. In POS induction experiments, we will set D to be a context-word distribution where Y is a random word and X is the surrounding context of Y (thus Y is the vocabulary and X is the space of all possible contexts). Let m be the number of labels to induce. We introduce a pair of trainable classifiers that define conditional label distributions p ( z | x ) and q ( z | y ) for all x  X , y  Y , and z  [ m ] . For instance, p ( | y ) can be the output of a softmax layer on some transformation of y . Our goal is to learn these classifiers without observing the latent variable z by optimizing an appropriate objective. For training data, we assume N iid samples ( x 1 , y 1 ) . . . ( x N , y N )  D . Our first attempt is to maximize the mutual information between the predictions of p and q . Intuitively, this encourages p and q to agree on some annotation scheme (up to a permutation of labels), modeling the dynamics of inter-annotator agreement (Artstein, 2017). It can be seen as a differentiable generalization of the Brown clustering objective. To this end, define p ( z ) = E x  D [ p ( z | x )]  z  [ m ] q ( z ) = E y  D [ q ( z | y )]  z  [ m ] The mutual information between the predictions of p and q on a single sample ( x, y ) is then J mi x,y = (cid:88) z,z (cid:48) p ( z | x ) q ( z (cid:48) | y ) log p ( z | x ) q ( z (cid:48) | y ) p ( z ) q ( z (cid:48) ) and the objective (to maximize) is J mi = E ( x,y )  D (cid:2) J mi x,y (cid:3) Note that this becomes exactly the original Brown objective (3) if ( X, Y ) is defined as a random bigram and p and q are tied and constrained to be a hard clustering. p ( z ) = 1 NN (cid:88) i =1 p ( z | x i )  z  [ m ]  q ( z ) = 1 NN (cid:88) i =1 q ( z | y i )  z  [ m ] (cid:98) J mi i = (cid:88) z,z (cid:48) p ( z | x i ) q ( z (cid:48) | y i ) log p ( z | x i ) q ( z (cid:48) | y i )  p ( z ) q ( z (cid:48) ) (cid:98) J mi = 1 NN (cid:88) i =1 (cid:98) J mi i (4) the objective cannot be written as a sum of local objectives because we take log of the estimates  q ( z ) and  p ( z ) computed from all samples. This makes the stochastic gradient estimator biased (i.e., it does not match the gradient of (4) in expectation) and compromises the correctness of SGD. This bias is investigated more closely in Section 4. The second training objective we consider can be derived in a rather heuristic but helpful manner as follows. Since X, Y are always drawn together, if q ( z | y ) is the target label distribution for the pair ( x, y ) , then we can train p ( z | x ) by minimizing the cross entropy between q and p over samples H ( q, p ) = E ( x,y )  D (cid:34)  (cid:88) z q ( z | y ) log p ( z | x ) (cid:35) which is minimized to zero at p = q . However, q is also untrained and needs to be trained along with p . Thus this loss alone admits trivial solutions such as setting p (1 | x ) = p (1 | y ) = 1 for all ( x, y ) . This undesirable behavior can be prevented by simultaneously maximizing the entropy of q . Let Z denote a random label from q with distribution q ( z ) = E y  D [ q ( z | y )] (thus Z is a function of q ). The entropy of Z is H ( Z ) =  (cid:88) z q ( z ) log q ( z ) Putting together, the objective (to maximize) is J var = H ( Z )  H ( q, p ) Variational interpretation. The reason this objective is named a variational lower bound is due to McAllester (2018) who shows the following. Consider the mutual information between Z and the raw signal X : I ( X, Z ) = H ( Z )  H ( Z | X ) (5) Because Z is drawn from q conditioning on Y , which is co-distributed with X , we have a Markov chain X  Y q  Z . Thus maximizing I ( X, Z ) over the choice of q is a reasonable objective that enforces predictive coding: the label predicted by q from Y must be as informative of X as possible. It can be seen as a special case of the objective underlying the information bottleneck method (Tishby et al., 2000). So what is the problem with optimizing (5) directly? The problem is that the conditional entropy under the model H ( Z | X ) = E ( x,y )  D z  q ( | y ) (cid:20) log 1  ( z | x ) (cid:21) (6) involves the posterior probability of z given x  ( z | x ) = (cid:80) y D ( x, y ) q ( z | y ) (cid:80) y,z D ( x, y ) q ( z | y ) This conditional marginalization is generally intractable and cannot be approximated by sampling since the chance of seeing a particular x is small. However, we can introduce a variational distribution p ( z | x ) to model  ( z | x ) . Plugging this into (6) we observe that E ( x,y )  D z  q ( | y ) (cid:20) log 1 p ( z | x ) (cid:21) = H ( q, p ) Moreover, E ( x,y )  D z  q ( | y ) (cid:20) log 1 p ( z | x ) (cid:21) = E ( x,y )  D z  q ( | y ) (cid:20) log  ( z | x )  ( z | x ) p ( z | x ) (cid:21) = H ( Z | X ) + DKL (  || p ) Thus H ( q, p ) is an upper bound on H ( Z | X ) for any p with equality iff p matches the true posterior distribution  . This in turn means that J var = H ( Z )  H ( q, p ) is a lower bound on I ( X, Z ) , hence the name. Empirical objective. As with the generalized Brown objective, the variational lower bound can be estimated from the training data as (cid:98) H ( Z ) =  (cid:88) z  q ( z ) log  q ( z ) (cid:98) H ( q, p ) = 1 NN (cid:88) i =1 (cid:32)  (cid:88) z q ( z | y i ) log p ( z | x i ) (cid:33) (cid:98) J var = (cid:98) H ( Z )  (cid:98) H ( q, p ) (7) where  q ( z ) is defined as in Section 3.1. Our task is again to maximize this empirical objective (7) by taking gradient steps at random minibatches. Once again, it cannot be written as a sum of local objectives because the entropy term involves log of  q ( z ) computed from all samples. Thus it is not clear if stochastic optimization will be effective. The discussion of the two objectives in the previous section is incomplete because the stochastic gradient estimator is biased under both objectives. In this section, we formalize this issue and analyze the bias. Let B 1 . . . BK be a partition of the N training examples ( x 1 , y 1 ) . . . ( x N , y N ) into K (iid) minibatches. For simplicity, assume | B k | = M for all k and N = MK . We will adopt the same notation in Section 3 for  p ( z ) and  q ( z ) estimated from all N samples. Define analogous estimates based on the k -th minibatch by  p k ( z ) = 1 M (cid:88) x  B k p ( z | x )  z  [ m ]  q k ( z ) = 1 M (cid:88) y  B k q ( z | y )  z  [ m ] If l N denotes an objective function computed from all N samples in the training data and l k denotes the same objective computed from B k , a condition on the correctness of SGD is that the gradient of l k (with respect to model parameters) is consistent with the gradient of l N on average:  l N = 1 KK (cid:88) k =1  l k + (cid:15) (8) where (cid:15) denotes the bias of the stochastic gradient estimator. In particular, any loss of the form l N = (1 /K ) (cid:80) k l k that decomposes over inde-pendent minibatches (e.g., the cross-entropy loss for supervised classification) satisfies (8) with (cid:15) = 0 . The bias is nonzero for the unsupervised objectives considered in this work due to the issues discussed in Section 3. The following theorem precisely quantifies the bias for the empirical losses associated with the variational bound and the generalized Brown objectives. We only show the result with the gradient with respect to q , but the result with the gradient with respect to p is analogous and omitted for brevity. Theorem 4.1. Assume the setting in Section 4.1 and the gradient is taken with respect to the parameters of q . (cid:15) = 1 KK (cid:88) k =1 (cid:88) z log  q ( z )  q k ( z )   q k ( z ) On the other hand, for l N =  (cid:98) J mi defined in (4) (cid:15) = 1 NK (cid:88) k =1 (cid:88) z,z (cid:48) (cid:18) (cid:15) k ( z, z (cid:48) )   q k ( z (cid:48) ) + log  p ( z ) q ( z (cid:48) )  p k ( z ) q k ( z (cid:48) ) (cid:88) ( x,y )  B k p ( z | x )  q ( z (cid:48) | y ) (cid:19) where (cid:15) k ( z, z (cid:48) ) = 1 KN (cid:88) i =1 p ( z | x i ) q ( z (cid:48) | y i )  q ( z (cid:48) )  (cid:88) ( x,y )  B k p ( z | x ) q ( z (cid:48) | y )  q k ( z (cid:48) ) A proof can be found in the appendix. We see that both biases go to zero as  p k and  q k approach  p and  q . However, the bias is logarithmic in the ratio  q ( z ) /  q k ( z ) for the variational lower bound but roughly linear in the difference between 1  q ( z (cid:48) ) and 1  q k ( z (cid:48) ) for the generalized Brown objective. In this sense, the variational lower bound is exponentially more robust to noise in minibatch estimates than the generalized Brown objective. This is con-firmed in experiments: we are able to optimize (cid:98) J var with minibatches as small as 80 examples but unable to optimize (cid:98) J mi unless minibatches are prohibitively large. We demonstrate the effectiveness of our training objectives on the task of POS induction. The goal of this task is to induce the correct POS tag for a given word in context (Merialdo, 1994). As typical in unsupervised tasks, evaluating the quality of induced labels is challenging; see Christodoulopoulos et al. (2010) for an in-depth discussion. To avoid complications, we follow a standard practice (Berg-Kirkpatrick et al., 2010; Ammar et al., 2014; Lin et al., 2015; Stratos et al., 2016) and adopt the following setting for all compared methods. ground-truth POS tag in the annotated data and report the resulting accuracy. We also use the V-measure (Rosenberg and Hirschberg, 2007) when comparing with CRF autoencoders to be consistent with reported results (Ammar et al., 2014; Lin et al., 2015). We use the number of ground-truth POS tags as the value of m (i.e., number of labels to induce). This is a data-dependent quantity, for instance 45 in the Penn WSJ and 12 in the universal treebank. Fixing the number of tags this way obviates many evaluation issues. Model-specific hyperparameters are tuned on the English Penn WSJ dataset. This configuration is then fixed and used for all other datasets: 10 languages in the universal treebank 2 and 7 languages from CoNLL-X and CoNLL 2007. We set D to be a uniform distribution over context-word pairs in the training corpus. Given N samples ( x 1 , y 1 ) . . . ( x N , y N )  D , we optimize the variational objective (7) or the generalized Brown objective (4) by taking gradient steps at random minibatches. This gives us conditional label distributions p ( z | x ) and q ( z | y ) for all contexts x , words y , and labels z . At test time, we use z  = arg max z q ( z | y ) as the induced label of word y . We experimented with different inference methods such as taking arg max z p ( z | x ) q ( z | y ) but did not find it helpful. Let V denote the vocabulary. We assume an integer H  1 that specifies the width of local context. Given random word y  V , we set x  V 2 H to be an ordered list of H left and H right words of y . For example, with H = 2 , a typical context-target pair ( x, y )  D may look like x = ( had, these, in, my ) y = keys We find this simple fixed-window definition of observed variables to be the best inductive bias for POS induction. The correct label can be inferred from either x or y in many cases: in the above example, we can infer that the correct POS tag is plural noun by looking at the target or the context. 2 https://github.com/ryanmcd/uni-dep-tb 5.3 Architecture We use the following simple architecture to parameterize the label distribution p ( | x ) conditioned on context x  VH and the label distribution q ( | y ) conditioned on word y  V . Context architecture. The parameters of p are word embeddings e w  R d for all w  V and matrices W j  R m  d for all j = 1 . . . 2 H . Given 2 H ordered contextual words x = ( w j ) 2 Hj =1 , we define p ( | x ) = softmax   2 H (cid:88) j =1 W j e w j   Word architecture. The parameters of q are the same word embeddings e w  R d shared with p , character embeddings e c  R d/ 2 for all distinct characters c , two single-layer LSTMs with input/output dimension d/ 2 , and matrices W c , W w  R m  d . Given the word y with character sequence c 1 . . . c T , we define ( f 1 . . . f T ) = LSTM f ( e c 1 . . . e c T ) ( b 1 . . . b T ) = LSTM b ( e c T . . . e c 1 ) q ( | y ) = softmax (cid:18) W c (cid:20) f T b T (cid:21) + W w e y (cid:19) The overall architecture is illustrated in Figure 1. Our hyperparameters are the embedding dimension d = 200 , the context width H = 2 , the learning rate of the Adam optimizer r = 0 . 001 , and the minibatch size B = 80 . 3 Their values are tuned on the 45-tag Penn WSJ dataset to maximize accuracy. We focus on comparing with the following models which are some of the strongest baselines in the literature we are aware of. Berg-Kirkpatrick et al. (2010) extend a standard hidden Markov Model (HMM) to incorporate linguistic features. Stratos et al. (2016) develop a factorization-based algorithm for learning a constrained HMM. Ammar et al. (2014) propose a CRF autoencoder that reconstructs words from a structured label sequence. Lin et al. (2015) extend Ammar et al. (2014) by switching a categorical reconstruction distribution with a Gaussian distribution. In addition to these 3 An implementation is available at: https:// github.com/karlstratos/mmi-tagger . baselines, we also report results with Brown clustering (Brown et al., 1992), the Baum-Welch algorithm (Baum and Petrie, 1966), and k -means clustering of 300 -dimensional GloVe vectors (Pen-nington et al., 2014). The 45-tag Penn WSJ dataset. The 45-tag Penn WSJ dataset is a corpus of around one million words each tagged with one of m = 45 tags. It is used to optimize hyperparameter values for all compared methods. Table 1 shows the average accuracy over 10 random restarts with the best hyperparameter configurations; standard deviation is given in parentheses (except for deterministic methods Stratos et al. (2016) and Brown cluster-ing). Our model trained with the variational objective (7) outperforms all baselines. 4 We also observe that our model trained with the generalized Brown objective (4) does not work. We have found that unless the minibatch size is as large as 10,000 the gradient steps do not effectively increase the true data-wide mutual information (4). This supports our bias analysis in Section 4. While it may be possible to develop techniques to resolve the difficulty, for instance keeping a moving average of estimates to stabilize estimation, we leave this as future work and focus on the variational objective in the remainder of the paper. 4 We remark that Tran et al. (2016) report a single number 79.1 with a neuralized HMM. We also note that the concurrent work by He et al. (2018) obtains 80.8 by using word embeddings carefully pretrained on one billion words. model (accuracy 80.1) to better understand the sources of its strong performance. Context size H = 2 is a sizable improvement over H = 3 or H = 1 . Random sampling is significantly more effective than sentence-level batching (i.e., each minibatch is the set of context-word pairs within a single sentence as done in McAllester (2018)). G lo V e initialization of word embeddings e w is harmful. As expected for POS tagging, morphological modeling with LSTMs gives the largest improvement. While it may be surprising that G lo V e initialization is harmful, it is well known that pretrained word embeddings do not necessarily capture syntactic relationships (as evident in the poor performance of k -means clustering). Consider the top ten nearest neighbors of the word made under GloVe embeddings (840B.300d, within PTB vocab) shown in Table 3. The neighbors are clearly not in the same syntactic category. The embeddings can be made more syntactic by controlling the context window. But we found it much more effective (and simpler) to start from randomly initialized embeddings and let the objective induce appropriate representations. The 12-tag universal treebank. The universal treebank v2.0 is a corpus in ten languages tagged with m = 12 universal POS tags (McDonald et al., 2013). We use this corpus to be compatible with existing results. Table 4 shows results on the dataset, using the same setting in the experiments on the Penn WSJ dataset. Our model significantly outperforms the previous state-of-the-art, achieving an absolute gain of 4.7 over Stratos et al. (2016) in average accuracy. Comparison with CRF autoencoders. Table 5 shows a direct comparison with CRF autoencoders (Ammar et al., 2014; Lin et al., 2015) in many-to-one accuracy and the V-measure. We compare against their reported numbers by running our model once on the same datasets using the same setting in the experiments on the Penn WSJ dataset. The data consists of the training portion of CoNLL-X and CoNLL 2007 labeled with 12 universal tags. Our model is competitive with all baselines. Information theory, in particular mutual information, has played a prominent role in NLP (Church and Hanks, 1990; Brown et al., 1992). It has intimate connections to the representation learning capabilities of neural networks (Tishby and Zaslavsky, 2015) and underlies many celebrated modern approaches to unsupervised learning such as generative adversarial networks (GANs) (Goodfellow et al., 2014). There is a recent burst of effort in learning continuous representations by optimizing various lower bounds on mutual information (Belghazi et al., 2018; Oord et al., 2018; Hjelm et al., 2018). These representations are typically eval-Method de en es fr id it ja ko pt-br sv Mean Variational (cid:98) J var (7) (  1.5) 75.4 (  1.7) 73.1 (  1.0) 73.1 (  2.9) 70.4 (  1.5) 73.6 (  3.3) 67.4 (  0.4) 77.9 (  1.2) 65.6 (  2.3) 70.7 (  1.5) 67.1 71.4 Stratos et al. 63.4 71.4 74.3 71.9 67.3 60.2 69.4 61.8 65.8 61.0 66.7 Berg-Kirkpatrick et al. (  1.8) 67.5 (  3.5) 62.4 (  3.1) 67.1 (  4.5) 62.1 (  3.9) 61.3 (  2.9) 52.9 (  2.9) 78.2 (  3.6) 60.5 (  2.2) 63.2 (  2.5) 56.7 63.2 Brown et al. 60.0 62.9 67.4 66.4 59.3 66.1 60.3 47.5 67.4 61.9 61.9 Baum-Welch (  4.8) 45.5 (  3.4) 59.8 (  2.2) 60.6 (  3.6) 60.1 (  3.1) 49.6 (  2.6) 51.5 (  2.1) 59.5 (  0.6) 51.7 (  3.7) 59.5 (  3.0) 42.4 54.0 Table 4: Many-to-one accuracy on the 12-tag universal treebank dataset. uated on extrinsic tasks as features. In contrast, we learn discrete representations by optimizing a novel generalization of the Brown clustering objective (Brown et al., 1992) and a variational lower bound on mutual information proposed by McAllester (2018). We focus on intrinsic evaluation of these representations on POS induction. Extrinsic evaluation of these representations in downstream tasks is an important future direction. The issue of biased stochastic gradient estimators is a common challenge in unsupervised learning (e.g., see Wang et al., 2015). This arises mainly because the objective involves a nonlinear transformation of all samples in a training dataset, for instance the whitening constraints in deep canonical correlation analysis (CCA) (An-drew et al., 2013). In this work, the problem arises because of entropy. This issue is not considered in the original work of McAllester (2018) and the error analysis we present in Section 4 is novel. Our finding is that the feasibility of stochastic optimization greatly depends on the size of the bias in gradient estimates, as we are able to effectively optimize the variational objective while not the generalized Brown objective. Our POS induction system has some practical advantages over previous approaches. Many rely on computationally expensive structured inference or pre-optimized features (or both). For instance, Tran et al. (2016) need to calculate for-ward/backward messages and is limited to truncated sequences by memory constraints. Berg-Kirkpatrick et al. (2010) rely on extensively hand-engineered linguistic features. Ammar et al. (2014), Lin et al. (2015), and He et al. (2018) rely on carefully pretrained lexical representations like Brown clusters and word embeddings. In contrast, the model presented in this work requires no expensive structured computation or feature engineering and uses word/character embeddings trained from scratch. It is easy to implement using a standard neural network library and outperforms these previous works in many cases. The author thanks David McAllester for many insightful discussions, and Sam Wiseman for helpful comments. The Titan Xp used for this research was donated by the NVIDIA Corporation.
abstain objective objective result abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain objective result abstain result abstain other other other other method other other other other other method abstain other other other other other other other method abstain abstain method abstain abstain method abstain abstain abstain other other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain other abstain abstain other abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain result abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain other other other other other abstain method other other other abstain abstain objective abstain other other other other objective other other other