A JOINT MANY-TASK MODEL: GROWING A NEURAL NETWORK FOR MULTIPLE NLP TASKS
Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce such a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. All layers include shortcut connections to both word representations and lower-level task predictions. We use a simple regularization term to allow for optimizing all model weights to improve one task’s loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end trainable model obtains state-of-the-art results on chunking, dependency parsing, semantic relatedness and textual entailment. It also performs competitively on POS tagging. Our dependency parsing layer relies only on a single feed-forward pass and does not require a beam search.
Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce such a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. All layers include shortcut connections to both word representations and lower-level task predictions. We use a simple regularization term to allow for optimizing all model weights to improve one task’s loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end trainable model obtains state-of-the-art results on chunking, dependency parsing, semantic relatedness and textual entailment. It also performs competitively on POS tagging. Our dependency parsing layer relies only on a single feed-forward pass and does not require a beam search.
1 INTRODUCTION
The potential for leveraging multiple levels of representation has been demonstrated in a variety of ways in the field of Natural Language Processing (NLP). For example, Part-Of-Speech (POS) tags are used to train syntactic parsers. The parsers are used to improve higher-level tasks, such as natural language inference (Chen et al., 2016), relation classification (Socher et al., 2012), sentiment analysis (Socher et al., 2013; Tai et al., 2015), or machine translation (Eriguchi et al., 2016). However, higher level tasks are not usually able to improve lower level tasks, often because systems are pipelines and not trained end-to-end.
In deep learning, unsupervised word vectors are useful representations and often used to initialize recurrent neural networks for subsequent tasks (Pennington et al., 2014). However, not being jointly trained, deep NLP models have yet shown benefits from predicting many (> 4) increasingly complex linguistic tasks each at a successively deeper layer. Instead, existing models are often designed to predict different tasks either entirely separately or at the same depth (Collobert et al., 2011), ignoring linguistic hierarchies.
We introduce a Joint Many-Task (JMT) model, outlined in Fig. 1, which predicts increasingly complex NLP tasks at successively deeper layers. Unlike traditional NLP pipeline systems, our single JMT model can be trained end-to-end for POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment. We propose an adaptive training and regularization strategy to grow this model in its depth. With the help of this strategy we avoid catastrophic interference between tasks, and instead show that both lower and higher level tasks benefit from the joint training. Our model is influenced by the observation of Søgaard & Goldberg (2016) who showed that predicting two different tasks is more accurate when performed in different layers than in the same layer (Collobert et al., 2011).
∗Work was done while the first author was an intern at Salesforce Research. †Corresponding author.
2 THE JOINT MANY-TASK MODEL
In this section, we assume that the model is trained and describe its inference procedure. We begin at the lowest level and work our way to higher layers and more complex tasks.
2.1 WORD REPRESENTATIONS
For each wordwt in the input sentence s of length L, we construct a representation by concatenating a word and a character embedding.
Word embeddings: We use Skip-gram (Mikolov et al., 2013) to train a word embedding matrix, which will be shared across all of the tasks. The words which are not included in the vocabulary are mapped to a special UNK token.
Character n-gram embeddings: Character n-gram embeddings are learned using the same skipgram objective function as the word vectors. We construct the vocabulary of the character n-grams in the training data and assign an embedding for each character n-gram. The final character embedding is the average of the unique character n-gram embeddings of a word wt.1 For example, the character n-grams (n = 1, 2, 3) of the word “Cat” are {C, a, t, #BEGIN#C, Ca, at, t#END#, #BEGIN#Ca, Cat, at#END#}, where “#BEGIN#” and “#END#” represent the beginning and the end of each word, respectively. The use of the character n-gram embeddings efficiently provides morphological features and information about unknown words. The training procedure for the character n-gram embeddings is described in Section 3.1, and for further details, please see Appendix A. Each word is subsequently represented as xt, the concatenation of its corresponding word and character vectors.
2.2 WORD-LEVEL TASK: POS TAGGING
The first layer of the model is a bi-directional LSTM (Graves & Schmidhuber, 2005; Hochreiter & Schmidhuber, 1997) whose hidden states are used to predict POS tags. We use the following Long Short-Term Memory (LSTM) units for the forward direction:
it = σ (Wigt + bi) , ft = σ (Wfgt + bf ) , ot = σ (Wogt + bo) , ut = tanh (Wugt + bu) , ct = it ut + ft ct−1, ht = ot tanh (ct) , (1)
1Wieting et al. (2016) used a nonlinearity, but we have observed that the simple averaging also works well.
where we define the input gt as gt = [ −→ h t−1;xt], i.e. the concatenation of the previous hidden state and the word representation of wt. The backward pass is expanded in the same way, but a different set of weights are used.
For predicting the POS tag of wt, we use the concatenation of the forward and backward states in a one-layer bi-LSTM layer corresponding to the t-th word: ht = [ −→ h t; ←− h t]. Then each ht (1 ≤ t ≤ L) is fed into a standard softmax classifier with a single ReLU layer which outputs the probability vector y(1) for each of the POS tags.
2.3 WORD-LEVEL TASK: CHUNKING
Chunking is also a word-level classification task which assigns a chunking tag (B-NP, I-VP, etc.) for each word. The tag specifies the region of major phrases (or chunks) in the sentence.
Chunking is performed in the second bi-LSTM layer on top of the POS layer. When stacking the bi-LSTM layers, we use Eq. (1) with input g(2)t = [h (2) t−1;h (1) t ;xt; y (pos) t ], where h (1) t is the hidden state of the first (POS) layer. We define the weighted label embedding y(pos)t as follows:
y (pos) t = C∑ j=1 p(y (1) t = j|h (1) t )`(j), (2)
where C is the number of the POS tags, p(y(1)t = j|h (1) t ) is the probability value that the j-th POS tag is assigned to wt, and `(j) is the corresponding label embedding. The probability values are automatically predicted by the POS layer working like a built-in POS tagger, and thus no gold POS tags are needed. This output embedding can be regarded as a similar feature to the K-best POS tag feature which has been shown to be effective in syntactic tasks (Andor et al., 2016; Alberti et al., 2015). For predicting the chunking tags, we employ the same strategy as POS tagging by using the concatenated bi-directional hidden states h(2)t = [ −→ h (2) t ; ←− h (2) t ] in the chunking layer. We also use a single ReLU hidden layer before the classifier.
2.4 SYNTACTIC TASK: DEPENDENCY PARSING
Dependency parsing identifies syntactic relationships (such as an adjective modifying a noun) between pairs of words in a sentence. We use the third bi-LSTM layer on top of the POS and chunking layers to classify relationships between all pairs of words. The input vector for the LSTM includes hidden states, word representations, and the label embeddings for the two previous tasks: g (3) t = [h (3) t−1;h (2) t ;xt; (y (pos) t + y (chk) t )], where we computed the chunking vector in a similar fashion as the POS vector in Eq. (2). The POS and chunking tags are commonly used to improve dependency parsing (Attardi & DellOrletta, 2008).
Like a sequential labeling task, we simply predict the parent node (head) for each word in the sentence. Then a dependency label is predicted for each of the child-parent node pairs. To predict the parent node of the t-th word wt, we define a matching function between wt and the candidates of the parent node as m (t, j) = h(3)t T Wdh (3) j , where Wd is a parameter matrix. For the root, we
define h(3)L+1 = r as a parameterized vector. To compute the probability that wj (or the root node) is the parent of wt, the scores are normalized:
p(j|h(3)t ) = exp (m (t, j))∑L+1
k=1,k 6=t exp (m (t, k)) , (3)
where L is the sentence length.
Next, the dependency labels are predicted using [h(3)t ;h (3) j ] as input to a standard softmax classifier with a single ReLU layer. At test time, we greedily select the parent node and the dependency label for each word in the sentence.2 At training time, we use the gold child-parent pairs to train the label predictor.
2.5 SEMANTIC TASK: SEMANTIC RELATEDNESS
The next two tasks model the semantic relationships between two input sentences. The first task measures the semantic relatedness between two sentences. The output is a real-valued relatedness score for the input sentence pair. The second task is a textual entailment task, which requires one to determine whether a premise sentence entails a hypothesis sentence. There are typically three classes: entailment, contradiction, and neutral.
The two semantic tasks are closely related to each other. If the semantic relatedness between two sentences is very low, they are unlikely to entail each other. Based on this intuition and to make use of the information from lower layers, we use the fourth and fifth bi-LSTM layer for the relatedness and entailment task, respectively.
2This method currently assumes that each word has only one parent node, but it can be expanded to handle multiple parent nodes, which leads to cyclic graphs.
Now it is required to obtain the sentence-level representation rather than the word-level representation h(4)t used in the first three tasks. We compute the sentence-level representation h (4) s as the element-wise maximum values across all of the word-level representations in the fourth layer:
h(4)s = max ( h (4) 1 , h (4) 2 , . . . , h (4) L ) . (4)
To model the semantic relatedness between s and s′, we follow Tai et al. (2015). The feature vector for representing the semantic relatedness is computed as follows:
d1(s, s ′) = [∣∣∣h(4)s − h(4)s′ ∣∣∣ ;h(4)s h(4)s′ ] , (5) where
∣∣∣h(4)s − h(4)s′ ∣∣∣ is the absolute values of the element-wise subtraction, and h(4)s h(4)s′ is the element-wise multiplication. Both of them can be regarded as two different similarity metrics of the two vectors. Then d1(s, s′) is fed into a softmax classifier with a single Maxout hidden layer (Goodfellow et al., 2013) to output a relatedness score (from 1 to 5 in our case) for the sentence pair.
2.6 SEMANTIC TASK: TEXTUAL ENTAILMENT
For entailment classification between two sentences, we also use the max-pooling technique as in the semantic relatedness task. To classify the premise-hypothesis pair (s, s′) into one of the three classes, we compute the feature vector d2(s, s′) as in Eq. (5) except that we do not use the absolute values of the element-wise subtraction, because we need to identify which is the premise (or hypothesis). Then d2(s, s′) is fed into a standard softmax classifier.
To make use of the output from the relatedness layer directly, we use the label embeddings for the relatedness task. More concretely, we compute the class label embeddings for the semantic relatedness task similar to Eq. (2). The final feature vectors that are concatenated and fed into the entailment classifier are the weighted relatedness label embedding and the feature vector d2(s, s′).3 We use three Maxout hidden layers before the classifier.
3 TRAINING THE JMT MODEL
The model is trained jointly over all datasets. During each epoch, the optimization iterates over each full training dataset in the same order as the corresponding tasks described in the modeling section.
3.1 PRE-TRAINING WORD REPRESENTATIONS
We pre-train word embeddings using the Skip-gram model with negative sampling (Mikolov et al., 2013). We also pre-train the character n-gram embeddings using Skip-gram. The only difference is that each input word embedding in the Skip-gram model is replaced with its corresponding average embedding of the character n-gram embeddings described in Section 2.1. These embeddings are fine-tuned during the training of our JMT model. We denote the embedding parameters as θe.
3.2 TRAINING THE POS LAYER
Let θPOS = (WPOS, bPOS, θe) denote the set of model parameters associated with the POS layer, where WPOS is the set of the weight matrices in the first bi-LSTM and the classifier, and bPOS is the set of the bias vectors. The objective function to optimize θPOS is defined as follows:
J1(θPOS) = − ∑ s ∑ t log p ( y (1) t = α|h (1) t ) + λ‖WPOS‖2 + δ‖θe − θ′e‖2, (6)
where p(y(1)t = αwt |h (1) t ) is the probability value that the correct label α is assigned to wt in the sentence s, λ‖WPOS‖2 is the L2-norm regularization term, and λ is a hyperparameter. 3This modification does not affect the LSTM transitions, and thus it is still possible to add other singlesentence-level tasks on top of our model.
We call the second regularization term δ‖θe− θ′e‖2 a successive regularization term. The successive regularization is based on the idea that we do not want the model to forget the information learned for the other tasks. In the case of POS tagging, the regularization is applied to θe, and θ′e is the embedding parameter after training the final task in the top-most layer at the previous training epoch. δ is a hyperparameter.
3.3 TRAINING THE CHUNKING LAYER
The objective function is defined as follows: J2(θchk) = − ∑ s ∑ t log p(y (2) t = α|h (2) t )d+ λ‖Wchk‖2 + δ‖θPOS − θ′POS‖2, (7)
which is similar to that of POS tagging, and θchk is (Wchk, bchk, EPOS, θe), where Wchk and bchk are the weight and bias parameters including those in θPOS, and EPOS is the set of the POS label embeddings. θ′POS is the one after training the POS layer at the current training epoch.
3.4 TRAINING THE DEPENDENCY LAYER
The objective function is defined as follows: J3(θdep) = − ∑ s ∑ t log p(α|h(3)t )p(β|h (3) t , h (3) α )+λ(‖Wdep‖2+‖Wd‖2)+δ‖θchk−θ′chk‖2, (8)
where p(α|h(3)t ) is the probability value assigned to the correct parent node α for wt, and p(β|h(3)t , h (3) α ) is the probability value assigned to the correct dependency label β for the childparent pair (wt, α). θdep is defined as (Wdep, bdep,Wd, r, EPOS, Echk, θe), where Wdep and bdep are the weight and bias parameters including those in θchk, and Echk is the set of the chunking label embeddings.
3.5 TRAINING THE RELATEDNESS LAYER
Following Tai et al. (2015), the objective function is defined as follows: J4(θrel) = ∑ (s,s′) KL ( p̂(s, s′) ∥∥∥p(h(4)s , h(4)s′ ))+ λ‖Wrel‖2 + δ‖θdep − θ′dep‖2, (9) where p̂(s, s′) is the gold distribution over the defined relatedness scores, p(h(4)s , h (4) s′ ) is the pre-
dicted distribution given the the sentence representations, and KL ( p̂(s, s′) ∥∥∥p(h(4)s , h(4)s′ )) is the KL-divergence between the two distributions. θrel is defined as (Wrel, brel, EPOS, Echk, θe).
3.6 TRAINING THE ENTAILMENT LAYER
The objective function is defined as follows: J5(θent) = − ∑ (s,s′) log p(y (5) (s,s′) = α|h (5) s , h (5) s′ ) + λ‖Went‖ 2 + δ‖θrel − θ′rel‖2, (10)
where p(y(5)(s,s′) = α|h (5) s , h (5) s′ ) is the probability value that the correct label α is assigned to the premise-hypothesis pair (s, s′). θent is defined as (Went, bent, EPOS, Echk, Erel, θe), where Erel is the set of the relatedness label embeddings.
4 RELATED WORK
Many deep learning approaches have proven to be effective in a variety of NLP tasks and are becoming more and more complex. They are typically designed to handle single tasks, or some of them are designed as general-purpose models (Kumar et al., 2016; Sutskever et al., 2014) but applied to different tasks independently.
For handling multiple NLP tasks, multi-task learning models with deep neural networks have been proposed (Collobert et al., 2011; Luong et al., 2016), and more recently Søgaard & Goldberg (2016) have suggested that using different layers for different tasks is more effective than using the same layer in jointly learning closely-related tasks, such as POS tagging and chunking. However, the number of tasks was limited or they have very similar task settings like word-level tagging, and it was not clear how lower-level tasks could be also improved by combining higher-level tasks.
In the field of computer vision, some transfer and multi-task learning approaches have also been proposed (Li & Hoiem, 2016; Misra et al., 2016). For example, Misra et al. (2016) proposed a multi-task learning model to handle different tasks. However, they assume that each data sample has annotations for the different tasks, and do not explicitly consider task hierarchies.
Recently, Rusu et al. (2016) have proposed a progressive neural network model to handle multiple reinforcement learning tasks, such as Atari games. Like our JMT model, their model is also successively trained according to different tasks using different layers called columns in their paper. In their model, once the first task is completed, the model parameters for the first task are fixed, and then the second task is handled by adding new model parameters. Therefore, accuracy of the previously trained tasks is never improved. In NLP tasks, multi-task learning has the potential to improve not only higher-level tasks, but also lower-level tasks. Rather than fixing the pre-trained model parameters, our successive regularization allows our model to continuously train the lower-level tasks without significant accuracy drops.
5 EXPERIMENTAL SETTINGS
5.1 DATASETS
POS tagging: To train the POS tagging layer, we used the Wall Street Journal (WSJ) portion of Penn Treebank, and followed the standard split for the training (Section 0-18), development (Section 19- 21), and test (Section 22-24) sets. The evaluation metric is the word-level accuracy.
Chunking: For chunking, we also used the WSJ corpus, and followed the standard split for the training (Section 15-18) and test (Section 20) sets as in the CoNLL 2000 shared task. We used Section 19 as the development set, following Søgaard & Goldberg (2016), and employed the IOBES tagging scheme. The evaluation metric is the F1 score defined in the shared task.
Dependency parsing: We also used the WSJ corpus for dependency parsing, and followed the standard split for the training (Section 2-21), development (Section 22), and test (Section 23) sets. We converted the treebank data to Stanford style dependencies using the version 3.3.0 of the Stanford converter. The evaluation metrics are the Unlabeled Attachment Score (UAS) and the Labeled Attachment Score (LAS), and punctuations are excluded for the evaluation.
Semantic relatedness: For the semantic relatedness task, we used the SICK dataset (Marelli et al., 2014), and followed the standard split for the training (SICK train.txt), development (SICK trial.txt), and test (SICK test annotated.txt) sets. The evaluation metric is the Mean Squared Error (MSE) between the gold and predicted scores.
Textual entailment: For textual entailment, we also used the SICK dataset and exactly the same data split as the semantic relatedness dataset. The evaluation metric is the accuracy.
5.2 TRAINING DETAILS
Pre-training embeddings: We used the word2vec toolkit to pre-train the word embeddings. We created our training corpus by selecting lowercased English Wikipedia text and obtained 100- dimensional Skip-gram word embeddings trained with the context window size 1, the negative sampling method (15 negative samples), and the sub-sampling method (10−5 of the sub-sampling coefficient).4 We also pre-trained the character n-gram embeddings using the same parameter settings with the case-sensitive Wikipedia text. We trained the character n-gram embeddings for n = 1, 2, 3, 4 in the pre-training step.
4It is empirically known that such a small window size in leads to better results on syntactic tasks than large window sizes. Moreover, we have found that such word embeddings work well even on the semantic tasks.
Embedding initialization: We used the pre-trained word embeddings to initialize the word embeddings, and the word vocabulary was built based on the training data of the five tasks. All words in the training data were included in the word vocabulary, and we employed the word-dropout method (Kiperwasser & Goldberg, 2016) to train the word embedding for the unknown words. We also built the character n-gram vocabulary for n = 2, 3, 4, following Wieting et al. (2016), and the character n-gram embeddings were initialized with the pre-trained embeddings. All of the label embeddings were initialized with uniform random values in [− √ 6/(dim+ C), √ 6/(dim+ C)], where dim = 100 is the dimensionality of the label embeddings and C is the number of labels.
Weight initialization: The dimensionality of the hidden layers in the bi-LSTMs was set to 100. We initialized all of the softmax parameters and bias vectors, except for the forget biases in the LSTMs, with zeros, and the weight matrix Wd and the root node vector r for dependency parsing were also initialized with zeros. All of the forget biases were initialized with ones. The other weight matrices were initialized with uniform random values in [− √ 6/(row + col), √ 6/(row + col)], where row and col are the number of rows and columns of the matrices, respectively.
Optimization: At each epoch, we trained our model in the order of POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment. We used mini-batch stochastic gradient decent to train our model. The mini-batch size was set to 25 for POS tagging, chunking, and the SICK tasks, and 15 for dependency parsing. We used a gradient clipping strategy with growing clipping values for the different tasks; concretely, we employed the simple function: min(3.0, depth), where depth is the number of bi-LSTM layers involved in each task, and 3.0 is the maximum value. The learning rate at the k-th epoch was set to ε1.0+ρ(k−1) , where ε is the initial learning rate, and ρ is the hyperparameter to decrease the learning rate. We set ε to 1.0 and ρ to 0.3. At each epoch, the same learning rate was shared across all of the tasks.
Regularization: We set the regularization coefficient to 10−6 for the LSTM weight matrices, 10−5 for the weight matrices in the classifiers, and 10−3 for the successive regularization term excluding the classifier parameters of the lower-level tasks, respectively. The successive regularization coefficient for the classifier parameters was set to 10−2. We also used dropout (Hinton et al., 2012). The dropout rate was set to 0.2 for the vertical connections in the multi-layer bi-LSTMs (Pham et al., 2014), the word representations and the label embeddings of the entailment layer, and the classifier of the POS tagging, chunking, dependency parsing, and entailment. A different dropout rate of 0.4 was used for the word representations and the label embeddings of the POS, chunking, and dependency layers, and the classifier of the relatedness layer.
6 RESULTS AND DISCUSSION
6.1 SUMMARY OF MULTI-TASK RESULTS
Table 1 shows our results of the test sets on the five different tasks.5 The column “Single” shows the results of handling each task separately using single-layer bi-LSTMs, and the column “JMTall” shows the results of our JMT model. The single task settings only use the annotations of their own tasks. For example, when treating dependency parsing as a single task, the POS and chunking tags are not used. We can see that all results of the five different tasks are improved in our JMT model, which shows that our JMT model can handle the five different tasks in a single model. Our JMT model allows us to access arbitrary information learned from the different tasks. If we want to use the model just as a POS tagger, we can use the output from the first bi-LSTM layer. The output can be the weighted POS label embeddings as well as the discrete POS tags.
Table 1 also shows the results of three subsets of the different tasks. For example, in the case of “JMTABC”, only the first three layers of the bi-LSTMs are used to handle the three tasks. In the case of “JMTDE”, only the top two layers are used just as a two-layer bi-LSTM by omitting all information from the first three layers. The results of the closely-related tasks show that our JMT model improves not only the high-level tasks, but also the low-level tasks.
5The development and test sentences of the chunking dataset are included in the dependency parsing dataset, although our model does not explicitly use the chunking annotations of the development and test data. In such cases, we show the results in parentheses.
Table 2: POS tagging results.
Method MSE JMTall 0.233 JMTDE 0.238 Zhou et al. (2016) 0.243 Tai et al. (2015) 0.253
Table 5: Semantic relatedness results.
6.2 COMPARISON WITH PUBLISHED RESULTS
POS tagging: Table 2 shows the results of POS tagging, and our JMT model achieves the score close to the state-of-the-art results. The best result to date has been achieved by Ling et al. (2015), which uses character-based LSTMs. Incorporating the character-based encoders into our JMT model would be an interesting direction, but we have shown that the simple pre-trained character n-gram embeddings lead to the promising result.
Chunking: Table 3 shows the results of chunking, and our JMT model achieves the state-of-the-art result. Søgaard & Goldberg (2016) proposed to jointly learn POS tagging and chunking in different layers, but they only showed improvement for chunking. By contrast, our results show that the low-level tasks are also improved by the joint learning.
Dependency parsing: Table 4 shows the results of dependency parsing by using only the WSJ corpus in terms of the dependency annotations, and our JMT model achieves the state-of-the-art result.6 It is notable that our simple greedy dependency parser outperforms the previous state-ofthe-art result which is based on beam search with global information. The result suggests that the bi-LSTMs efficiently capture global information necessary for dependency parsing. Moreover, our single task result already achieves high accuracy without the POS and chunking information. Further analysis on our dependency parser can be found in Appendix B.
Semantic relatedness: Table 5 shows the results of the semantic relatedness task, and our JMT model achieves the state-of-the-art result. The result of “JMTDE” is already better than the previous state-of-the-art results. Both of Zhou et al. (2016) and Tai et al. (2015) explicitly used syntactic tree structures, and Zhou et al. (2016) relied on attention mechanisms. However, our method uses the simple max-pooling strategy, which suggests that it is worth investigating such simple methods before developing complex methods for simple tasks. Currently, our JMT model does not explicitly use the learned dependency structures, and thus the explicit use of the output from the dependency layer should be an interesting direction of future work.
6Choe & Charniak (2016) employed the tri-training technique to expand the training data with automatically-generated 400,000 trees in addition to the WSJ data, and they reported 95.9 UAS and 94.1 LAS.
Textual entailment: Table 6 shows the results of textual entailment, and our JMT model achieves the state-of-the-art result.7 The previous state-of-the-art result in Yin et al. (2016) relied on attention mechanisms and dataset-specific data pre-processing and features. Again, our simple max-pooling strategy achieves the state-of-the-art result boosted by the joint training. These results show the importance of jointly handling related tasks. Error analysis can be found in Appendix C.
6.3 ANALYSIS ON MULTI-TASK LEARNING ARCHITECTURES
Here, we first investigate the effects of using deeper layers for the five different single tasks. We then show the effectiveness of our training strategy: the successive regularization, the shortcut connections of the word representations, the embeddings of the output labels, the character n-gram embeddings, the use of the different layers for the different tasks, and the vertical connections of multi-layer bi-LSTMs. All of the results shown in this section are the development set results.
- Depth: The single task settings shown in Table 1 are obtained by using single layer bi-LSTMs, but in our JMT model, the higher-level tasks use successively deeper layers. To investigate the gap between the different number of the layers for each task, we also show the results of using multi-layer bi-LSTMs for the single task settings, in the column of “Single+” in Table 7. More concretely, we use the same number of the layers with our JMT model; for example, three layers are used for dependency parsing, and five layers are used for textual entailment. As shown in these results, deeper layers do not always lead to better results,
and the joint learning is more important than making the models complex only for single tasks.
- Successive regularization: In Table 8, the column of “w/o SR” shows the results of omitting the successive regularization terms described in Section 3. We can see that the accuracy of chunking is improved by the successive regularization, while other results are not affected so much. The chunking dataset used here is relatively small compared with other low-level tasks, POS tagging and dependency parsing. Thus, these results suggest that the successive regularization is effective when dataset sizes are imbalanced.
- Shortcut connections: Our JMT model feeds the word representations into all of the bi-LSTM layers, which is called the shortcut connection. Table 9 shows the results of “JMTall” with and without the shortcut connections. The results without the shortcut connections are shown in the column of “w/o SC”. These results clearly show that the importance of the shortcut connections in our JMT model, and in particular, the semantic tasks in the higher layers strongly rely on the shortcut connections. That is, simply stacking the LSTM layers is not sufficient to handle a variety of NLP tasks in a single model. In Appendix D, we show
how the shared word representations change according to each task (or layer).
- Output label embeddings: Table 10 shows the results without using the output labels of the POS, chunking, and relatedness layers, in the column of “w/o LE”. These results show that the explicit use of the output information from the classifiers of the lower layers is important in our JMT model. The results in the column of “w/o SC&LE” are the ones without both of the shortcut connections and the label embeddings.
7The result of “JMTall” is slightly worse than that of “JMTDE”, but the difference is not significant because the training data is small.
- Character n-gram embeddings: Table 11 shows the results for the three single tasks, POS tagging, chunking, and dependency parsing, with and without the pre-trained character n-gram embeddings. The column of “W&C” corresponds to using both of the word and character n-gram embeddings, and that of “Only W” corresponds to using only the word embeddings. These results clearly show that jointly using the pre-trained word and character n-gram embeddings is helpful in improving the results.
The pre-training of the character n-gram embeddings is also effective; for example, without the pre-training, the POS accuracy drops from 97.52% to 97.38% and the chunking accuracy drops from 95.65% to 95.14%, but they are still better than those of using word2vec embeddings alone. Further analysis can be found in Appendix A.
- Different layers for different tasks: Table 12 shows the results for the three tasks of our “JMTABC” setting and that of not using the shortcut connections and the label embeddings as in Table 10. In addition, in the column of “All-3”, we show the results of using the highest (i.e., the third) layer for all of the three tasks without any shortcut connections and label embeddings, and thus the two settings “w/o SC&LE” and “All-3” require exactly
the same number of the model parameters. The results show that using the same layers for the three different tasks hampers the effectiveness of our JMT model, and the design of the model is much more important than the number of the model parameters.
- Vertical connections: Finally, we investigated our JMT results without using the vertical connections in the five-layer bi-LSTMs. More concretely, when constructing the input vectors gt, we do not use the bi-LSTM hidden states of the previous layers. Table 13 shows the JMTall results with and without the vertical connections. As shown in the column of “w/o VC”, we observed the competitive results. Therefore, in the target tasks used in our model, sharing the word representations and the output label embeddings is more effective than just stacking the bi-LSTM layers.
7 CONCLUSION
We presented a joint many-task model to handle a variety of NLP tasks with growing depth of layers in a single end-to-end deep model. Our model is successively trained by considering linguistic hierarchies, directly connecting word representations to all layers, explicitly using predictions in lower tasks, and applying successive regularization. In our experiments on five different types of NLP tasks, our single model achieves the state-of-the-art results on chunking, dependency parsing, semantic relatedness, and textual entailment.
ACKNOWLEDGMENTS
We thank the Salesforce Research team members for their fruitful comments and discussions.
A DETAILS OF CHARACTER N -GRAM EMBEDDINGS
Here we first describe the pre-training process of the character n-gram embeddings in detail and then show further analysis on the results in Table 11.
A.1 PRE-TRAINING WITH SKIP-GRAM OBJECTIVE
We pre-train the character n-gram embeddings using the objective function of the Skip-gram model with negative sampling (Mikolov et al., 2013). We build the vocabulary of the character n-grams based on the training corpus, the case-sensitive English Wikipedia text. This is because such casesensitive information is important in handling some types of words like named entities. Assuming that a word w has its corresponding K character n-grams {cn1, cn2, . . . , cnK}, where any overlaps and unknown ones are removed. Then the wordw is represented with an embedding vc(w) computed as follows:
vc(w) = 1
K K∑ i=1 v(cni), (11)
where v(cni) is the parameterized embedding of the character n-gram cni, and the computation of vc(w) is exactly the same as the one used in our JMT model explained in Section 2.1.
The remaining part of the pre-training process is the same as the original Skip-gram model. For each word-context pair (w,w) in the training corpus, N negative context words are sampled, and the objective function is defined as follows:
∑ (w,w)
( − log σ(vc(w) · ṽ(w))−
N∑ i=1 log σ(−vc(w) · ṽ(wi))
) , (12)
where σ(·) is the logistic sigmoid function, ṽ(w) is the weight vector for the context word w, and wi is a negative sample. It should be noted that the weight vectors for the context words are parameterized for the words without any character information.
A.2 EFFECTIVENESS ON UNKNOWN WORDS
One expectation from the use of the character n-gram embeddings is to better handle unknown words. We verified this assumption in the single task setting for POS tagging, based on the results reported in Table 11. Table 14 shows that the joint use of the word and character n-gram embeddings improves the score by about 19% in terms of the accuracy for unknown words.
We also show the results of the single task setting for dependency parsing in Table 15. Again, we can see that using the character-level information is effective, and in particular, the improvement of the LAS score is large. These results suggest that it is better to use not only the word embeddings, but also the character n-gram embeddings by default. Recently, the joint use of word and character information has proven to be effective in language modeling (Miyamoto & Cho, 2016), but just using the simple character n-gram embeddings is fast and also effective.
B ANALYSIS ON DEPENDENCY PARSING
Our dependency parser is based on the idea of predicting a head (or parent) for each word, and thus the parsing results do not always lead to correct trees. To inspect this aspect, we checked the parsing results on the development set (1,700 sentences), using the “JMTABC” setting.
In the dependency annotations used in this work, each sentence has only one root node, and we have found 11 sentences with multiple root nodes and 11 sentences with no root nodes in our parsing results. We show two examples below:
(a) Underneath the headline “ Diversification , ” it counsels , “ Based on the events of the past week , all investors need to know their portfolios are balanced to help protect them against the market ’s volatility . ”
(b) Mr. Eskandarian , who resigned his Della Femina post in September , becomes chairman and chief executive of Arnold .
In the example (a), the two boldfaced words “counsels” and “need” are predicted as child nodes of the root node, and the underlined word “counsels” is the correct one based on the gold annotations. This example sentence (a) consists of multiple internal sentences, and our parser misunderstood that both of the two verbs are the heads of the sentence.
In the example (b), none of the words is connected to the root node, and the correct child node of the root is the underlined word “chairman”. Without the internal phrase “who resigned... in September”,
the example sentence (b) is very simple, but we have found that such a simplified sentence is still not parsed correctly. In many cases, verbs are linked to the root nodes, but sometimes other types of words like nouns can be the candidates. In our model, the single parameterized vector r is used to represent the root node for each sentence. Therefore, the results of the examples (a) and (b) suggest that it would be needed to capture various types of root nodes, and using sentence-dependent root representations would lead to better results in future work.
C ANALYSIS ON SEMANTIC TASKS
We inspected the development set results on the semantic tasks using the “JMTall” setting. In our model, the highest-level task is the textual entailment task. We show an example premise-hypothesis pair which is misclassified in our results:
Premise: “A surfer is riding a big wave across dark green water”, and
Hypothesis: “The surfer is riding a small wave”.
The predicted label is entailment, but the gold label is contradiction. This example is very easy by focusing on the difference between the two words “big” and “small”. However, our model fails to correctly classify this example because there are few opportunities to learn the difference. Our model relies on the pre-trained word embeddings based on word co-occurrence statistics (Mikolov et al., 2013), and it is widely known that such co-occurrence-based embeddings can rarely discriminate between antonyms and synonyms (Ono et al., 2015). Moreover, the other four tasks in our JMT model do not explicitly provide the opportunities to learn such semantic aspects. Even in the training data of the textual entailment task, we can find only one example to learn the difference between the two words, which is not enough to obtain generalization capacities. Therefore, it is worth investigating the explicit use of external dictionaries or the use of pre-trained word embeddings learned with such dictionaries (Ono et al., 2015), to see whether our JMT model is further improved not only for the semantic tasks, but also for the low-level tasks.
D HOW DO SHARED EMBEDDINGS CHANGE
In our JMT model, the word and character n-gram embedding matrices are shared across all of the five different tasks. To better qualitatively explain the importance of the shortcut connections shown in Table 9, we inspected how the shared embeddings change when fed into the different biLSTM layers. More concretely, we checked closest neighbors in terms of the cosine similarity for the word representations before and after fed into the forward LSTM layers. In particular, we used the corresponding part of Wu in Eq. (1) to perform linear transformation of the input embeddings, because ut directly affects the hidden states of the LSTMs. Thus, this is a context-independent analysis.
Table 16 shows the examples of the word “standing”. The row of “Embedding” shows the cases of using the shared embeddings, and the others show the results of using the linear-transformed embeddings. In the column of “Only word”, the results of using only the word embeddings are shown. The closest neighbors in the case of “Embedding” capture the semantic similarity, but after fed into the POS layer, the semantic similarity is almost washed out. This is not surprising because it is sufficient to cluster the words of the same POS tags: here, NN, VBG, etc. In the chunking layer, the similarity in terms of verbs is captured, and this is because it is sufficient to identify the coarse chunking tags: here, VP. In the dependency layer, the closest neighbors are adverbs, gerunds of verbs, and nouns, and all of them can be child nodes of verbs in dependency trees. However, this information is not sufficient in further classifying the dependency labels. Then we can see that in the column of “Word and char”, jointly using the character n-gram embeddings adds the morphological information, and as shown in Table 11, the LAS score is substantially improved.
In the case of semantic tasks, the projected embeddings capture not only syntactic, but also semantic similarities. These results show that different tasks need different aspects of the word similarities, and our JMT model efficiently transforms the shared embeddings for the different tasks by the simple linear transformation. Therefore, without the shortcut connections, the information about the word representations are fed into the semantic tasks after transformed in the lower layers where the
semantic similarities are not always important. Indeed, the results of the semantic tasks are very poor without the shortcut connections.

The paper introduce a way to train joint models for many NLP tasks. Traditionally, we treat these tasks as “pipeline” — the later tasks will depending on the output of the previous tasks. Here, the authors propose a neural approach which includes all the tasks in one single model. The higher level tasks takes (1) the predictions from the lower level tasks and (2) the hidden representations of the lower level tasks. Also proposed in this paper, is the successive regularization. Intuitively, this means that, when training the high level tasks, we don’t want to change the model in the lower levels by too much so that the lower level tasks can keep a reasonable accuracy of prediction.
On the modeling side, I think the proposed model is very similar comparing to (Zhang and Weiss, ACL 2016) and SPINN (Bowman et al, 2016) in a even simpler way. The number of the experiments are good. But I am not sure I am convinced by the numbers in Table 1 since the patterns are not very clear there — sometimes, the performance of the higher level tasks even goes down when training with more tasks (sometimes it does go up, but also not very significant and stable). The dependency scores, although I don’t think this is a serious problem, comparing the UAS/LAS when the output is not guaranteed to be a well-formed tree isn’t strictly speaking fair.
I admit that the successive regularization make sense intuitively and is a very interesting direction to try. However, without a careful study of the training schema of such model, the current results on successive regularization do not convince me that it should be the right thing to do in such models (the current results are not strong enough to show that). The training methods need to be explored here including things as iteratively train on different tasks, and the relationship between the number of training iterations of a task and it’s training set size (and loss on this task etc).


A NEURAL KNOWLEDGE LANGUAGE MODEL
Current language models have significant limitations in their ability to encode and decode factual knowledge. This is mainly because they acquire such knowledge based on statistical co-occurrences, even if most of the knowledge words are rarely observed named entities. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by a knowledge graph with the RNN language model. The model predicts whether the word to generate has an underlying fact or not. Then, a word is either generated from the vocabulary or copied from the description of the predicted fact. We train and test the model on a new dataset, WikiFacts. In experiments, we show that the NKLM significantly improves the perplexity while generating a much smaller number of unknown words. In addition, we demonstrate that the sampled descriptions include named entities which used to be the unknown words in RNN language models.
1 INTRODUCTION
Kanye West, a famous <unknown> and the husband of <unknown>, released his latest album <unknown> in <unknown>.
A core purpose of language is to communicate knowledge. Thus, for human-level language understanding, it is important for a language model to take advantage of knowledge. Although traditional language models are good at capturing statistical co-occurrences of entities as long as they are observed frequently in a corpus (e.g., words like verbs, pronouns, and prepositions), they are in general limited in their ability to encode or decode knowledge, which is often represented by named entities such as person names, place names, years, etc. (as shown in the above example sentence of Kanye West.) When trained with a very large corpus, traditional language models have demonstrated to some extent the ability to encode/decode knowledge (Vinyals & Le, 2015; Serban et al., 2015). However, we claim that simply feeding a larger corpus into a bigger model hardly results in a good knowledge language model.
The primary reason for this is the difficulty in learning good representations for rare or unknown words because these are a majority of the knowledge-related words. In particular, for applications such as question answering (Iyyer et al., 2014; Weston et al., 2016; Bordes et al., 2015) and dialogue modeling (Vinyals & Le, 2015; Serban et al., 2015), these words are of our main interest. Specifically, in the recurrent neural network language model (RNNLM) (Mikolov et al., 2010) the computational complexity is linearly dependent on the number of vocabulary words. Thus, including all words of a language is computationally prohibitive. Instead, we typically fill our vocabulary with a limited number of frequent words and regard all the other words as the unknown (UNK) word. Even if we can include a large number of words in the vocabulary, according to Zipf’s law, a large portion of the words will be rarely observed in the corpus and thus learning good representations for these words remains a problem.
The fact that languages and knowledge can change over time also makes it difficult to simply rely on a large corpus. Media produce an endless stream of new knowledge every day (e.g., the results of baseball games played yesterday) that is even changing over time (e.g., “the current president of the
∗This work was done while HC was in Samsung Advanced Institute of Technology
United States is ”). Furthermore, a good language model should exercise some level of reasoning. For example, it may be possible to observe several occurrences of Barack Obama’s year of birth in a large corpus and thus the model may be able to predict it. However, after seeing mentions of his year of birth, presented with a simple reformulation of that piece of knowledge into a sentence such as “Barack Obama’s age is ”, one would not expect current language models to handle the required amount of reasoning in order to predict the next word (i.e. the age) easily. However, a good model should be able to reason the answer from this context1.
In this paper, we propose a Neural Knowledge Language Model (NKLM) as a step towards addressing the limitations of traditional language modeling when it comes to exploiting factual knowledge. In particular, we incorporate symbolic knowledge provided by a knowledge graph (Nickel et al., 2015) into the RNNLM. A knowledge graph (KG) is a collection of facts which have a form of (subject, relationship, object). We observe particularly the following properties of KGs that make the connection to the language model sensible. First, facts in KGs are mostly about rare words in text corpora. KGs are managed and updated in a similar way that Wikipedia pages are managed to date. The KG embedding methods (Bordes et al., 2011; 2013) provide distributed representations for the entities in the KG. The graph can be traversed for reasoning (Gu et al., 2015). Finally, facts come along with textual representations which we call the fact description and take advantage of here.
There are a few differences between the NKLM and the traditional RNNLM. First, we assume that a word generation is either based on a fact or not. Thus, at each time step, before predicting a word, we predict whether the word to generate has an underlying fact or not. As a result, our model provides the predictions over facts in a topic in addition to the word predictions. Similarly to how context information of previous words flows through the hidden states in the RNNLM, in the NKLM the previous information on both facts and words flow through an RNN and provide richer context. Second, the model has two ways to generate the next word. One option is to generate a “vocabulary word” from the vocabulary softmax as is in the RNNLM. The other option is to generate a “knowledge word” by copying a word contained in the description of the predicted fact. Considering that the fact description is often short and consists of out-of-vocabulary words, we predict the position of the word to copy within the fact description. This knowledge-copy mechanism makes it possible to generate words which are not in the predefined vocabulary. Thus, it does not require to learn explicit embeddings of the words to generate, and consequently resolves the rare/unknown word problem. Lastly, the NKLM can immediately adapt to adding or modifying knowledge because the model learns to predict facts, which can easily be modified without having to retrain the model.
Training the above model in a supervised way requires to align words with facts. To this end, we introduce a new dataset, called WikiFacts. For each topic in the dataset, a set of facts from the Freebase KG (Bollacker et al., 2008) and a Wikipedia description of the same topic is provided along with the alignment information. This alignment is done automatically by performing string matching between the fact description and the Wikipedia description.
2 RELATED WORK
There have been remarkable advances in language modeling research based on neural networks (Bengio et al., 2003; Mikolov et al., 2010). In particular, the RNNLMs are interesting for their ability to take advantage of longer-term temporal dependencies without a strong conditional independence assumption. It is especially noteworthy that the RNNLM using the Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) has recently advanced to the level of outperforming carefully-tuned traditional n-gram based language models (Jozefowicz et al., 2016).
There have been many efforts to speed up the language models so that they can cover a larger vocabulary. These methods approximate the softmax output using hierarchical softmax (Morin & Bengio, 2005; Mnih & Hinton, 2009), importance sampling (Jean et al., 2015), noise contrastive estimation (Mnih & Teh, 2012), etc. Although helpful to mitigate the computational problem, these approaches still suffer from the statistical problem due to rare or unknown words. Having the UNK word as the output of a generative language model is also inconvenient (e.g, dialogue system).
1We do not investigate the reasoning ability in this paper but highlight this example because the explicit representation of facts would help to handle such examples.
To help deal with the rare/unknown word problem, the pointer networks (Vinyals et al., 2015) have been adopted to implement the copy mechanism (Gulcehre et al., 2016; Gu et al., 2016) and applied to machine translation and text summarization. With this approach, the (unknown) word to copy from the context sentence is inferred from neighboring words. However, because in our case the context can be very short and often contains no known relevant words (e.g., person names), we cannot use the existing approach directly.
Our knowledge memory is also related to the recent literature on neural networks with external memory (Bahdanau et al., 2014; Weston et al., 2015; Graves et al., 2014). In Weston et al. (2015), given simple sentences as facts which are stored in the external memory, the question answering task is studied. In fact, the tasks that the knowledge-based language model aims to solve (i.e. predict the next word) can be considered as a fill-in-the-blank type of question answering. The idea of jointly using Wikipedia and knowledge graphs has also been used in the context of enriching word embedding (Celikyilmaz et al., 2015; Long et al., 2016).
3 MODEL
3.1 PRELIMINARY
A topic2 k in a set of entities E is associated with topic knowledge Fk (e.g., from Freebase) and topic description Wk (e.g., from Wikipedia). Topic knowledge Fk is a set of facts {ak,1, ak,2, . . . , ak,|Fk|} where each fact a is a triple of subject ∈ E , relationship, and object ∈ E , e.g., (Barack Obama, Married-To, Michelle Obama). Topic description Wk is a sequence of words (wk1 , w k 2 , . . . , w k |Wk|) describing the topic (e.g., a description of a topic in Wikipedia). Because the subject entities in Fk are all equal to the topic entity k3 and the words describing relationships can easily be found in the vocabulary, we use the description of the object entity (e.g., Michelle Obama) as our fact description.
Given Fk and Wk, we perform simple string matching between words in Wk and words in the fact descriptions in Fk and thereby build a sequence of augmented observations Yk = {ykt = (wt, at, zt)}t=1:|Wk|. Here, wt ∈ Wk is an observed word, at ∈ Fk a fact on which the generated word wt is based, and zt a binary variable indicating whether wt is in the vocabulary V (including UNK) or not. Because not all words are based on a fact (e.g., words like, is, a, the, have), we introduce a special type of fact, called Not-a-Fact (NaF), and assign NaF to such words.
For example, a description “Rogers was born in Latrobe, Pennsylvania in 1928” from a topic Fred Rogers in Wikipedia, is augmented to, Y = {(w=“Rogers”, a=0, z=0), (“was”, NaF, 1), (“born”, NaF, 1), (“in”, NaF, 1), (“Latrobe”, 42, 0), (“Pennsylvania”, 42, 1), (“in”, NaF, 1), (“1928”, 83, 0)}. Here, we use facts on Fred Rogers, a42 = (Fred Rogers, Place of Birth, Latrobe Pennsylvania), a83 = (Fred Rogers, Year of Birth, 1928), and a special fact a0 = (Fred Rogers, Topic Itself, Fred Rogers) which we define in order to refer to the topic string itself. We also assume here that the words Rogers, Latrobe and 1928 are not in the vocabulary.
During the inference and training of topic k, we assume that the topic knowledge Fk is loaded in the knowledge memory in a form of a matrix Fk ∈ RDa×|Fk| where the i-th column is a fact embedding ak,i ∈ RDa . The fact embedding is the concatenation of subject, relationship, and object embeddings. We obtain these entity embeddings from a preliminary run of a knowledge graph embedding method such as TransE (Bordes et al., 2013). Note that we fix the fact embedding during the training of our model to help the model predict new facts at test time. But, we learn the embedding of the Topic Itself. For notation, to denote the vector representation of any object of our interest, we use bold lowercase characters. For example, the embedding of a word wt is represented by wt = W[wt] where WDw×|V| is the word embedding matrix, and W[wt] denotes the wt-th column of W.
2In this work, a topic is one of the entities which exist in both Wikipedia and Freebase. This is different to the concept in topic modeling where a topic is represented by a distribution over words.
3Although in Freebase the topic entity can be either the subject or the object, for convenience we process them such that the subject is always equal to the topic entity k.
a1 a2 a3 a4 … aN NaF
𝑤𝑡𝑜
O1 O2 O3 O4 … ON
Topic Knowledge
𝑤𝑡- 𝑧/
ℎ/ ℎ/𝑎/
𝑘/
𝑎/34 𝑤𝑡34- 𝑤𝑡345
ℎ/34 ℎ/ 𝑒 𝑥/
Figure 1: The NKLM model. The input consisting of a word (either wot−1 or w v t−1) and a fact (at−1) goes into LSTM. The LSTM’s output ht together with the knowledge context e generates the fact key kt. Using the fact key, the fact embedding at is retrieved from the topic knowledge memory. Using at and ht, knowledge-copy switch zt is determined, which in turn determines the next word generation source wvt or w o t . The copied word w o t is a symbol taken from the fact description Oat .
3.2 INFERENCE
At each time step, the NKLM follows four sub-steps. First, using both the word and fact outputs from the previous time step as the input of the current time step, we update the LSTM controller. Second, given the output of the LSTM, the NKLM predicts a fact (including NaF) and extracts corresponding fact embedding from the knowledge memory. Thirdly, with the extracted fact and the state of the LSTM controller, the NKLM makes a binary decision to choose the source of word generation. Finally, a word is generated according to the chosen source. A model diagram is depicted in Fig. 1. In the following, we describe these four steps in more detail.
1) Input Representation and LSTM Controller. As shown in Fig. 1, the input at time step t is the concatenation of three embedding vectors corresponding to a fact at−1, a vocabulary word wvt−1, and a copied word wot−1, all predicted in the previous time step. However, because at a time step, the predicted word comes only either from the vocabulary or by copying from the fact description, we set either wvt−1 or w o t−1 to a zero vector when it is not selected in the previous step. As we shall see, we use position embeddings to represent the copied words by its position within the fact description. And, because the dimensions of the vocabulary word embedding and the position embedding for copied words are different, we use such concatenation of wvt−1 and w o t−1 to represent the word input. The resulting input representation xt = fconcat(at−1,wvt−1,w o t−1) is then fed into the LSTM controller, and obtain the output states (ht, ct) = fLSTM(xt,ht−1). Note that at−1 and wot−1 (e.g., corresponding to n-th position) together can deliver information that a symbol in n-th position in the description of fact at−1 was used in the previous time step.
2) Fact Extraction. Then, we predict a relevant fact at on which the word wt will be based. If the word wt is supposed to be irrelevant to any fact, the NaF type is predicted. Unlike the fact embeddings, we learn the NaF embedding during training.
Predicting a fact is done in two steps. First, a fact-key kfact ∈ RDa is generated by kfact = ffactkey(ht, ek). Here, ek ∈ RDa is the topic context embedding (or a subgraph embedding of the topic) which encodes information about what facts are available in the knowledge memory so that the key generator adapts to changes in the knowledge memory. For example, if we remove a fact from the memory, without retraining, the fact-key generator should be aware of the absence of that information and thus should not generate a key vector for the removed fact. Although, in the experiments, we use mean-pooling (average of the all fact embeddings in the knowledge memory) to obtain ek, one can also consider using the soft-attention mechanism (Bahdanau et al., 2014). For the fact-key generator ffactkey, we use an MLP with one hidden layer of ReLU nonlinearity.
Then, using the generated fact-key kfact, we perform key-value lookup over the knowledge memory Fk to predict a fact and retrieve its embedding at,
P (at|ht) = exp(k>factFk[at])∑ a′ exp(k > factFk[a ′]) , (1)
at = argmax at∈Fk
P (at|ht), (2)
at = Fk[at]. (3)
Note that in order to perform the copy mechanism, we need to pick a single fact from the knowledge memory instead of using the weighted average of the fact embeddings as in the soft-attention.
3) Knowledge-Copy Switch. Given the encoding of the context ht and the embedding of the extracted fact at, the model decides the source for the next word generation: either from the vocabulary or from the fact description by copy. As zt = 1 if the word wt is in the vocabulary, we define the probability of selecting copy as:
ẑt = p(1− zt|ht) = sigmoid(fcopy(ht,at)). (4) Here, fcopy is an MLP with one ReLU hidden layer and a single linear output unit. For facts about attributes such as nationality or profession, the words in the fact description (e.g., “American” or “actor”) are likely to be in the vocabulary, but for facts like the year of birth or father name, the model is likely to choose to copy.
4) Word Generation. Word wt is generated from the source indicated by the copy-switch ẑt as follows:
wt = { wvt ∈ V, if ẑt < 0.5, wot ∈ Oat , otherwise.
For vocabulary word wvt ∈ V , we use the softmax function where each output dimension corresponds to a word in the vocabulary including UNK,
P (wvt = w|ht) = exp(k>vocaW[w])∑
w′∈V exp(k > vocaW[w
′]) . (5)
where kvoca ∈ RDw is obtained by fvoca(ht,at) which is an MLP with a ReLU hidden layer and linear output units of dimension Dw.
For knowledge word wot ∈ Oat , we predict the position of the word in the fact description and then copy the word on the predicted position to output. This is because, unlike with the traditional copy mechanism, our context words (i.e., the fact description) often consist of all unknown words and/or are short in length. Copying allows us not to rely on the word embeddings for the knowledge words. Instead, we learn the position embeddings shared among all knowledge words. This makes sense because words in the fact description usually appear one by one in increasing order. Thus, given that the first symbol o1 = “Michelle” was used in the previous time step and prior to that other words such as “President” and “US” were also observed, the model can easily predict that it is time to select the second symbol, i.e., o2 = “Obama”.
For this copy-by-position, we first generate the position key kpos ∈ RDo by a function fposkey(ht,at) which is again an MLP with one hidden layer and linear outputs whose dimension is equal to the maximum length of the fact descriptions Nomax = maxa∈F |Oa| where F = ∪kFk. Then, the n-th symbol on ∈ Oat is chosen by
P (wot = on|ht, at) = exp(k>posP[n])∑ n′ exp(k > posP[n ′]) , (6)
with n′ running from 0 to |Oat | − 1. Here, PDo×N o max is the position embedding matrix. Note that Nomax is typically a much smaller number (e.g., 20 in our experiments) than the size of vocabulary. The position embedding matrix P is learned during training.
Although in this paper we find that the simple position prediction performs well, we note that one could also consider a more advanced encoding such as one based on a convolutional network (Kim, 2014) to model the fact description. At test time, to compute p(wkt |wk<t), we can obtain {zk<t, ak<t} from {wk<t} and Fk using the automatic labeling script, and perform the above inference process with hard decisions taken about zt and at based on the model’s predictions.
3.3 LEARNING
Given word observations {Wk}Kk=1 and knowledge {Fk}Kk=1, our objective is to maximize the log-likelihood of the observed words w.r.t the model parameter θ,
θ∗ = argmax θ ∑ k logPθ(Wk|Fk). (7)
Because, given Wk and Fk, a sequence of Yk = {yt = (wt, zt, at)}t=1:|Wk| is deterministically induced for each word wt, the following equality is satisfied
Pθ(Wk|Fk) = Pθ(Yk|Fk). (8)
By the chain rule, we can decompose the probability of the observation Yk as
logPθ(Yk|Fk) = |Yk|∑ t=1 logPθ(y k t |yk1:t−1,Fk). (9)
Then, after omitting Fk and k for simplicity, we can rewrite the single step conditional probability as
Pθ(yt|y1:t−1) = Pθ(wt, at, zt|ht) = Pθ(wt|at, zt, ht)Pθ(at|ht)Pθ(zt|ht). (10)
We maximize the above objective using stochastic gradient optimization.
4 EVALUATION
4.1 WIKIFACTS DATASET
An obstacle in developing the above model is the lack of the dataset where the text corpus is aligned with facts at the word level. To this end, we produced the WikiFacts dataset by aligning Wikipedia descriptions with corresponding Freebase facts. Because many Freebase topics provide a link to its corresponding topic in Wikipedia, we choose a set of topics for which both a Freebase entity and a Wikipedia description exist. In the experiments, we used a version called WikiFacts-FilmActor-v0.1 where the domain is restricted to the /Film/Actor in Freebase.
For all object entity descriptions {Oak} associated with Fk, we performed string matching to the Wikipedia description Wk. We used the summary part (first few paragraphs) of the Wikipedia page as text to be modeled but discarded topics for which the number of facts is greater than 1000 or the Wikipedia description is too short (< 3 sentences). For the string matching, we also used the synonyms and alias provided by WordNet (Miller, 1995) and Freebase.
We augmented the fact set Fk with the anchor facts Ak whose relationship is all set to UnknownRelation. That is, observing that an anchor (words under hyperlink) in Wikipedia descriptions has a corresponding Freebase entity as well as being semantically closely related to the topic in which the anchor is found, we make a synthetic fact of the form (Topic, UnknownRelation, Anchor). This potentially compensates for some missing facts in Freebase. Because we extract the anchor facts from the full Wikipedia page and they all share the same relation, it is more challenging for the model to use these anchor facts than using the Freebase facts. As a result, for each word w in the dataset, we have a tuple (w, zw, aw, kw). Here, kw is the topic where w appears. We provide a summary of the dataset statistics in Table 1. The dataset will be available on a public webpage4.
4.2 EXPERIMENTS
Setup. We split the dataset into 80/10/10 for train, validation, and test. As a baseline model, we use the RNNLM. For both the NKLM and the RNNLM, two-layer LSTMs with dropout regularization (Zaremba et al., 2014) are used. We tested models with different numbers of LSTM hidden units [200, 500, 1000], and report results from the 1000 hidden-unit model. For the NKLM, we set the symbol embedding dimension to 40 and word embedding dimension to 400. Under this setting, the number of parameters in the NKLM is slightly smaller than that of the RNNLM. We used
4https://bitbucket.org/skaasj/wikifact_filmactor
100-dimension TransE embeddings for Freebase entities and relations, and concatenate the relation and object embeddings to obtain fact embeddings. We averaged all fact embeddings in Fk to obtain the topic context embedding ek. We unrolled the LSTMs for 30 steps and used minibatch size 20. We trained the models using stochastic gradient ascent with gradient clipping range [-5,5]. The initial learning rate was set to 0.5 for the NKLM and 1.5 for the RNNLM, and decayed after every epoch by a factor of 0.98. We trained for 50 epochs and report the results chosen by the best validation set results.
Evaluation metric. The perplexity exp(− 1N ∑N i=1 log pwi) is the standard performance metric for language modeling. This, however, has a problem in evaluating language models for a corpus containing many named entities: a model can get good perplexity by accurately predicting UNK words. As an extreme example, when all words in a sentence are unknown words, a model predicting everything as UNK will get a good perplexity. Considering that unknown words provide virtually no useful information, this is clearly a problem in tasks such as question answering, dialogue modeling, and knowledge language modeling.
To this end, we introduce a new evaluation metric, called the Unknown-Penalized Perplexity (UPP), and evaluate the models on this metric as well as the standard perplexity (PPL). Because the actual word underlying the UNK should be one of the out-of-vocabulary (OOV) words, in UPP, we penalize the likelihood of unknown words as follows:
PUPP(wunk) = P (wunk)/|Vtotal \ Vvoca|. Here, Vtotal is a set of all unique words in the corpus, and Vvoca is the vocabulary used in the softmax. In other words, in UPP we assume that the OOV set is equal to |Vtotal\Vvoca| and thus assign a uniform probability to OOV words. In another version, UPP-fact, we consider the fact that the RNNLM can also use the knowledge given to the NKLM to some extent, but with limited capability (because the model is not designed for it). For this, we assume that the OOV set is equal to the total knowledge vocabulary of a topic k, i.e., PUPP-fact(wunk) = P (wunk)/|Ok|, where Ok = ∪iOak,i . In other words, by using UPP-fact, we assume that, for an unknown word, the RNNLM can pick one of the knowledge words with uniform probability. We describe the detail results and discussion on the experiments in the captions of Table 2, 3, and 4.
Observations from the experiment results. Our observations from the experiment results are as follows. (a) The NKLM outperforms the RNNLM in all three perplexity measures. (b) The copy mechanism is the key of the significant performance improvement. Without the copy mechanism, the NKLM still performs better than the RNNLM due to its usage of the fact information, but the improvement is not so significant. (c) The NKLM results in a much smaller number of UNKs (roughly, a half of the RNNLM). (d) When no knowledge is available, the NKLM performs as well as the
RNNLM. (e) KG embedding using TransE is an efficient way to initialize the fact embeddings. (f) The NKLM generates named entities in the provided facts whereas the RNNLM generates many more UNKs. (g) The NKLM shows its ability to adapt immediately to the change of the knowledge. (h) The standard perplexity is significantly affected by the prediction accuracy on the unknown words. Thus, one need carefully consider it as a metric for knowledge-related language models.
5 CONCLUSION
In this paper, we presented a novel Neural Knowledge Language Model (NKLM) that brings the symbolic knowledge from a knowledge graph into the expressive power of RNN language models. The
NKLM significantly outperforms the RNNLM in terms of perplexity and generates named entities which are not observed during training, as well as immediately adapting to changes in knowledge. We believe that the WikiFact dataset introduced in this paper, can be useful in other knowledge-related language tasks as well. In addition, the Unknown-Penalized Perplexity introduced in this paper in order to resolve the limitation of the standard perplexity, can be useful in evaluating other language tasks. The task that we investigated in this paper is limited in the sense that we assume that the true topic of a given description is known. Relaxing this assumption by making the model search for proper topics on-the-fly will make the model more practical. We believe that there are many more open research challenges related to the knowledge language models.
ACKNOWLEDGMENTS
The authors would like to thank Alberto Garcı́a-Durán, Caglar Gulcehre, Chinnadhurai Sankar, Iulian Serban and Sarath Chandar for feedback and discussions as well as the developers of Theano (Bastien et al., 2012), NSERC, CIFAR, Samsung and Canada Research Chairs for funding, and Compute Canada for computing resources.
APPENDIX: HEATMAPS

This paper addresses the practical problem of generating rare or unseen words in the context of language modeling. Since language follows a Zipf’s law, most approaches limit the vocabulary (because of computation reasons) and hence rare words are often mapped to a UNK token. Rare words are especially important in context of applications such as question answering. MT etc. This paper proposes a language modeling technique which incorporates facts from knowledge bases (KBs) and thus has the ability to generate (potentially unseen) words from KBs. This paper also releases a dataset by aligning words with Freebase facts and corresponding Wikipedia descriptions.
The model first selects a KB fact based on the previously generated words and facts. Based on the selected fact, it then predicts whether to generate a word based on the vocabulary or to output a symbolic word from the KB. For the latter, the model is trained to predict the position of the word from the fact description.
Overall the paper could use some rewriting especially the notations in section 3. The experiments are well executed and they definitely get good results. The heat maps at the end are very insightful.
Comments
This contributions of this paper would be much stronger if it showed improvements in a practical applications such as Question Answering (although the paper clearly mentions that this technique could be applied to improve QA)
In section 3, it is unclear why the authors refer the entity as a ‘topic'. This makes the text a little confusing since a topic can also be associated with something abstract, but in this case the topic is always a freebase entity.
Is it really necessary to predict a fact at every step before generating a word. In other words, how many distinct facts on average does the model choose to generate a sentence. Intuitively a natural language sentence would be describe few facts about an entity. If the fact generation step could be avoided (by adding a latent variable which decides if the fact should be generated or not), the model will also be faster.
In equation 2, the model has to make a hard decision to choose the fact. For this to be end to end trained, every word needs to be annotated with a corresponding fact which might not be always a realistic scenario. For e.g., in domains such as social media text.
Learning position embeddings for copying knowledge words seems a little counter-intuitive. Does the sequence of knowledge words follow any particular structure like word O_2 is always the last name (e.g. Obama).
It would also be nice to compare to char-level LM's which inherently solves the unknown token problem.