We propose a language-agnostic way of automatically generating sets of semantically similar clusters of entities along with sets of “outlier” elements, which may then be used to perform an intrinsic evaluation of word embeddings in the outlier detection task. We used our methodology to create a gold-standard dataset, which we call WikiSem500, and evaluated multiple state-of-the-art embeddings. The results show a correlation between performance on this dataset and performance on sentiment analysis.
1 INTRODUCTION
High quality datasets for evaluating word and phrase representations are essential for building better models that can advance natural language understanding. Various researchers have developed and shared datasets for syntactic and semantic intrinsic evaluation. The majority of these datasets are based on word similarity (e.g., Finkelstein et al. (2001); Bruni et al. (2012); Hill et al. (2016)) and analogy tasks (e.g., Mikolov et al. (2013a;b)). While there has been a significant amount of work in this area which has resulted in a large number of publicly available datasets, many researchers have recently identified problems with existing datasets and called for further research on better evaluation methods (Faruqui et al., 2016; Gladkova et al., 2016; Hill et al., 2016; Avraham & Goldberg, 2016; Linzen, 2016; Batchkarov et al., 2016). A significant problem with word similarity tasks is that human bias and subjectivity result in low inter-annotator agreement and, consequently, human performance that is lower than automatic methods (Hill et al., 2016). Another issue is low or no correlation between intrinsic and extrinsic evaluation metrics (Chiu et al., 2016; Schnabel et al., 2015).
Recently, Camacho-Collados & Navigli (2016) proposed the outlier detection task as an intrinsic evaluation method that improved upon some of the shortcomings of word similarity tasks. The task builds upon the “word intrusion” task initially described in Chang et al. (2009): given a set of words, the goal is to identify the word that does not belong in the set. However, like the vast majority of existing datasets, this dataset requires manual annotations that suffer from human subjectivity and bias, and it is not multilingual.
Inspired by Camacho-Collados & Navigli (2016), we have created a new outlier detection dataset that can be used for intrinsic evaluation of semantic models. The main advantage of our approach is that it is fully automated using Wikidata and Wikipedia, and it is also diverse in the number of included topics, words and phrases, and languages. At a high-level, our approach is simple: we view Wikidata as a graph, where nodes are entities (e.g., 〈Chicago Bulls, Q128109〉, 〈basketball team, Q13393265〉), edges represent “instance of” and “subclass of” relations (e.g., 〈Chicago Bulls, Q128109〉 is an instance of 〈basketball team, Q13393265〉, 〈basketball team, Q13393265〉 is a subclass of 〈sports team, Q12973014〉), and the semantic similarity between two entities is inversely proportional to their graph distance (e.g., 〈Chicago Bulls, Q128109〉 and 〈Los Angeles Lakers, Q121783〉 are semantically similar since they are both instance of 〈basketball team, Q13393265〉). This way we can form semantic clusters
by picking entities that are members of the same class, and picking outliers with different notions of dissimilarity based on their distance from the cluster entities.
We release the first version of our dataset, which we call WikiSem500, to the research community. It contains around 500 per-language cluster groups for English, Spanish, German, Chinese, and Japanese (a total of 13,314 test cases). While we have not studied yet the correlation between performance on this dataset and various downstream tasks, our results show correlation with sentiment analysis. We hope that this diverse and multilingual dataset will help researchers to advance the state-of-the-art of word and phrase representations.
2 RELATED WORK
Word similarity tasks have been popular for evaluating distributional similarity models. The basic idea is having annotators assigning similarity scores for word pairs. Models that can automatically assign similarity scores to the same word pairs are evaluated by computing the correlation between their and the human assigned scores. Schnabel et al. (2015) and Hill et al. (2016) review many of these datasets. Hill et al. (2016) also argue that the predominant gold standards for semantic evaluation in NLP do not measure the ability of models to reflect similarity. Their main argument is that many such benchmarks measure association and relatedness and not necessarily similarity, which limits their suitability for a wide range of applications. One of their motivating examples is the word pair “coffee” and “cup,” which have high similarity ratings in some benchmarks despite not being very similar. Consequently, they developed guidelines that distinguish between association and similarity and used five hundred Amazon Mechanical Turk annotators to create a new dataset called SimLex-999, which has higher inter annotator agreement than previous datasets. Avraham & Goldberg (2016) improved this line of work further by redesigning the annotation task from rating scales to ranking, in order to alleviate bias, and also redefined the evaluation measure to penalize models more for making wrong predictions on reliable rankings than on unreliable ones.
Another popular task based on is word analogies. The analogy dataset proposed by Mikolov et al. (2013a) has become a standard evaluation set. The dataset contains fourteen categories, but only about half of them are for semantic evaluation (e.g. “US Cities”, “Common Capitals”, “All Capitals”). In contrast, WikiSem500 contains hundreds of categories, making it a far more diverse and challenging dataset for the general-purpose evaluation of word representations. The Mikolov dataset has the advantage of additionally including syntactic categories, which we have left for future work.
Camacho-Collados & Navigli (2016) addressed some of the issues mentioned previously by proposing the outlier detection task. Given a set of words, the goal is to identify the word that does not belong in the set. Their pilot dataset consists of eight different topics each made up of a cluster of eight words and eight possible outliers. Four annotators were used for the creation of the dataset. The main advantage of this dataset is its near perfect human performance. However, we believe a major reason for that is the specific choice of clusters and the small size of the dataset.
3 GENERATING THE DATASET
In a similar format to the one used in the dataset furnished by Camacho-Collados & Navigli (2016), we generated sets of entities which were semantically similar to one another, known as a “cluster”, followed by up to three pairs (as available) of dissimilar entities, or “outliers”, each with different levels of semantic similarity to the cluster. The core thesis behind our design is that our knowledge base, Wikidata (2016), can be treated like a graph, where the semantic similarity between two elements is inversely proportional to their graph distance.
Informally, we treat Wikidata entities which are instances of a common entity as a cluster (see Figure 1). Then, starting from that common entity (which we call a ‘class’), we follow “subclass of” relationships to find a sibling class (see “American Football Team” in Figure 1). Two items which are instances of the sibling class (but not instances of the original class) are chosen as outliers. The process is then repeated with a ‘cousin’ class with a common grandparent to the original class (see “Ice Hockey Team” in Figure 1). Finally, we choose two additional outliers by randomly selecting items which are a distance of at least 7 steps away from the original class. These three “outlier classes” are referred to as O1, O2, and O3 outlier classes, respectively.
A full formalization of our approach is described in Appendix A.
3.1 REFINING THE DATASET QUALITY
Prior to developing a framework to improve the quality of the generated dataset, we performed a small amount of manual pruning of our Wikidata graph. Disambiguation pages led to bizarre clusters of entities, for their associated relationships are not true semantic connections, but are instead artifacts of the structure of our knowledge base. As such, they were removed. Additionally, classes within a distance of three from the entity for “Entity” itself1 (Q35120) had instances which had quite weak semantic similarity (one example being “human”). We decided that entities at this depth range ought to be removed from the Wikidata graph as well.
Once our Wikidata dump was pruned, we employed a few extra steps at generation time to further improve the quality of the dataset; first and foremost were how we chose representative instances and outliers for each class (see σi and σo in Appendix A). While “San Antonio Spurs” and “Chicago Bulls” may both be instances of “basketball team”, so are “BC Andorra” and “Olimpia Milano.” We wanted the cluster entities to be as strongly related as possible, so we sought a class-agnostic heuristic to accomplish this. Ultimately, we found that favoring entities whose associated Wikipedia pages had higher sitelink counts gave us the desired effect.
As such, we created clusters by choosing the top eight instances of a given class, ranked by sitelink count. Additionally, we only chose items as outliers when they had at least ten sitelinks so as to remove those which were ‘overly obscure,’ for the ability of word embeddings to identify rare words (Schnabel et al., 2015) would artificially decrease the difficulty of such outliers.
We then noticed that many cluster entities had similarities in their labels that could be removed if a different label was chosen. For example, 80% of the entities chosen for “association football club” ended with the phrase “F.C.” This essentially invalidates the cluster, for the high degree of syntactic overlap artificially increases the cosine similarity of all cluster items in word-level embeddings. In order to increase the quality of the surface forms chosen for each entity, we modified our resolution of entity QIDs to surface forms (see τ in Appendix A) to incorporate a variant2 of the work from
1Q35120 is effectively the “root” node of the Wikidata graph; 95.5% of nodes have “subclass of” chains which terminate at this node.
2By ‘variant,’ we are referring to the fact that the dictionaries in which we perform the probability lookups are constructed for each language, as opposed to the cross-lingual dictionaries originally described by Spitkovsky & Chang (2012).
Spitkovsky & Chang (2012):
τ(QID) = argmax s {P (s | wikipedia page(QID))} (1)
That is, the string for an entity is the string which is most likely to link to the Wikipedia page associated with that entity. For example, half of the inlinks to the page for Manchester United FC are the string “Manchester United,” which is the colloquial way of referring to the team.
Next, we filter out remaining clusters using a small set of heuristics. The following clusters are rejected:
• Clusters with more than two items are identical after having all digits removed. This handles cases such as entities only differing by years (e.g. “January 2010,” “January 2012,” etc.).
• Clusters with more than three elements have identical first or last six characters3. Characters are compared instead of words in order to better support inflected languages. This was inspired by clusters for classes such as “counties of Texas” (Q11774097), where even the dictionary-resolved aliases have high degrees of syntactic overlap (namely, over half of the cluster items ended with the word “County”).
• Clusters in which any item has an occurrence of a ‘stop affix,’ such as the prefix “Category:” or the suffix “一覧” (a Japanese Wikipedia equivalent of “List of”). In truth, this could be done during preprocessing, but doing it at cluster generation time instead has no bearing on the final results. These were originally all included under an additional stop class (“Wikimedia page outside the main knowledge tree”) at prune time, but miscategorizations in the Wikidata hierarchy prevented us from doing so; for example, a now-removed link resulted in every country being pruned from the dataset. As such, we opted to take a more conservative approach and perform this on at cluster-generation time and fine tune our stoplist as needed.
• Clusters with more than one entity with a string length of one. This prevents clusters such as “letters of the alphabet” being created. Note that this heuristic was disabled for the creation of Chinese and Japanese clusters.
• Clusters with too few entities, after duplicates introduced by resolving entities to surface forms (τ ) are removed.
3.2 THE WIKISEM500 DATASET
Using the above heuristics and preprocessing, we have generated a dataset, which we call WikiSem5004. Our dataset is formatted as a series of files containing test groups, comprised of a cluster and a series of outliers. Test cases can be constructed by taking each outlier in a given group with that group’s cluster. Table 1 shows the number of included test groups and test cases for each language. Each group contains a cluster of 7-8 entities and up to two entities from each of the three outlier classes. Table 2 shows example clusters taken from the dataset.
4 EVALUATION
For clarity, we first restate the definitions of the scoring metrics defined by Camacho-Collados & Navigli (2016) in terms of test groups (in contrast to the original definition, which is defined in terms of test cases). The way in which out-of-vocabulary entities are handled and scores are reported makes this distinction important, as will be seen in Section 4.3.
3 For Chinese and Japanese, this is modified such that the at least six entities must have identical (non-kana) first or last characters, or more than three must have identical the same first or last two characters. Because English is not inflected, we simply use spaces as approximate word boundaries and check that the first or last of those does not occur too often.
4The dataset is available for download at https://github.com/belph/wiki-sem-500
The core measure during evaluation is known as the compactness score; given a set W of words, it is defined as follows:
∀w ∈W, c(w) = 1 (|W | − 1)(|W | − 2) ∑ wi∈W\{w} ∑ wj∈W\{w}
wj 6=wi
sim(wi, wj) (2)
where sim is a vector similarity measure (typically cosine similarity). Note that Camacho-Collados & Navigli (2016) reduces the asymptotic complexity of c(w) from O(n3) to O(n2). We denote P (W,w) to be the (zero-indexed) position of w in the list of elements of W , sorted by compactness score in descending order. From this, we can describe the following definition for Outlier Position (OP), where 〈C,O〉 is a test group and o ∈ O:
OP (C ∪ {o}) = P (C ∪ {o}, o) (3)
This gives rise to the boolean-valued Outlier Detection (OD) function:
OD(C ∪ {o}) = { 1 OP (C ∪ {o}) = |C| 0 otherwise
(4)
Finally, we can now describe the Outlier Position Percentage (OPP) and Accuracy scores:
OPP (D) =
∑ 〈C,O〉∈D ∑ o∈O
OP (C∪{o}) |C|∑
〈C,O〉∈D|O| (5)
Accuracy(D) =
∑ 〈C,O〉∈D ∑ o∈O OD(C ∪ {o})∑
〈C,O〉∈D|O| (6)
4.1 HANDLING OUT-OF-VOCABULARY WORDS
One thing Camacho-Collados & Navigli (2016) does not address is how out-of-vocabulary (OOV) items should be handled. Because our dataset is much larger and contains a wider variety of words, we have extended their work to include additional scoring provisions which better encapsulate the performance of vector sets trained on different corpora.
There are two approaches to handling out-of-vocabulary entities: use a sentinel vector to represent all such entities or discard such entities entirely. The first approach is simpler, but it has a number of drawbacks; for one, a poor choice of sentinel can have a drastic impact on results. For example, an implementation which uses the zero vector as a sentinel and defines sim(~x,~0) = 0∀~x places many non-out-of-vocabulary outliers at a large disadvantage in a number of vector spaces, for we have found that negative compactness scores are rare. The second approach avoids deliberately introducing invalid data into the testing evaluation, but comparing scores across vector embeddings with different vocabularies is difficult due to them having different in-vocabulary subsets of the test set.
We have opted for the latter approach, computing the results on both the entire dataset and on only the intersection of in-vocabulary entities between all evaluated vector embeddings. This allows us to compare embedding performance both when faced with the same unknown data and when evaluated on the same, in-vocabulary data.
4.2 HUMAN BASELINE
In order to gauge how well embeddings should perform on our dataset, we conducted a human evaluation. We asked participants to select the outlier from a given test case, providing us with a human baseline for the accuracy score on the dataset. We computed the non-out-of-vocabulary intersection of the embeddings shown in Table 4, from which 60 test groups were sampled. Due to the wide array of domain knowledge needed to perform well on the dataset, participants were allowed to refer to Wikipedia (but explicitly told not to use Wikidata). We collected 447 responses, with an overall precision of 68.9%.
The performance found is not as high as on the baseline described in Camacho-Collados & Navigli (2016), so we conducted a second human evaluation on a smaller hand-picked set of clusters in order to determine whether a lack of domain knowledge or a systemic issue with our method was to blame. We had 6 annotators fully annotate 15 clusters generated with our system. Each cluster had one outlier, with a third of the clusters having each of the three outlier classes. Human performance was at 93%, with each annotator missing exactly one cluster. Five out of the six annotators missed the same cluster, which was based on books and contained an O1 outlier (the most difficult class). We interviewed the annotators, and three of them cited a lack of clarity on Wikipedia over whether or not the presented outlier was a book (leading them to guess), while the other two cited a conflation with one of the book titles and a recently popular Broadway production.
With the exception of this cluster, the performance was near-perfect, with one annotator missing one cluster. Consequently, we believe that the lower human performance on our dataset is primarily a result of the dataset’s broad domain.
4.3 EMBEDDING RESULTS
We evaluated our dataset on a number of publicly available vector embeddings: the Google Newstrained CBOW model released by Mikolov et al. (2013a), the 840-billion token Common Crawl corpus-trained GloVe model released by Pennington et al. (2014), and the English, Spanish, German, Japanese, and Chinese MultiCCA vectors5 from Ammar et al. (2016), which are trained on a combination of the Europarl (Koehn, 2005) and Leipzig (Quasthoff et al., 2006) corpora. In ad-
5The vectors are word2vec CBOW vectors, and the non-English vectors are aligned to the English vector space. Reproducing the original (unaligned) non-English vectors yields near-identical results to the aligned vectors.
dition, we trained GloVe, CBOW, and Skip-Gram (Mikolov et al., 2013a) models on an identical corpus comprised of an English Wikipedia dump and Gigaword corpus6.
The bulk of the embeddings we evaluated were word embeddings (as opposed to phrase embeddings), so we needed to combine each embeddings’ vectors in order to represent multi-word entities. If the embedding does handle phrases (only Google News), we perform a greedy lookup for the longest matching subphrase in the embedding, averaging the subphrase vectors; otherwise, we take a simple average of the vectors for each token in the phrase. If a token is out-of-vocabulary, it is ignored. If all tokens are out-of-vocabulary, the entity is discarded. This check happens as a preprocessing step in order to guarantee that a test case does not have its outlier thrown away. As such, we report the percentage of cluster entities filtered out for being out-of-vocabulary separately from the outliers which are filtered out, for the latter results in an entire test case being discarded.
In order to compare how well each vector embedding would do when run on unknown input data, we first collected the scores of each embedding on the entire dataset. Table 3 shows the Outlier Position Percentage (OPP) and accuracy scores of each embedding, along with the number of test groups which were skipped entirely7 and the mean percentage of out-of-vocabulary cluster entities and outliers among all test groups8. As in Camacho-Collados & Navigli (2016), we used cosine similarity for the sim measure in Equation 2.
The MultiCCA (Leipzig+Europarl) CBOW vectors have the highest rate of out-of-vocabulary entities, likely due in large part to the fact that its vocabulary is an order of magnitude smaller than the other embeddings (176,691, while the other embeddings had vocabulary sizes of over 1,000,000). Perhaps most surprising is the below-average performance of the Google News vectors. While attempting to understand this phenomenon, we noticed that disabling the phrase vectors boosted performance; as such, we have reported the performance of the vectors with and without phrase vectors enabled.
Inspecting the vocabulary of the Google News vectors, we have inferred that the vocabulary has undergone some form of normalization; performing the normalizations which we can be reasonably certain were done before evaluating has a negligible impact (≈ +0.01%) on the overall score. The Google News scores shown in Table 3 are with the normalization enabled. Ultimately, we hypothesize that the discrepancy in Google News scores comes down to the training corpus. We observe a bias in performance on our training set towards Wikipedia-trained vectors (discussed below; see Table 5), and, additionally, we expect that the Google News corpus did not have the wide regional
6We used the July 2016 Wikipedia dump (Wikimedia, 2016) and the 2011 Gigaword corpus (Parker et al., 2011).
7This happens when either all outliers are out-of-vocabulary or fewer than two cluster items are invocabulary. No meaningful evaluation can be performed on the remaining data, so the group is skipped.
8This includes the out-of-vocabulary rates of the skipped groups.
coverage that Wikidata has, limiting the training exposure to many of the more niche classes in the training set.
In order to get a better comparison between the embeddings under identical conditions, we then took the intersection of in-vocabulary entities across all embeddings and reevaluated on this subset. 23.88% of cluster entities and 22.37% of outliers were out-of-vocabulary across all vectors, with 23 test groups removed from evaluation. Table 4 shows the results of this evaluation.
The scores appear to scale roughly linearly when compared to Table 3, but these results serve as a more reliable ‘apples to apples’ comparison of the algorithms and training corpora.
Because Wikidata was the source of the dataset, we analyzed how using Wikipedia as a training corpus influenced the evaluation results. We trained three GloVe models with smaller vocabularies: one trained on only Gigaword, one trained on only Wikipedia, and one trained on both. The results of evaluating on the embeddings’ common intersection are shown in Table 5. We observe a slight (≈ 3.15% relative change) bias in OPP scores with Wikipedia over Gigaword, while finding a significantly larger (≈ 19.12% relative change) bias in accuracy scores. We believe that this bias is acceptable, for OPP scores (which we believe to be more informative) are not as sensitive to the bias and the numerous other factors involved in embedding generation (model, window size, etc.) can still be compared by controlling for the training corpora.
Additionally, we wanted to verify that the O1 outlier class (most similar) was the most difficult to distinguish from the cluster entities, followed by the O2 and O3 classes. We generated three separate datasets, each with only one class of outliers, and evaluated each embedding on each dataset. Figure 2 illustrates a strong positive correlation between outlier class and both OPP scores and accuracy.
Finally, we used the non-English MultiCCA vectors (Ammar et al., 2016) to evaluate the multilingual aspect of our dataset. We expect to see Spanish and German perform similarly to the English Europarl+Leipzig vectors, for the monolingual training corpora used to generate them consisted of
Spanish and German equivalents of the English training corpus. Table 6 shows the results of the non-English evaluations.
We observe a high degree of consistency with the results of the English vectors. The Japanese and Chinese scores are somewhat lower, but this is likely due to their having smaller training corpora and more limited vocabularies than their counterparts in other languages.
4.4 CORRELATION WITH DOWNSTREAM PERFORMANCE
In light of recent concerns raised about the correlation between intrinsic word embedding evaluations and performance in downstream tasks, we sought to investigate the correlation between WikiSem500 performance and extrinsic evaluations. We used the embeddings from Schnabel et al. (2015) and ran the outlier detection task on them with our dataset.
As a baseline measurement of how well our dataset correlates with performance on alternative intrinsic tasks, we our evaluation with the scores reported in Schnabel et al. (2015) on the well-known analogy task (Mikolov et al., 2013a). Figure 3a illustrates strong correlations between analogy task performance and our evaluation’s OPP scores and accuracy.
Figure 3b displays the Pearson’s correlation between the performance of each embedding on the WikiSem500 dataset and the extrinsic scores of each embedding on noun-phrase chunking and sentiment analysis reported in Schnabel et al. (2015).
Similar to the results seen in the paper, performance on our dataset correlates strongly with performance on a semantic-based task (sentiment analysis), with Pearson’s correlation coefficients higher than 0.97 for both accuracy and OPP scores. On the other hand, we observe a weak-to-nonexistent correlation with chunking. This is expected, however, for the dataset we have constructed consists of items which differ in semantic meaning; syntactic meaning is not captured by the dataset. It is worth noting the inconsistency between this and the intrinsic results in Figure 3a, which indicate a stronger correlation with the syntactic subset of the analogy task than its semantic subset. This is
expected, for it agrees with the poor correlation between chunking and intrinsic performance shown in Schnabel et al. (2015).
5 FUTURE WORK
Due to the favorable results we have seen from the WikiSem500 dataset, we intend to release test groups in additional languages using the method described in this paper. Additionally, we plan to study further the downstream correlation of performance on our dataset with additional downstream tasks.
Moreover, while we find a substantial correlation between performance on our dataset and on a semantically-based extrinsic task, the relationship between performance and syntactically-based tasks leaves much to be desired. We believe that the approach taken in this paper to construct our dataset could be retrofitted to a system such as WordNet (2010) or Wiktionary (2016) (for multilingual data) in order to construct syntactically similar clusters of items in a similar manner. We hypothesize that performance on such a dataset would correlate much more strongly with syntactically-based extrinsic evaluations such as chunking and part of speech tagging.
6 CONCLUSION
We have described a language-agnostic technique for generating a dataset consisting of semantically related items by treating a knowledge base as a graph. In addition, we have used this approach to construct the WikiSem500 dataset, which we have released. We show that performance on this dataset correlates strongly with downstream performance on sentiment analysis. This method allows for creation of much larger scale datasets in a larger variety of languages without the time-intensive task of human creation. Moreover, the parallel between Wikidata’s graph structure and the annotation guidelines from Camacho-Collados & Navigli (2016) preserve the simple-to-understand structure of the original dataset.
A FORMALIZATION
We now provide a formal description of the approach taken to generate our dataset.
Let V be the set of entities in Wikidata. For all v1, v2 ∈ V , we denote the relations v1≺Iv2 when v1 is an instance of v2, and v1≺Sv2 when v1 is a subclass of v2. We then define I : V → V ∗ as the following ‘instances’ mapping:
I(v) = {v′ ∈ V | v′≺Iv} (7)
For convenience, we then denote C = {v ∈ V | |I(v)| ≥ 2}; the interpretation being that C is the set of entities which have enough instances to possibly be viable clusters. We now formally state the following definition: Definition 1. A set A ⊆ V is a cluster if A = I(v) for some v ∈ C. We additionally say that v is the class associated with the cluster A.
Let P : V → V ∗ be the following ‘parent of’ mapping:
P (v) = {v′ ∈ V | v≺Sv′} (8)
Furthermore, let P−1 : V → V ∗ be the dual of P :
P−1(v) = {v′ ∈ V | v′≺Sv} (9)
For additional convenience, we denote the following:
P k(v) = { P (v) k = 1⋃
v′∈P (v) P k−1(v′) k > 1
(10)
As an abuse of notation, we define the following:
I∗(v) = I(v) ∪  ⋃ v′∈P−1(v) I∗(v)  (11) That is, I∗(v) is the set of all instances of v and all instances of anything that is a subclass of v (recursively).
We then define the measure d : V × V → N to be the graph distance between any entities in V , using the following set of edges:
ESU = {(v1, v2) | v1≺Sv2 ∨ v2≺Sv1} (12)
Finally, we define9 three additional mappings for outliers parametrized10 by µ ∈ N+:
O1(v) =  ⋃ p∈P (v)  ⋃ c∈P−1(p)\{v} I∗(c)  \ I(v) (13) 9For the definition of O2, note that we do not say that it must be true that p ∈ P 2(v) \ P (v). In practice, however, avoiding (if not excluding) certain values of p in this manner can help improve the quality of resulting clusters, at the cost of reducing the number of clusters which can be produced.
10The WikiSem500 dataset was generated with a value of µ = 7.
O2(v) =  ⋃ p∈P 2(v)  ⋃ c∈P−1(p)\{v} I∗(c)  \ I(v) (14) O3(v) =
 ⋃ p∈P (v) {e ∈ I(v′) | µ ≤ d(p, v′)}  \ I(v) (15) To simplify the model, we assume that all three of the above sets are mutually exclusive. Given these, we can formally state the following definition: Definition 2. Let A = I(v) be a cluster based on a class v. An outlier for A is any o ∈ O1(v) ∪ O2(v)∪O3(v). If o is in O1(v), O2(v), or O3(v), we denote the outlier class of o as O1, O2, or O3 (respectively).
Intuitively, the three outlier classes denote different degrees of ‘dissimilarity’ from the original cluster; O1 outliers are the most challenging to distinguish, for they are semantically quite similar to the cluster. O2 outliers are slightly easier to distinguish, and O3 outliers should be quite simple to pick out.
The final dataset (a set of 〈cluster, outliers〉 pairs) is then created by serializing the following:
D = τ ( fD (⋃ c∈C 〈fi(σi[I(c)]), fo (σo[O1(c)] ∪ σo[O2(c)] ∪ σo[O3(c)])〉 )) (16)
Where σi and σo are functions which select up to a given number of elements from the given set of instances and outliers (respectively), and fD, fi, and fo are functions which filter out dataset elements, instances, and outliers (respectively) based on any number of heuristics (see Section 3.1). Finally, τ takes the resulting tuples and resolves their QIDs to the appropriate surface strings.
The benefit of stating the dataset in the above terms is that it is highly configurable. In particular, different languages can be targeted by simply changing τ to resolve Wikidata entities to their labels in that language.

First, let me praise the authors for generating and releasing an NLP data set: a socially useful task.
The authors use an algorithm to generate a 500-cluster-per-language data set in semantic similarity. This brings up a few points.
1. If the point of using the algorithm is to be scalable, why release such a small data set? It's roughly the same order of magnitude as the data sets released in the SemEval tasks over the recent years. I would have expected something orders of magnitude larger.
2. The authors hand checked a small subset of the clusters: they found one where it was ambiguous, and should probably have been removed. Mechanical Turk can scale pretty well -- why not post-facto filter all of the clusters using MT? This is (in effect) how ImageNet was created, and it has millions of items.
3. Evaluating data set papers is an tricky issue. What makes a data set "good" or publishable? There are a number of medium-sized NLP data sets released every year (e.g., through SemEval). Those are designed to address tasks in NLP that people find interesting. I don't know of a data set that exactly addresses the task that the authors propose: the task is trying to address the idea of semantic similarity, which has had multiple data sets thrown at it since SemEval 2012. I wish that the paper had included comparisons to show that the particular task / data combination is better suited for analyzing semantic similarity than other existing data sets.
Two final notes:
A. This paper doesn't seem very well-suited to ICLR.  New NLP data sets may be indirectly useful for evaluating word embeddings (and hence representations). But, I didn't learn much from the paper: GloVe is empirically less good for semantic similarity than other embeddings? If true, why? That would be interesting.
B. The first proposal for the "put a word into a cluster and see if it stands out" task (in the context of human evaluation of topic models), is
Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M. Blei. Reading Tea Leaves: HowHumans Interpret Topic Models. Neural Information Processing Systems, 2009
which deserves a citation, I think.


The success of neural network methods for computing word embeddings has motivated methods for generating semantic embeddings of longer pieces of text, such as sentences and paragraphs. Surprisingly, Wieting et al (ICLR’16) showed that such complicated methods are outperformed, especially in out-of-domain (transfer learning) settings, by simpler methods involving mild retraining of word embeddings and basic linear regression. The method of Wieting et al. requires retraining with a substantial labeled dataset such as Paraphrase Database (Ganitkevitch et al., 2013). The current paper goes further, showing that the following completely unsupervised sentence embedding is a formidable baseline: Use word embeddings computed using one of the popular methods on unlabeled corpus like Wikipedia, represent the sentence by a weighted average of the word vectors, and then modify them a bit using PCA/SVD. This weighting improves performance by about 10% to 30% in textual similarity tasks, and beats sophisticated supervised methods including RNN’s and LSTM’s. It even improves Wieting et al.’s embeddings. This simple method should be used as the baseline to beat in future, especially when labeled training data is scarce or nonexistent. The paper also gives a theoretical explanation of the success of the above unsupervised method using a latent variable generative model for sentences, which is a simple extension of the model in Arora et al. (TACL’16) with new “smoothing” terms that allow for words occurring out of context, as well as high probabilities for words like and, not in all contexts.
1 INTRODUCTION
Word embeddings computed using diverse methods are basic building blocks for Natural Language Processing (NLP) and Information Retrieval (IR). They capture the similarities between words (e.g., (Bengio et al., 2003; Collobert & Weston, 2008; Mikolov et al., 2013a; Pennington et al., 2014)). Recent work has tried to compute embeddings that capture the semantics of word sequences (phrases, sentences, and paragraphs), with methods ranging from simple additional composition of the word vectors to sophisticated architectures such as convolutional neural networks and recurrent neural networks (e.g., (Iyyer et al., 2015; Le & Mikolov, 2014; Kiros et al., 2015; Socher et al., 2011; Blunsom et al., 2014; Tai et al., 2015; Wang et al., 2016)). Recently, (Wieting et al., 2016) learned general-purpose, paraphrastic sentence embeddings by starting with standard word embeddings and modifying them based on supervision from the Paraphrase pairs dataset (PPDB), and constructing sentence embeddings by training a simple word averaging model. This simple method leads to better performance on textual similarity tasks than a wide variety of methods and serves as a good initialization for textual classification tasks. However, supervision from the paraphrase dataset seems crucial, since they report that simple average of the initial word embeddings does not work very well.
Here we give a new sentence embedding method that is embarrassingly simple: just compute the weighted average of the word vectors in the sentence and then remove the projections of the average vectors on their first principal component (“common component removal”). Here the weight of a word w is a/(a+ p(w)) with a being a parameter and p(w) the (estimated) word frequency; we call
this smooth inverse frequency (SIF). This method achieves significantly better performance than the unweighted average on a variety of textual similarity tasks, and on most of these tasks even beats some sophisticated supervised methods tested in (Wieting et al., 2016), including some RNN and LSTM models. The method is well-suited for domain adaptation settings, i.e., word vectors trained on various kinds of corpora are used for computing the sentence embeddings in different testbeds. It is also fairly robust to the weighting scheme: using the word frequencies estimated from different corpora does not harm the performance; a wide range of the parameters a can achieve close-to-best results, and an even wider range can achieve significant improvement over unweighted average.
Of course, this SIF reweighting is highly reminiscent of TF-IDF reweighting from information retrieval (Sparck Jones, 1972; Robertson, 2004) if one treats a “sentence” as a “document” and make the reasonable assumption that the sentence doesn’t typically contain repeated words. Such reweightings (or related ideas like removing frequent words from the vocabulary) are a good rule of thumb but has not had theoretical justification in a word embedding setting.
The current paper provides a theoretical justification for the reweighting using a generative model for sentences, which is a simple modification for the Random Walk on Discourses model for generating text in (Arora et al., 2016). In that paper, it was noted that the model theoretically implies a sentence embedding, namely, simple average of embeddings of all the words in it.
We modify this theoretical model, motivated by the empirical observation that most word embedding methods, since they seek to capture word cooccurence probabilities using vector inner product, end up giving large vectors to frequent words, as well as giving unnecessarily large inner products to word pairs, simply to fit the empirical observation that words sometimes occur out of context in documents. These anomalies cause the average of word vectors to have huge components along semantically meaningless directions. Our modification to the generative model of (Arora et al., 2016) allows “smoothing” terms, and then a max likelihood calculation leads to our SIF reweighting.
Interestingly, this theoretically derived SIF does better (by a few percent points) than traditional TFIDF in our setting. The method also improves the sentence embeddings of Wieting et al., as seen in Table 1. Finally, we discovered that —contrary to widespread belief—Word2Vec(CBOW) also does not use simple average of word vectors in the model, as misleadingly suggested by the usual expression Pr[w|w1, w2, . . . , w5] ∝ exp(vw · ( 15 ∑ i vwi)). A dig into the implementation shows it implicitly uses a weighted average of word vectors —again, different from TF-IDF— and this weighting turns out to be quite similar in effect to ours. (See Section 3.1.)
2 RELATED WORK
Word embeddings. Word embedding methods represent words as continuous vectors in a low dimensional space which capture lexical and semantic properties of words. They can be obtained from the internal representations from neural network models of text (Bengio et al., 2003; Collobert & Weston, 2008; Mikolov et al., 2013a) or by low rank approximation of co-occurrence statistics (Deerwester et al., 1990; Pennington et al., 2014). The two approaches are known to be closely related (Levy & Goldberg, 2014; Hashimoto et al., 2016; Arora et al., 2016).
Our work is most directly related work to (Arora et al., 2016), which proposed a random walk model for generating words in the documents. Our sentence vector can be seen as approximate inference of the latent variables in their generative model.
Phrase/Sentence/Paragraph embeddings. Previous works have computed phrase or sentence embeddings by composing word embeddings using operations on vectors and matrices e.g., (Mitchell & Lapata, 2008; 2010; Blacoe & Lapata, 2012). They found that coordinate-wise multiplication of the vectors performed very well among the binary operations studied. Unweighted averaging is also found to do well in representing short phrases (Mikolov et al., 2013a). Another approach is recursive neural networks (RNNs) defined on the parse tree, trained with supervision (Socher et al., 2011) or without (Socher et al., 2014). Simple RNNs can be viewed as a special case where the parse tree is replaced by a simple linear chain. For example, the skip-gram model (Mikolov et al., 2013b) is extended to incorporate a latent vector for the sequence, or to treat the sequences rather than the word as basic units. In (Le & Mikolov, 2014) each paragraph was assumed to have a latent paragraph vector, which influences the distribution of the words in the paragraph. Skip-thought
of (Kiros et al., 2015) tries to reconstruct the surrounding sentences from surrounded one and treats the hidden parameters as their vector representations. RNNs using long short-term memory (LSTM) capture long-distance dependency and have also been used for modeling sentences (Tai et al., 2015). Other neural network structures include convolution neural networks, such as (Blunsom et al., 2014) that uses a dynamic pooling to handle input sentences of varying length and do well in sentiment prediction and classification tasks.
The directed inspiration for our work is (Wieting et al., 2016) which learned paraphrastic sentence embeddings by using simple word averaging and also updating standard word embeddings based on supervision from paraphrase pairs; the supervision being used for both initialization and training.
3 A SIMPLE METHOD FOR SENTENCE EMBEDDING
We briefly recall the latent variable generative model for text in (Arora et al., 2016). The model treats corpus generation as a dynamic process, where the t-th word is produced at step t. The process is driven by the random walk of a discourse vector ct ∈ <d. Each word w in the vocabulary has a vector in <d as well; these are latent variables of the model. The discourse vector represents “what is being talked about.” The inner product between the discourse vector ct and the (time-invariant) word vector vw for word w captures the correlations between the discourse and the word. The probability of observing a word w at time t is given by a log-linear word production model from Mnih and Hinton:
Pr[w emitted at time t | ct] ∝ exp (〈ct, vw〉) . (1)
The discourse vector ct does a slow random walk (meaning that ct+1 is obtained from ct by adding a small random displacement vector), so that nearby words are generated under similar discourses. It was shown in (Arora et al., 2016) that under some reasonable assumptions this model generates behavior –in terms of word-word cooccurence probabilities—that fits empirical works like word2vec and Glove. The random walk model can be relaxed to allow occasional big jumps in ct, since a simple calculation shows that they have negligible effect on cooccurrence probabilities of words. The word vectors computed using this model are reported to be similar to those from Glove and word2vec(CBOW).
Our improved Random Walk model. Clearly, it is tempting to define the sentence embedding as follows: given a sentence s, do a MAP estimate of the discourse vectors that govern this sentence. We note that we assume the discourse vector ct doesn’t change much while the words in the sentence were emitted, and thus we can replace for simplicity all the ct’s in the sentence s by a single discourse vector cs. In the paper (Arora et al., 2016), it was shown that the MAP estimate of cs is —up to multiplication by scalar—the average of the embeddings of the words in the sentence.
In this paper, towards more realistic modeling, we change the model (1) as follows. This model has two types of “smoothing term”, which are meant to account for the fact that some words occur out of context, and that some frequent words (presumably “the”, “and ” etc.) appear often regardless of the discourse. We first introduce an additive term αp(w) in the log-linear model, where p(w) is the unigram probability (in the entire corpus) of word and α is a scalar. This allows words to occur even if their vectors have very low inner products with cs. Secondly, we introduce a common discourse vector c0 ∈ <d which serves as a correction term for the most frequent discourse that is often related to syntax. (Other possible correction is left to future work.) It boosts the co-occurrence probability of words that have a high component along c0.
Concretely, given the discourse vector cs, the probability of a word w is emitted in the sentence s is modeled by,
Pr[w emitted in sentence s | cs] = αp(w) + (1− α) exp (〈c̃s, vw〉)
Zc̃s , (2)
where c̃s = βc0 + (1− β)cs, c0 ⊥ cs where α and β are scalar hyperparameters, and Zc̃s = ∑ w∈V exp (〈c̃s, vw〉) is the normalizing constant (the partition function). We see that the model allows a word w unrelated to the discourse cs to be omitted for two reasons: a) by chance from the term αp(w); b) if w is correlated with the common discourse vector c0.
Algorithm 1 Sentence Embedding Input: Word embeddings {vw : w ∈ V}, a set of sentences S, parameter a and estimated probabil-
ities {p(w) : w ∈ V} of the words. Output: Sentence embeddings {vs : s ∈ S}
1: for all sentence s in S do 2: vs ← 1|s| ∑ w∈s a a+p(w)vw 3: end for 4: Compute the first principal component u of {vs : s ∈ S} 5: for all sentence s in S do 6: vs ← vs − uu>vs 7: end for
Computing the sentence embedding. The word embeddings yielded by our model are actually the same 1 The sentence embedding will be defined as the max likelihood estimate for the vector cs that generated it. ( In this case MLE is the same as MAP since the prior is uniform.) We borrow the key modeling assumption of (Arora et al., 2016), namely that the word vw’s are roughly uniformly dispersed, which implies that the partition function Zc is roughly the same in all directions. So assume that Zc̃s is roughly the same, say Z for all c̃s. By the model (2) the likelihood for the sentence is
p[s | cs] = ∏ w∈s p(w | cs) = ∏ w∈s [ αp(w) + (1− α)exp (〈vw, c̃s〉) Z ] .
Let
fw(c̃s) = log
[ αp(w) + (1− α)exp (〈vw, c̃s〉)
Z ] denote the log likelihood of sentence s. Then, by simple calculus we have,
∇fw(c̃s) = 1 αp(w) + (1− α) exp (〈vw, c̃s〉) /Z 1− α Z exp (〈vw, c̃s〉) vw.
Then by Taylor expansion, we have,
fw(c̃s) ≈ fw(0) +∇fw(0)>c̃s
= constant + (1− α)/(αZ)
p(w) + (1− α)/(αZ) 〈vw, c̃s〉 .
Therefore, the maximum likelihood estimator for c̃s on the unit sphere (ignoring normalization) is approximately,2
arg max ∑ w∈s fw(c̃s) ∝ ∑ w∈s
a
p(w) + a vw, where a = 1− α αZ . (3)
That is, the MLE is approximately a weighted average of the vectors of the words in the sentence. Note that for more frequent words w, the weight a/(p(w) + a) is smaller, so this naturally leads to a down weighting of the frequent words.
To estimate cs, we estimate the direction c0 by computing the first principal component of c̃s’s for a set of sentences. In other words, the final sentence embedding is obtained by subtracting the projection of c̃s’s to their first principal component. This is summarized in Algorithm 1.
3.1 CONNECTION TO SUBSAMPLING PROBABILITIES IN WORD2VEC
Word2vec (Mikolov et al., 2013b) uses a sub-sampling technique which downsamples word w with probability proportional to 1/ √ p(w) where p(w) is the marginal probability of the word w. This
1We empirically discovered the significant common component c0 in word vectors built by existing methods, which inspired us to propose our theoretical model of this paper.
2Note that maxc:‖c‖=1 C + 〈c, g〉 = g/‖g‖ for any constant C.
heuristic not only speeds up the training but also learns more regular word representations. Here we explain that this corresponds to an implicit reweighting of the word vectors in the model and therefore the statistical benefit should be of no surprise.
Recall the vanilla CBOW model of word2vec:
Pr[wt | wt−1, . . . , wt−5] ∝ exp (〈v̄t, vw〉) , where v̄t = 1
5 5∑ i=1 vwt−i . (4)
It can be shown that the loss (MLE) for the single word vector vw (from this occurrence) can be abstractly written in the form,
g(vw) = γ(〈v̄t, vw〉) + negative sampling terms ,
where γ(x) = log(1/(1 + e−x)) is the logistic function. Therefore, the gradient of g(vw) is
∇g(vw) = γ′(〈v̄t, vw〉)v̄t = α(vwt−5 + vwt−4 + vwt−3 + vwt−2 + vwt−1) , (5)
where α is a scalar. That is, without the sub-sampling trick, the update direction is the average of the word vectors in the window.
The sub-sampling trick in (Mikolov et al., 2013b) randomly selects the summands in equation (5) to “estimate” the gradient. Specifically, the sampled update direction is
∇̃g(vw) = α(J5vwt−5 + J4vwt−4 + J3vwt−3 + J2vwt−2 + J1vwt−1) (6)
where Jk’s are Bernoulli random variables with Pr [Jk = 1] = q(wt−k) , min { 1, √ 10−5
p(wt−k)
} .
However, we note that ∇̃g(vw) is (very) biased estimator! We have that the expectation of ∇̃g(vw) is a weighted sum of the word vectors,
E [ ∇̃g(vw) ] = α(q(wt−5)vwt−5 + q(wt−4)vwt−4 + q(wt−3)vwt−3 + q(wt−2)vwt−2 + q(wt−1)vwt−1) .
In fact, the expectation E[∇̃g(vw)] corresponds to the gradient of a modified word2vec model with the average v̄t (in equation (4)) being replaced by the weighted average ∑5 k=1 q(wt−k)vwt−k . Such a weighted model can also share the same form of what we derive from our random walk model as in equation (3). Moreover, the weighting q(wi) closely tracks our weighting scheme a/(a + p(w)) when using parameter a = 10−4; see Figure 1 for an illustration. Therefore, the expected gradient here is approximately the estimated discourse vector in our model! Thus, word2vec with sub-sampling gradient heuristic corresponds to a stochastic gradient update method for using our weighting scheme.
4 EXPERIMENTS
4.1 TEXTUAL SIMILARITY TASKS
Datasets. We test our methods on the 22 textual similarity datasets including all the datasets from SemEval semantic textual similarity (STS) tasks (2012-2015) (Agirre et al., 2012; 2013; 2014; Agirrea et al., 2015), and the SemEval 2015 Twitter task (Xu et al., 2015) and the SemEval 2014 Semantic Relatedness task (Marelli et al., 2014). The objective of these tasks is to predict the similarity between two given sentences. The evaluation criterion is the Pearson’s coefficient between the predicted scores and the ground-truth scores.
Experimental settings. We will compare our method with the following:
1. Unsupervised: ST, avg-GloVe, tfidf-GloVe. ST denotes the skip-thought vectors (Kiros et al., 2015), avg-GloVe denotes the unweighted average of the GloVe vectors (Pennington et al., 2014),3 and tfidf-GloVe denotes the weighted average of GloVe vectors using TF-IDF weights.
2. Semi-supervised: avg-PSL. This method uses the unweighted average of the PARAGRAMSL999 (PSL) word vectors from (Wieting et al., 2015). The word vectors are trained using labeled data, but the sentence embedding are computed by unweighted average without training.
3. Supervised: PP, PP-proj., DAN, RNN, iRNN, LSTM (o.g.), LSTM (no). All these methods are initialized with PSL word vectors and then trained on the PPDB dataset. PP and PPproj. are proposed in (Wieting et al., 2016). The first is an average of the word vectors, and the second additionally adds a linear projection. The word vectors are updated during the training. DAN denotes the deep averaging network of (Iyyer et al., 2015). RNN denotes the classical recurrent neural network, and iRNN denotes a variant with the activation being the identity, and the weight matrices initialized to identity. The LSTM is the version from (Gers et al., 2002), either with output gates (denoted as LSTM(o.g.)) or without (denoted as LSTM (no)).
Our method can be applied to any types of word embeddings. So we denote the sentence embeddings obtained by applying our method to word embeddings method “XXX” as “XXX+WR”.4 To get a completely unsupervised method, we apply it to the GloVe vectors, denoted as GloVe+WR. The weighting parameter a is fixed to 10−3, and the word frequencies p(w) are estimated from the
3We used the vectors that are publicly available at http://nlp.stanford.edu/projects/glove/. They are 300- dimensional vectors that were trained on the 840 billion token Common Crawl corpus.
4“W” stands for the smooth inverse frequency weighting scheme, and “R” stands for removing the common components.
commoncrawl dataset.5 This is denoted by GloVe+WR in Table 1. We also apply our method on the PSL vectors, denoted as PSL+WR, which is a semi-supervised method.
Results. The results are reported in Table 1. Each year there are 4 to 6 STS tasks. For clarity, we only report the average result for the STS tasks each year; the detailed results are in the appendix.
The unsupervised method GloVe+WR improves upon avg-GloVe significantly by 10% to 30%, and beats the baselines by large margins. It achieves better performance than LSTM and RNN and is comparable to DAN, even though the later three use supervision. This demonstrates the power of this simple method: it can be even stronger than highly-tuned supervisedly trained sophisticated models. Using TF-IDF weighting scheme also improves over the unweighted average, but not as much as our method.
The semi-supervised method PSL+WR achieves the best results for four out of the six tasks and is comparable to the best in the rest of two tasks. Overall, it outperforms the avg-PSL baseline and all the supervised models initialized with the same PSL vectors. This demonstrates the advantage of our method over the training for those models.
We also note that the top singular vectors c0 of the datasets seem to roughly correspond to the syntactic information or common words. For example, closest words (by cosine similarity) to c0 in the SICK dataset are “just”, “when”, “even”, “one”, “up”, “little”, “way”, “there”, “while”, and “but.”
Finally, in the appendix, we showed that our two ideas all contribute to the improvement: for GloVe vectors, using smooth inverse frequency weighting alone improves over unweighted average by about 5%, using common component removal alone improves by 10%, and using both improves by 13%.
4.1.1 EFFECT OF WEIGHTING PARAMETER ON PERFORMANCE
We study the sensitivity of our method to the weighting parameter a, the method for computing word vectors, and the estimated word probabilities p(w). First, we test the performance of three
5It is possible to tune the parameter a to get better results. The effect of a and the corpus for estimating word frequencies are studied in Section 4.1.1.
types of word vectors (PSL, GloVe, and SN) on the STS 2012 tasks. SN vectors are trained on the enwiki dataset (Wikimedia, 2012) using the method in (Arora et al., 2016), while PSL and GloVe vectors are those used in Table 1. We enumerate a ∈ {10−i, 3 × 10−i : 1 ≤ i ≤ 5} and use the p(w) estimated on the enwiki dataset. Figure 2a shows that for all three kinds of word vectors, a wide range of a leads to significantly improved performance over the unweighted average. Best performance occurs from a = 10−3 to a = 10−4.
Next, we fix a = 10−3 and use four very different datasets to estimate p(w): enwiki (wikipedia, 3 billion tokens), poliblogs (Yano et al., 2009) (political blogs, 5 million), commoncrawl (Buck et al., 2014) (Internet crawl, 800 billion), text8 (Mahoney, 2008) (wiki subset, 1 million). Figure 2b shows performance is almost the same for all four settings.
The fact that our method can be applied on different types of word vectors trained on different corpora also suggests it should be useful across different domains. This is especially important for unsupervised methods, since the unlabeled data available may be collected in a different domain from the target application.
4.2 SUPERVISED TASKS
The sentence embeddings obtained by our method can be used as features for downstream supervised tasks. We consider three tasks: the SICK similarity task, the SICK entailment task, and the Stanford Sentiment Treebank (SST) binary classification task (Socher et al., 2013). To highlight the representation power of the sentence embeddings learned unsupervisedly, we fix the embeddings and only learn the classifier. Setup of supervised tasks mostly follow (Wieting et al., 2016) to allow fair comparison, i.e., the classifier a linear projection followed by the classifier in (Kiros et al., 2015). The linear projection maps the sentence embeddings into 2400 dimension (the same as the skip-thought vectors), and is learned during the training. We compare our method to PP, DAN, RNN, and LSTM, which are the methods used in Section 4.1. We also compare to the skip-thought vectors (with improved training in (Lei Ba et al., 2016)).
Results. Our method gets better or comparable performance compared to the competitors. It gets the best results for two of the tasks. This demonstrates the power of our simple method. We emphasize that our embeddings are unsupervisedly learned, while DAN, RNN, LSTM are trained with supervision. Furthermore, skip-thought vectors are much higher dimensional than ours (though projected into higher dimension, the original 300 dimensional embeddings contain all the information).
The advantage is not as significant as in the textual similarity tasks. This is possibly because similarity tasks rely directly upon cosine similarity, which favors our method’s approach of removing the common components (which can be viewed as a form of denoising), while in supervised tasks, with the cost of some label information, the classifier can pick out the useful components and ignore the common ones.
Finally, we speculate that our method doesn’t outperform RNN’s and LSTM’s for sentiment tasks because (a) the word vectors —and more generally the distributional hypothesis of meaning —has known limitations for capturing sentiment due to the “antonym problem”, (b) also in our weighted average scheme, words like “not” that may be important for sentiment analysis are downweighted a lot. To address (a), there is existing work on learning better word embeddings for sentiment analysis (e.g., (Maas et al., 2011)). To address (b), it is possible to design weighting scheme (or learn weights) for this specific task.
4.3 THE EFFECT OF THE ORDER OF WORDS IN SENTENCES
A interesting feature of our method is that it ignores the word order. This is in contrast to that RNN’s and LSTM’s can potentially take advantage of the word order. The fact that our method achieves better or comparable performance on these benchmarks raise the following question: is word order not important in these benchmarks? We conducted an experiment suggesting that word order does play some role.
We trained and tested RNN/LSTM on the supervised tasks where the words in each sentence are randomly shuffled, and the results are reported in Table 3.6 It can be observed that the performance drops noticeably. Thus our method —which ignores word order—must be much better at exploiting the semantics than RNN’s and LSTM’s. An interesting future direction is to explore if some ensemble idea can combine the advantages of both approaches.
5 CONCLUSIONS
This work provided a simple approach to sentence embedding, based on the discourse vectors in the random walk model for generating text (Arora et al., 2016). It is simple and unsupervised, but achieves significantly better performance than baselines on various textual similarity tasks, and can even beat sophisticated supervised methods such as some RNN and LSTM models. The sentence embeddings obtained can be used as features in downstream supervised tasks, which also leads to better or comparable results compared to the sophisticated methods.
6 ACKNOWLEDGEMENTS
We thank the reviewers for insightful comments. We also thank the authors of (Wieting et al., 2016; Bowman et al., 2015) for sharing their code or the preprocessed datasets.
This work was supported in part by NSF grants CCF-1527371, DMS-1317308, Simons Investigator Award, Simons Collaboration Grant, and ONRN00014- 16-1-2329. Tengyu Ma was supported in addition by Simons Award in Theoretical Computer Science and IBM PhD Fellowship.
A DETAILS OF EXPERIMENTAL SETTING
A.1 UNSUPERVISED TASK: TEXTUAL SIMILARITY
The competitors. We give a brief overview of the competitors. RNN is the classical recurrent neural network:
ht = f(WxW xt w +Whht−1 + b)
where f is the activation, Wx,Wh and b are parameters, and xt is the t-th token in the sentence. The sentence embedding of RNN is just the hidden vector of the last token. iRNN is a special RNN with the activation being the identity, the weight matrices initialized to identity, and b initialized to zero. LSTM (Hochreiter & Schmidhuber, 1997) is a recurrent neural network architecture designed to capture long-distance dependencies. Here, the version from (Gers et al., 2002) is used.
The supervised methods are initialized with PARAGRAM-SL999 (PSL) vectors, and trained using the approach of (Wieting et al., 2016) on the XL section of the PPDB data (Pavlick et al., 2015) which contains about 3 million unique phrase pairs. After training, the final models can be used to generate sentence embeddings on the test data. For hyperparameter tuning they used 100k examples sampled from PPDB XXL and trained for 5 epochs. Then after finding the hyperparameters that maximize Spearmans coefficients on the Pavlick et al. PPDB task, they are trained on the entire XL section of PPDB for 10 epochs. See (Wieting et al., 2016) and related papers for more details about these methods.
The tfidf-GloVe method is a weighted average of the GloVe vectors, where the weights are defined by the TF-IDF scheme. More precisely, the embedding of a sentence s is
vs = 1 |s| ∑ w∈s IDFwvw (7)
where IDFw is the inverse document frequency of w, and |s| denotes the number of words in the sentence. Here, the TF part of the TF-IDF scheme is taken into account by the sum over w ∈ s. Furthermore, when computing IDFw, each sentence is viewed as a “document”:
IDFw := log 1 +N
1 +Nw
where N is the total number of sentences and Nw is the number of sentences containing w, and 1 is added to avoid division by 0. In the experiments, we use all the textual similarity datasets to compute IDFw.
Detailed experimental results. In the main body we present the average results for STS tasks by year. Each year there are actually 4 to 6 STS tasks, as shown in Table 4. Note that tasks with the same name in different years are actually different tasks. Here we provide the results for each tasks in Table 5. PSL+WR achieves the best results on 12 out of 22 tasks, PP and PP-proj. achieves on 3, tfidf-GloVe achieves on 2, and DAN, iRNN, and GloVe+WR achieves on 1. In general, our method improves the performance significantly compared to the unweighted average, though on rare cases such as MSRpar it can decrease the performance.
Effects of smooth inverse frequency and common component removal. There are two key ideas in our methods: smooth inverse frequency weighting (W) and common component removal (R). It is instructive to see their effects separately. Let GloVe+W denote the embeddings using only smooth inverse frequency, and GloVe+R denote that using only common component removal. Similarly define PSL+W and PSL+R. The results for these methods are shown in Table 6. When using GloVe vectors, W alone improves the performance of the unweighted average baseline by about 5%, R alone improves by 10%, W and R together improves by 13%. When using PSL vectors, W gets 10%, R gets 10%, W and R together gets 13%. In summary, both techniques are important for obtaining significant advantage over the unweighted average.
A.2 SUPERVISED TASKS
Setup of supervised tasks mostly follow (Wieting et al., 2016) to allow fair comparison: the sentence embeddings are fixed and fed into some classifier which are trained. For the SICK similarity task, given a pair of sentences with embeddings vL and vR, first do a linear projection:
hL = WpvL, hR = WpvR
where Wp is of size 300 × dp and is learned during training. dp = 2400 matches the dimension of the skip-thought vectors. Then hL and hR are used in the objective function from (Tai et al., 2015). Almost the same approach is used for the entailment task.For the sentiment task, the classifier has a fully-connected layer with a sigmoid activation followed by a softmax layer.
Recall that our method has two steps: smooth inverse frequency weighting and common component removal. For the experiments here, we do not perform the common component removal, since it can be absorbed into the projection step. For the weighted average, the hyperparameter a is enumerated in {10−i, 3 × 10−i : 2 ≤ i ≤ 3}. The other hyperparameters are enumerated as in (Wieting et al., 2016), and the same validation approach is used to select the final values.
A.3 ADDITIONAL SUPERVISED TASKS
Here we report the experimental results on two more datasets, comparing to known results on them.
SNLI. The first experiment is for the 3-class classification task on the SNLI dataset (Bowman et al., 2015). To compare to the results in (Bowman et al., 2015), we used their experimental setup. In particular, we applied our method to 300 dimensional GloVe vectors and used an additional tanh neural network layer to map these 300d embeddings into 300 dimensional space, then used the code provided by the authors of (Bowman et al., 2015), trained the classifier on our 100 dimensional sentence embedding for 120 passes over the data, using their default hyperparameters. The results are shown in Table 7. Our method indeed gets slightly better performance.
Our test accuracy is worse than those using more sophisticated models (e.g., using attention mechanism), which are typically 83% - 88%; see the website of the SNLI project7 for a summary. An interesting direction is to study whether our idea can be combined with these sophisticated models to get improved performance.
IMDB. The second experiment is the sentiment analysis task on the IMDB dataset, studied in (Wang & Manning, 2012). Since the intended application is semi-supervised or transfer learning, we also compared performance with fewer labeled examples.
7http://nlp.stanford.edu/projects/snli/
Our method gets worse performance on the full dataset, but its decrease in performance is better with less labeled examples, showing the benefit of using word embeddings. Note that our sentence embeddings are unsupervised, while that in the NB-SVM method takes advantage of the labels. Another comment is that sentiment analysis appears to be the best case for Bag-Of-Word methods, whereas it may be the worst case for word embedding methods (See Table 2) due to the well-known antonymy problem —distributional hypothesis fails for distinguishing “good” from “bad.”

This paper proposes a simple way to reweight the word embedding in the simple composition function for sentence representation. This paper also shows the connection between this new weighting scheme and some previous work.
Here are some comments on technical details:
- The word "discourse" is confusing. I am not sure whether the words "discourse" in "discourse vector c_s" and the one in "most frequent discourse" have the same meaning.
- Is there any justification about $c_0$ related to syntac?
- Not sure what thie line means: "In fact the new model was discovered by our detecting the common component c0 in existing embeddings." in section "Computing the sentence embedding"
- Is there any explanation about the results on sentiment in Table 2?


Previous work combines word-level and character-level representations using concatenation or scalar weighting, which is suboptimal for high-level tasks like reading comprehension. We present a fine-grained gating mechanism to dynamically combine word-level and character-level representations based on properties of the words. We also extend the idea of fine-grained gating to modeling the interaction between questions and paragraphs for reading comprehension. Experiments show that our approach can improve the performance on reading comprehension tasks, achieving new state-of-the-art results on the Children’s Book Test and Who Did What datasets. To demonstrate the generality of our gating mechanism, we also show improved results on a social media tag prediction task.1
1 INTRODUCTION
Finding semantically meaningful representations of the words (also called tokens) in a document is necessary for strong performance in Natural Language Processing tasks. In neural networks, tokens are mainly represented in two ways, either using word-level representations or character-level representations. Word-level representations are obtained from a lookup table, where each unique token is represented as a vector. Character-level representations are usually obtained by applying recurrent neural networks (RNNs) or convolutional neural networks (CNNs) on the character sequence of the token, and their hidden states are combined to form the representation. Word-level representations are good at memorizing the semantics of the tokens while character-level representations are more suitable for modeling sub-word morphologies (Ling et al., 2015; Yang et al., 2016a). For example, considering “cat” and “cats”, word-level representations can only learn the similarities between the two tokens by training on a large amount of training data, while character-level representations, by design, can easily capture the similarities. Character-level representations are also used to alleviate the difficulties of modeling out-of-vocabulary (OOV) tokens (Luong & Manning, 2016).
Hybrid word-character models have been proposed to leverage the advantages of both word-level and character-level representations. The most commonly used method is to concatenate these two representations (Yang et al., 2016a). However, concatenating word-level and character-level representations is technically problematic. For frequent tokens, the word-level representations are usually accurately estimated during the training process, and thus introducing character-level representations can potentially bias the entire representations. For infrequent tokens, the estimation of wordlevel representations have high variance, which will have negative effects when combined with the character-level representations. To address this issue, recently Miyamoto & Cho (2016) introduced a scalar gate conditioned on the word-level representations to control the ratio of the two representations. However, for the task of reading comprehension, preliminary experiments showed that this method was not able to improve the performance over concatenation. There are two possible reasons. First, word-level representations might not contain sufficient information to support the decisions of selecting between the two representations. Second, using a scalar gate means applying the same ratio for each of the dimensions, which can be suboptimal.
In this work, we present a fine-grained gating mechanism to combine the word-level and characterlevel representations. We compute a vector gate as a linear projection of the token features followed
1Code is available at https://github.com/kimiyoung/fg-gating
by a sigmoid activation. We then multiplicatively apply the gate to the character-level and wordlevel representations. Each dimension of the gate controls how much information is flowed from the word-level and character-level representations respectively. We use named entity tags, part-ofspeech tags, document frequencies, and word-level representations as the features for token properties which determine the gate. More generally, our fine-grained gating mechanism can be used to model multiple levels of structure in language, including words, characters, phrases, sentences and paragraphs. In this work we focus on studying the effects on word-character gating.
To better tackle the problem of reading comprehension, we also extend the idea of fine-grained gating for modeling the interaction between documents and queries. Previous work has shown the importance of modeling interactions between document and query tokens by introducing various attention architectures for the task (Hermann et al., 2015; Kadlec et al., 2016). Most of these use an inner product between the two representations to compute the relative importance of document tokens. The Gated-Attention Reader (Dhingra et al., 2016a) showed improved performance by replacing the inner-product with an element-wise product to allow for better matching at the semantic level. However, they use aggregated representations of the query which may lead to loss of information. In this work we use a fine-grained gating mechanism for each token in the paragraph and each token in the query. The fine-grained gating mechanism applies an element-wise multiplication of the two representations.
We show improved performance on reading comprehension datasets, including Children’s Book Test (CBT), Who Did What, and SQuAD. On CBT, our approach achieves new state-of-the-art results without using an ensemble. Our model also improves over state-of-the-art results on the Who Did What dataset. To demonstrate the generality of our method, we apply our word-character finegrained gating mechanism to a social media tag prediction task and show improved performance over previous methods.
Our contributions are two-fold. First, we present a fine-grained word-character gating mechanism and show improved performance on a variety of tasks including reading comprehension. Second, to better tackle the reading comprehension tasks, we extend our fine-grained gating approach to modeling the interaction between documents and queries.
2 RELATED WORK
Hybrid word-character models have been proposed to take advantages of both word-level and character-level representations. Ling et al. (2015) introduce a compositional character to word (C2W) model based on bidirectional LSTMs. Kim et al. (2016) describe a model that employs a convolutional neural network (CNN) and a highway network over characters for language modeling. Miyamoto & Cho (2016) use a gate to adaptively find the optimal mixture of the character-level and word-level inputs. Yang et al. (2016a) employ deep gated recurrent units on both character and word levels to encode morphology and context information. Concurrent to our work, Rei et al. (2016) employed a similar gating idea to combine word-level and character-level representations, but their focus is on low-level sequence tagging tasks and the gate is not conditioned on linguistic features.
The gating mechanism is widely used in sequence modeling. Long short-term memory (LSTM) networks (Hochreiter & Schmidhuber, 1997) are designed to deal with vanishing gradients through the gating mechanism. Similar to LSTM, Gated Recurrent Unit (GRU) was proposed by Cho et al. (2014), which also uses gating units to modulate the flow of information. The gating mechanism can also be viewed as a form of attention mechanism (Bahdanau et al., 2015; Yang et al., 2016b) over two inputs.
Similar to the idea of gating, multiplicative integration has also been shown to provide a benefit in various settings. Yang et al. (2014) find that multiplicative operations are superior to additive operations in modeling relations. Wu et al. (2016) propose to use Hadamard product to replace sum operation in recurrent networks, which gives a significant performance boost over existing RNN models. Dhingra et al. (2016a) use a multiplicative gating mechanism to achieve state-of-the-art results on question answering benchmarks.
Reading comprehension is a challenging task for machines. A variety of models have been proposed to extract answers from given text (Hill et al., 2016; Kadlec et al., 2016; Trischler et al., 2016; Chen
et al., 2016; Sordoni et al., 2016; Cui et al., 2016). Yu et al. (2016) propose a dynamic chunk reader to extract and rank a set of answer candidates from a given document to answer questions. Wang & Jiang (2016) introduce an end-to-end neural architecture which incorporates match-LSTM and pointer networks (Vinyals et al., 2015).
3 FINE-GRAINED GATING
In this section, we will describe our fine-grained gating approach in the context of reading comprehension. We first introduce the settings of reading comprehension tasks and a general neural network architecture. We will then describe our word-character gating and document-query gating approaches respectively.
3.1 READING COMPREHENSION SETTING
The reading comprehension task involves a document P = (p1, p2, · · · , pM ) and a query Q = (q1, q2, · · · , qN ), where M and N are the lengths of the document and the query respectively. Each token pi is denoted as (wi,Ci), where wi is a one-hot encoding of the token in the vocabulary and Ci is a matrix with each row representing a one-hot encoding of a character. Each token in the query qj is similarly defined. We use i as a subscript for documents and j for queries. The output of the problem is an answer a, which can either be an index or a span of indices in the document.
Now we describe a general architecture used in this work, which is a generalization of the gated attention reader (Dhingra et al., 2016a). For each token in the document and the query, we compute a vector representation using a function f . More specifically, for each token pi in the document, we have h0i = f(wi,Ci). The same function f is also applied to the tokens in the query. Let H0p and Hq denote the vector representations computed by f for tokens in documents and queries respectively. In Section 3.2, we will discuss the “word-character” fine-grained gating used to define the function f .
Suppose that we have a network of K layers. At the k-th layer, we apply RNNs on Hk−1p and Hq to obtain hidden states Pk and Qk, where Pk is aM×dmatrix and Qk is aN×dmatrix with d being the number of hidden units in the RNNs. Then we use a function r to compute a new representation for the document Hkp = r(P
k,Qk). In Section 3.3, we will introduce the “document-query” finegrained gating used to define the function r.
After going through K layers, we predict the answer index a using a softmax layer over hidden states Hkp . For datasets where the answer is a span of text, we use two softmax layers for the start and end indices respectively.
3.2 WORD-CHARACTER FINE-GRAINED GATING
Given a one-hot encoding wi and a character sequence Ci, we now describe how to compute the vector representation hi = f(wi,Ci) for the token. In the rest of the section, we will drop the subscript i for notation simplicity.
We first apply an RNN on C and take the hidden state in the last time step c as the character-level representation (Yang et al., 2016a). Let E denote the token embedding lookup table. We perform a matrix-vector multiplication Ew to obtain a word-level representation. We assume c and Ew have the same length de in this work.
Previous methods defined f using the word-level representation Ew (Collobert et al., 2011), the character-level representation c (Ling et al., 2015), or the concatenation [Ew; c] (Yang et al., 2016a). Unlike these methods, we propose to use a gate to dynamically choose between the word-level and character-level representations based on the properties of the token. Let v denote a feature vector that encodes these properties. In this work, we use the concatenation of named entity tags, partof-speech tags, binned document frequency vectors, and the word-level representations to form the feature vector v. Let dv denote the length of v.
The gate is computed as follows: g = σ(Wgv + bg)
where Wg and bg are the model parameters with shapes de× dv and de, and σ denotes an elementwise sigmoid function.
The final representation is computed using a fine-grained gating mechanism,
h = f(c,w) = g c+ (1− g) (Ew)
where denotes element-wise product between two vectors. An illustration of our fine-grained gating mechanism is shown in Figure 1. Intuitively speaking, when the gate g has high values, more information flows from the character-level representation to the final representation; when the gate g has low values, the final representation is dominated by the word-level representation.
Though Miyamoto & Cho (2016) also use a gate to choose between word-level and character-level representations, our method is different in two ways. First, we use a more fine-grained gating mechanism, i.e., vector gates rather than scalar gates. Second, we condition the gate on features that better reflect the properties of the token. For example, for noun phrases and entities, we would expect the gate to bias towards character-level representations because noun phrases and entities are usually less common and display richer morphological structure. Experiments show that these changes are key to the performance improvements for reading comprehension tasks.
Our approach can be further generalized to a setting of multi-level networks so that we can combine multiple levels of representations using fine-grained gating mechanisms, which we leave for future work.
3.3 DOCUMENT-QUERY FINE-GRAINED GATING
Given the hidden states Pk and Qk, we now describe how to compute a representation Hk that encodes the interactions between the document and the query. In this section, we drop the superscript k (the layer number) for notation simplicity. Let pi denote the i-th row of P and qj denote the j-row of Q. Let dh denote the lengths of pi and qj .
Attention-over-attention (AoA) (Cui et al., 2016) defines a dot product between each pair of tokens in the document and the query, i.e., pTi qj , followed by row-wise and column-wise softmax nonlinearities. AoA imposes pair-wise interactions between the document and the query, but using a dot product is potentially not expressive enough and hard to generalize to multi-layer networks. The gated attention (GA) reader (Dhingra et al., 2016a) defines an element-wise product as pi gi where gi is a gate computed by attention mechanism on the token pi and the entire query. The intuition for the gate gi is to attend to important information in the document. However, there is no direct pair-wise interaction between each token pair.
We present a fine-grained gating method that combines the advantages of the above methods (i.e., both pairwise and element-wise). We compute the pairwise element-wise product between the hidden states in the document and the query, as shown in Figure 2. More specifically, for pi and qj , we have
Iij = tanh(pi qj) where qj can be viewed as a gate to filter the information in pi. We then use an attention mechanism over Iij to output hidden states hi as follows
hi = ∑ j softmax(uTh Iij +w T i wjbh1 + bh2)Iij
where uh is a dv-dimensional model parameter, bh1 and bh2 are scalar model parameters, wi and wj are one-hot encodings for pi and qj respectively. We additionally use one-hot encodings in the attention mechanism to reinforce the matching between the same tokens since such information is not fully preserved in Iij when k is large. The softmax nonlinearity is applied over all j’s. The final hidden states H are formed by concatenating the hi’s for each token pi.
4 EXPERIMENTS
We first present experimental results on the Twitter dataset where we can rule out the effects of different choices of network architectures, to demonstrate the effectiveness of our word-character fine-grained gating approach. Later we show experiments on more challenging datasets on reading comprehension to further show that our approach can be used to improve the performance on highlevel NLP tasks as well.
4.1 EVALUATING WORD-CHARACTER GATING ON TWITTER
We evaluate the effectiveness of our word-character fine-grained gating mechanism on a social media tag prediction task. We use the Twitter dataset and follow the experimental settings in Dhingra et al. (2016b). We also use the same network architecture upon the token representations, which is an LSTM layer followed by a softmax classification layer (Dhingra et al., 2016b). The Twitter dataset consists of English tweets with at least one hashtag from Twitter. Hashtags and HTML tags have been removed from the body of the tweet, and user names and URLs are replaced with special tokens. The dataset contains 2 million tweets for training, 10K for validation and 50K for testing, with a total of 2,039 distinct hashtags. The task is to predict the hashtags of each tweet.
We compare several different methods as follows. Word char concat uses the concatenation of word-level and character-level representations as in Yang et al. (2016a); word char feat concat concatenates the word-level and character-level representations along with the features described in
Section 3.2; scalar gate uses a scalar gate similar to Miyamoto & Cho (2016) but is conditioned on the features; fine-grained gate is our method described in Section 3.2. We include word char feat concat for a fair comparison because our fine-grained gating approach also uses the token features.
The results are shown in Table 1. We report three evaluation metrics including precision@1, recall@10, and mean rank. Our method outperforms character-level models used in Dhingra et al. (2016b) by 2.29%, 2.69%, and 2.5 points in terms of precision, recall and mean rank respectively. We can observe that scalar gating approach (Miyamoto & Cho, 2016) can only marginally improve over the baseline methods, while fine-grained gating methods can substantially improve model performance. Note that directly concatenating the token features with the character-level and word-level representations does not boost the performance, but using the token features to compute a gate (as done in fine-grained gating) leads to better results. This indicates that the benefit of fine-grained gating mainly comes from better modeling rather than using additional features.
4.2 PERFORMANCE ON READING COMPREHENSION
After investigating the effectiveness of the word-character fine-grained gating mechanism on the Twitter dataset, we now move on to a more challenging task, reading comprehension. In this section, we experiment with two datasets, the Children’s Book Test dataset (Hill et al., 2016) and the SQuAD dataset (Rajpurkar et al., 2016).
4.2.1 CLOZE-STYLE QUESTIONS
We evaluate our model on cloze-style question answering benchmarks.
The Children’s Book Test (CBT) dataset is built from children’s books. The whole dataset has 669,343 questions for training, 8,000 for validation and 10,000 for testing. We closely follow the setting in Dhingra et al. (2016a) and incrementally add different components to see the changes in performance. For the fine-grained gating approach, we use the same hyper-parameters as in Dhingra et al. (2016a) except that we use a character-level GRU with 100 units to be of the same size as the word lookup table. The word embeddings are updated during training.
In addition to different ways of combining word-level and character-level representations, we also compare two different ways of integrating documents and queries: GA refers to the gated attention reader (Dhingra et al., 2016a) and FG refers to our fine-grained gating described in Section 3.3.
The results are reported in Table 2. We report the results on common noun (CN) questions and named entity (NE) questions, which are two widely used question categories in CBT. Our finegrained gating approach achieves new state-of-the-art performance on both settings and outperforms the current state-of-the-art results by up to 1.76% without using ensembles. Our method outperforms the baseline GA reader by up to 2.4%, which indicates the effectiveness of the fine-grained gating mechanism. Consistent with the results on the Twitter dataset, using word-character fine-grained gating can substantially improve the performance over concatenation or scalar gating. Furthermore, we can see that document-query fine-grained gating also contributes significantly to the final results.
We also apply our fine-grained gating model to the Who Did What (WDW) dataset (Onishi et al., 2016). As shown in Table 3, our model achieves state-of-the-art results compared to strong baselines. We fix the word embeddings during training.
4.2.2 SQUAD
The Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset collected recently (Rajpurkar et al., 2016). It contains 23,215 paragraphs come from 536 Wikipedia articles. Unlike other reading comprehension datasets such as CBT, the answers are a span of text rather than a single word. The dataset is partitioned into a training set (80%, 87,636 question-answer pairs), a development set (10%, 10,600 question-answer pairs) and a test set which is not released.
We report our results in Table 4. “Exact match” computes the ratio of questions that are answered correctly by strict string comparison, and the F1 score is computed on the token level. We can observe that both word-character fine-grained gating and document-query fine-grained gating can substantially improve the performance, leading to state-of-the-art results among published papers. Note that at the time of submission, the best score on the leaderboard is 0.716 in exact match and 0.804 in F1 without published papers. A gap exists because our architecture described in Section 3.1 does not specifically model the answer span structure that is unique to SQuAD. In this work, we focus on this general architecture to study the effectiveness of fine-grained gating mechanisms.
4.3 VISUALIZATION AND ANALYSIS
We visualize the model parameter Wg as described in Section 3.2. For each feature, we average the corresponding weight vector in Wg . The results are described in Figure 3. We can see that named entities like “Organization” and noun phrases (with tags “NNP” or “NNPS”) tend to use characterlevel representations, which is consistent with human intuition because those tokens are usually infrequent or display rich morphologies. Also, DOCLEN-4, WH-adverb (“WRB”), and conjunction (“IN” and “CC”) tokens tend to use word-level representations because they appear frequently.
We also sample random span of text from the SQuAD dataset, and visualize the average gate values in Figure 4. The results are consistent with our observations in Figure 3. Rare tokens, noun phrases, and named entities tend to use character-level representations, while others tend to use word-level representations. To further justify this argument, we also list the tokens with highest and lowest gate values in Table 5.
5 CONCLUSIONS
We present a fine-grained gating mechanism that dynamically combines word-level and characterlevel representations based on word properties. Experiments on the Twitter tag prediction dataset show that fine-grained gating substantially outperforms scalar gating and concatenation. Our method also improves the performance on reading comprehension and achieves new state-of-the-art results on CBT and WDW. In our future work, we plan to to apply the fine-grained gating mechanism for combining other levels of representations, such as phrases and sentences. It will also be intriguing to integrate NER and POS networks and learn the token representation in an end-to-end manner.
ACKNOWLEDGMENTS
This work was funded by NVIDIA, the Office of Naval Research Scene Understanding grant N000141310721, the NSF grant IIS1250956, and Google Research.

SUMMARY.
The paper proposes a gating mechanism to combine word embeddings with character-level word representations.
The gating mechanism uses features associated to a word to decided which word representation is the most useful.
The fine-grain gating is applied as part of systems which seek to solve the task of cloze-style reading comprehension question answering, and Twitter hashtag prediction.
For the question answering task, a fine-grained reformulation of gated attention for combining document words and questions is proposed.
In both tasks the fine-grain gating helps to get better accuracy, outperforming state-of-the-art methods on the CBT dataset and performing on-par with state-of-the-art approach on the SQuAD dataset.
----------
OVERALL JUDGMENT
This paper proposes a clever fine-grained extension of a scalar gate for combining word representation.
It is clear and well written. It covers all the necessary prior work and compares the proposed method with previous similar models.
I liked the ablation study that shows quite clearly the impact of individual contributions.
And I also liked the fact that some (shallow) linguistic prior knowledge e.g., pos tags ner tags, frequency etc. has been used in a clever way.
It would be interesting to see if syntactic features can be helpful.


Machine learning models of music typically break down the task of composition into a chronological process, composing a piece of music in a single pass from beginning to end. On the contrary, human composers write music in a nonlinear fashion, scribbling motifs here and there, often revisiting choices previously made. We explore the use of blocked Gibbs sampling as an analogue to the human approach, and introduce COCONET, a convolutional neural network in the NADE family of generative models (Uria et al., 2016). Despite ostensibly sampling from the same distribution as the NADE ancestral sampling procedure, we find that a blocked Gibbs approach significantly improves sample quality. We provide evidence that this is due to some conditional distributions being poorly modeled. Moreover, we show that even the cheap approximate blocked Gibbs procedure from Yao et al. (2014) yields better samples than ancestral sampling. We demonstrate the versatility of our method on unconditioned polyphonic music generation. Step 0 Step 1 Step 4 Step 16 Step 64 Ground Truth Figure 1: Ancestral inpainting of a corrupted Bach chorale by COCONET. Colors are used to distinguish the four voices. Grayscale heatmaps show predictions p(xi | xC). The pitch sampled in the current step is indicated by a rectangular outline. The original Bach chorale is shown in the bottom right. Step 0 shows the corrupted Bach chorale. Step 64 shows the result. ∗Work done while at Google Brain. †Work done while at Google Brain.

The paper tackles the task of music generation. They use an orderless NADE model for the task of "fill in the notes". Given a roll of T timesteps of pitches, they randomly mask out some pitches, and the model is trained to predict the missing notes. This follows how the orderless NADE model can be trained. During sampling, one normally follows an ancestral sampling procedure. For this, an ordering is defined over outputs, and one runs the model on the current input, samples one of the outputs according to the order, adds this output to the next input, and continues this procedure until all outputs have been sampled. The key point of the paper is that this is a bad sampling strategy. Instead, they suggest the strategy of Yao et al. 2014, which uses a blocked Gibbs sampling approach. The blocked Gibbs strategy instead masks N inputs randomly and independently, samples them, and repeats this procedure. The point of this strategy is the make sure the sampling chain mixes well, which will happen for large N. However, since the samples are independent, having a large N gives incoherent samples. Thus, the authors follow an annealed schedule for N, making it smaller over time, which will eventually reduce to ancestral sampling (giving global structure to the sample). They conduct a variety of experiments involving both normal metrics and human evaluations, and find that this blocked Gibbs sampling outperforms other sampling procedures.
This is a well written paper - great job.
My main problem with the paper is that having read Uria and Yao, I don't know how much I have learned from this work in the context of this being an ICLR submission. If this was submitted to some computational music / art conference, this paper would be a clear accept. However, for ICLR, I don't see enough novelty compared with previous works this builds upon. Orderless NADE is an established model. The blocked Gibbs sampling and annealing scheme are basically the exact same one used in Yao. Thus, the main novelty of this paper is its application to the music domain, and finding that Yao's method works better for sampling music. This is a good contribution, but more tailored to those working in the music domain. If the authors found that these results also hold for other domains like images (e.g. on CIFAR / tiny Imagenet) and text (e.g. document generation), then I would change my mind and accept this paper for ICLR. Even just trying musical domains other than Bach chorales would be useful. However, as it stands, the experiments are not convincing enough.