The number of word embedding models is growing every year. Most of them learn word embeddings based on the co-occurrence information of words and their context. However, it’s still an open question what is the best definition of context. We provide the first systematical investigation of different context types and representations for learning word embeddings. We conduct comprehensive experiments to evaluate their effectiveness under 4 tasks (21 datasets), which give us some insights about context selection. We hope that this paper, along with the published code, can serve as a guideline of choosing context for our community.
1 INTRODUCTION
Recently, there is a growing research interest on word embedding models, where words are embedded into low-dimensional real vectors. Words that share similar meanings tend to have short distances in the vector space. The trained word embeddings are not only useful by themselves (e.g. used for calculating word similarities) but also effective when used as the input of the downstream models, such as chunking, tagging (Collobert & Weston, 2008; Collobert et al., 2011), parsing (Socher et al., 2011), text classification Socher et al. (2013); Kim (2014) and speech recognition (Schwenk, 2007).
For almost all word embedding models, the training objectives are based on Distributed Hypothesis (Harris, 1954), which can be stated as: “words that occur in the same contexts tend to have similar meanings”. The “context” is usually defined as the words which precede and follow the target word within some fixed distance in most word embedding models with various architectures (Bengio et al., 2003; Mnih & Hinton, 2007; Mikolov et al., 2013b; Pennington et al., 2014). Among them, Global Vectors (GloVe) proposed by Pennington et al. (2014), Continuous Skip-Gram (CSG) 1 and Continuous Bag-Of-Words (CBOW) proposed by Mikolov et al. (2013a) achieve the state-of-the-art results on a wide range of linguistic tasks, and scales well to corpus with billion words.
Since the simplest way of defining context is used by these classic word embedding models, it is worth investigating the best definition of “context”. For example, 1) the “context” can also be defined as the syntactic neighbours of the target word based on dependency parse tree. Is dependencybased context more reasonable than linear context defined by the positional neighbours of the target word in plain texts? 2) do the relative position of each contextual word and the relation between
1Many researches refer Continuous Skip-Gram as SG. However, in order to distinguish linear (continuous) context and dependency-based context, we refer it as CSG.
contextual word and target word contribute to the learning process? 3) do different word embedding models have preference for different context? This paper tries to answer these questions based on the experimental results according to different tasks.
Previously, Levy & Goldberg (2014a); Ling et al. (2015) 2 improve the CSG and CBOW by introducing position-aware context, where each contextual word is associated with their relative position to the target word. Levy & Goldberg (2014c) proposes DEPS, which considers the words that are connected to target word in dependency parse tree as context. We classify these models based on different context types (linear or dependency-based) and different context representations (word or bound word) in Table 1. We implement the models that previously not proposed and give systematical comparisons of different context types and context representations on popular CSG, CBOW, and GloVe. Comprehensive experiments are conducted on a wide range of word similarity, word analogy, sequence labeling, and text classification datasets. Some insights about determining the context in different situations are presented. We expect this paper to be an useful complementary in the word embedding literature.
2 METHODOLOGY
2.1 CONTEXT TYPES
It is necessary to discover more effective ways to define “context”. In the current literature, there are two types of context: linear (most word embedding models) and dependency-based (DEPS (Levy & Goldberg, 2014c)). Linear context is defined as the positional neighbours of the target word in text. Dependency based context is defined as the syntactic neighbours of the target word based on dependency parse tree, as shown in Figure 1 3.
Compared to linear context, dependency-based context can capture more long-range context. For example, linear context does not consider the word-context pair (discovers, telescope), while dependency-based context contains these information. Dependency-based context can also exclude some uninformative word-context pairs like (with, star) and (telescope, with).
2.2 CONTEXT REPRESENTATIONS
In the CSG and CBOW, context is represented by words without additional information. Levy & Goldberg (2014a); Ling et al. (2015) improve them by introducing position-bound words, where each contextual word is associated with their relative position to the target word. This allows CSG
2In these two papers, the description of position-aware context are quite different. However, their ideas is actually identical.
3This example is originally shown in Levy & Goldberg (2014c)
in the text. The context vocabulary C is thus identical to the word vocabulary W . However, this restriction is not required by the model; contexts need not correspond to words, and the number of context-types can be substantially larger than the number of word-types. We generalize SKIPGRAM by replacing the bag-of-words contexts with arbitrary contexts.
In this paper we experiment with dependencybased syntactic contexts. Syntactic contexts capture different information than bag-of-word contexts, as we demonstrate using the sentence “Australian scientist discovers star with telescope”.
Linear Bag-of-Words Contexts This is the context used by word2vec and many other neural embeddings. Using a window of size k around the target word w, 2k contexts are produced: the k words before and the k words after w. For k = 2, the contexts of the target word w are w−2, w−1, w+1, w+2. In our example, the contexts of discovers are Australian, scientist, star, with.2
Note that a context window of size 2 may miss some important contexts (telescope is not a context of discovers), while including some accidental ones (Australian is a context discovers). Moreover, the contexts are unmarked, resulting in discovers being a context of both stars and scientist, which may result in stars and scientists ending up as neighbours in the embedded space. A window size of 5 is commonly used to capture broad topical content, whereas smaller windows contain more focused information about the target word.
Dependency-Based Contexts An alternative to the bag-of-words approach is to derive contexts based on the syntactic relations the word participates in. This is facilitated by recent advances in parsing technology (Goldberg and Nivre, 2012; Goldberg and Nivre, 2013) that allow parsing to syntactic dependencies with very high speed and near state-of-the-art accuracy.
After parsing each sentence, we derive word contexts as follows: for a target word w with modifiers m1, . . . ,mk and a head h, we consider the contexts (m1, lbl1), . . . , (mk, lblk), (h, lbl−1h ),
2word2vec’s implementation is slightly more complicated. The software defaults to prune rare words based on their frequency, and has an option for sub-sampling the frequent words. These pruning and sub-sampling happen before the context extraction, leading to a dynamic window size. In addition, the window size is not fixed to k but is sampled uniformly in the range [1, k] for each word.
WORD CONTEXTS
australian scientist/amod−1
scientist australian/amod, discovers/nsubj−1
discovers scientist/nsubj, star/dobj, telescope/prep with star discovers/dobj−1 telescope discovers/prep with−1
Figure 1: Dependency-based context extraction example. Top: preposition relations are collapsed into single arcs, making telescope a direct modifier of discovers. Bottom: the contexts extracted for each word in the sentence.
where lbl is the type of the dependency relation between the head and the modifier (e.g. nsubj, dobj, prep with, amod) and lbl−1 is used to mark the inverse-relation. Relations that include a preposition are “collapsed” prior to context extraction, by directly connecting the head and the object of the preposition, and subsuming the preposition itself into the dependency label. An example of the de-
pendency context extraction is given in Figure 1.
Notice that syntactic dependencies are both
more inclusive and more focused than bag-ofwords. They capture relations to words that are far apart and thus “out-of-reach” with small window bag-of-words (e.g. the instrument of discover is telescope/prep with), and also filter out “coincidental” contexts which are within the window but not directly related to the target word (e.g. Australian is not used as the context for discovers). In addition, the contexts are typed, indicating, for example, that stars are objects of discovery and scientists are subjects. We thus expect the syntactic contexts to yield more focused embeddings, capturing more functional and less topical similarity.
4 Experiments and Evaluation
We experiment with 3 training conditions: BOW5 (bag-of-words contexts with k = 5), BOW2 (same, with k = 2) and DEPS (dependency-based syntactic contexts). We modified word2vec to support arbitrary contexts, and to output the context embeddings in addition to the word embeddings. For bag-of-words contexts we used the original word2vec implementation, and for syntactic contexts, we used our modified version. The negative-sampling parameter (how many negative contexts to sample for every correct one) was 15.
and CBOW to distinguish different sequential positions and capture context’s structural information. We name the method that bind additional information to contextual word as bound context representation, as opposite to unbound context representation where word is used alone.
For dependency-based context, the original DEPS uses bound context representation by default: words are associate with their dependency relation to the target word. Similar to bound context representation in linear context type, this allows word embedding models to capture more dependency information. An example is shown in Table 2
context representation
context type linear dependency-based
unbound australian, scientist, star, with scientist, star, telescope bound australian/-2, scientist/-1, star/+1, with/+2 scientist/nsubj, star/do j, telescope/prep with
Note that bound context representation is sparse, especially for dependency-based context. There are 47 dependency relations in dependency parse tree. Although not every combination of dependency relations and words appear in the word-context pair collection, it still enlarges the context vocabulary about 5 times in practice. In this paper, we investigate the simpler context representation where no
ependenc relation are considered. T is also makes a fair c mparison with linear context models like CSG, CBOW and GloVe, since they do not use bound context representation neither.
Table 3: Illustration of collection P , M and M for sentence “australian scientist discovers star with telescope”. Unbound context representation is used in this example. Words in the collections are Bold and contexts in the collections are Normal.
linear (window size equals 1) dependency-based
P (australian, scientist) (scientist, australian) (scientist, discovers) (discovers, scientist) (discovers, star) . . .
(australian, scientist) (scientist, australian) (scientist, discovers) (discovers, scientist) (discovers, star) (discovers, telescope) . . .
M (australian, scientist) (scientist, australian, discovers) (discovers, scientist, star) . . .
(australian, scientist) (scientist, australian, discovers) (discovers, scientist, star, telescope) . . .
M (australian, scientist, 1) (scientist, australian, 1) (scientist, discovers, 1) (discovers, scientist, 1) (discovers, star, 1) . . . (australian, scientist, 1) (scientist, australian, 1) (scientist, discovers, 1) (discovers, scientist, 1) (discovers, star, 1) (discovers, telescope, 1) . . .
3
2.3 GENERALIZATION
For convenient and general representation, we first define the collection of word-context pairs as P . P can be merged based on the words to form a collection M with size of |C|. Each element (w, c1, c2, .., cnw) ∈M is the wordw and its contexts, where nw is the number of wordw’s contexts. P can also be merge based on both words and contexts to form a collection M . Each element (w, c,#(w, c)) ∈ M is the word w, context c, and the times they appears in collection P . An example of these collections is shown in Table 3.
2.3.1 GENERALIZED BAG-OF-WORDS
The objective function of Generalized Bag-Of-Words (GBOW) with negative sampling technique is defined as:∑ (w,c1,..,cnw )∈M log p ( w ∣∣∣∣∣ nw∑ i=1 ~ci ) = ∑ (w,c1,..,cnw )∈M [ log σ ( ~w · nw∑ i=1 ~ci ) − K∑ k=1 log σ ( ~wN · nw∑ 1=i ~ci )] (1) where σ is the sigmoid function, K is the negative sampling size, ~w and ~c is the vector for word w and c respectively. The negatively sampled random word wN is selected based on its unigram distribution ( #(w)∑
w #(w) )ds, where #(w) is the number of times that word w appears in the corpus, ds
is the distribution smoothing hyper-parameter which is usually defined as 0.75. Note that in original CBOW with negative sampling technique, the probability is actually p (c| ∑
~wi) instead of p (w| ∑ ~ci). In other word, original CBOW uses the sum of word vectors to predict context. This works well for linear context. But for dependency-based context with bound word, there is only one contextual word available for prediction. For example in Figure 1, the context “scientist/nsubj” can only be predicted by word “discovers”. However, a word can be predicted by the sum of several contexts. Due to this reason, we exchange the role of word and context in GBOW. The negative sampling objective is also changed from context cN to word wN .
2.3.2 GENERALIZED SKIP-GRAM
For generalized Skip-Gram (GSG), the definition is straightforward and actually need no modification of the objective function, as discussed in (Levy & Goldberg, 2014a). However, in order to make it consistent with our GBOW, we also exchange the role of word and context. the objective function of GSG is defined as:∑
(w,c)∈P
log p (w|~c) = ∑
(w,c)∈P
[ log σ (~w · ~c)−
K∑ k=1 log σ ( ~wN · ~c)
] (2)
2.3.3 GLOVE
Unlike GSG and GBOW, GloVe explicitly optimizes a log-bilinear regression model based on word co-occurrence matrix. Since GloVe is already a very generalized model, with our previous defined collection M , the final objective function is easily written as:∑
(w,c)∈M
f(#(w, c))(~w · ~c+ ~bw + ~bc − log#(w, c)) (3)
where f is a non-decreasing weighting function and ensures the weight of large #(w, c) to be relatively small.
Note that the inputs of GSG, GBOW and Glove are the collection P , M and M respectively. Once the corpus and hyper-parameters are fixed, these collections (and thus the learned word embedings) are determined only by the choice of context types and representations.
3 EXPERIMENTS
We evaluate the effectiveness of different context types and representations on word similarity task, word analogy task, sequence labeling task, and text classification task. In this Section, we first
describe the training details of word embedding models. We then report and discuss the experimental results on each task. The full experimental results can be found in the Appendix.
3.1 TRAINING DETAILS
The word2vecf toolkit 4 (Levy et al., 2015) extends the word2vec toolkit 5 (Mikolov et al., 2013b) to accept the input of collection P rather than raw corpus. This makes CSG model accept any arbitrary contexts (e.g. dependency-based context). However, CBOW model is not considered in that toolkit. We implement word2vecPM 6, a further extension of word2vecf, which supports both generalized SG and generalized BOW with the input of collection P and M respectively.
We use English Wikipedia (August 2013 dump) as the training corpus in all of our experiments. The Stanford CoreNLP (Manning et al., 2014) is used for dependency parsing. All words and contexts are converted to lower case after parsing. Words and contexts that appear less than 100 times in collection P and M are directly ignored. Note that this is slightly different from ignoring rare word that appear less than 100 times in corpus, since each word may appear more times in collection than that in corpus.
Most hyper-parameters are the same as Levy et al. (2015)’s best configuration. For example, negative sampling size K is set to 5 for GSG and 2 for GBOW. Distribution smoothing cds is set to 0.75. No dynamic context or “dirty” sub-sampling is used. The window size wn is fixed to 2 for constructing linear context, which insures the number of the (merged) word-context pair collection for both linear context and dependency-based context is comparable. The number of iteration is set to 2, 5 and 30 for GSG, GBOW and GloVe respectively. Unless otherwise noted, the number of word embedding dimension is set to 500. Since the aim of this paper is not comparing the performance of different word embedding models, the results of GSG, GBOW and GloVe are reported respectively.
3.2 WORD SIMILARITY TASK
Word similarity task aims at producing a semantic similarity score of a word pair, which is compared with the human label. The cosine distance is used for scoring similarities between two words, and measured by Spearman’s correlation. Six datasets are used in our experiments: WordSim353 (Finkelstein et al., 2001) with similarity and relatedness partition (Zesch et al., 2008; Agirre et al., 2009), MEN dataset (Bruni et al., 2012), Mechanical Turk dataset (Radinsky et al., 2011), Rare Words dataset (Luong et al., 2013), SimLex-999 dataset (Hill et al., 2016).
As shown in the numerical results in Table 4, there is no single model consistently outperform the rest across all datasets. Although the overall trend of GSG, GBOW and GloVe using different context types and representations is similar, GBOW seems more benefit from linear context than
4https://bitbucket.org/yoavgo/word2vecf 5http://code.google.com/p/word2vec/ 6https://github.com/libofang/word2vecPM
GSG and GloVe. GBOW takes the sum of context vectors as prediction function’s input, thus is less sensitive syntactic structure. In other words, since the “right” context is summed with other context, it’s information contributes less than that in GSG.
More conclusion could be conducted if we focus on the WordSim353 dataset with similarity and relatedness partition. It’s previously commonly believed that compared to linear context, dependencybased context can capture more functional similarity (e.g. tiger/cat) rather than topical similarity/relatedness (e.g., tiger/jungle) (Levy & Goldberg, 2014c; Melamud et al., 2016). However, these experiments do not distinguish the effect of different context representations: unbound representation is used for linear context ((Mikolov et al., 2013b)) while bound representation is used for dependency-based context ((Levy & Goldberg, 2014c)). Moreover, only CSG model is compared.
We revisit previous claims based on more systematical results. As shown in Figure 2’s upper-left sub-figure, compared to linear context (solid and dotted blue line), the better results of dependencybased context for GSG and GloVe (solid and dotted red line) on ws353’s similarity partition confirms its ability of capturing functional similarity. However, the good performance of dependency-based context for GSG do not fully transfer to GBOW. Although dependency-based context with bound representation for GBOW is still the best performer, dependency-based context with unbound representation for GBOW (solid red line) performs worst on ws353’s similarity partition. Note that the results are also reversed on ws353’s relatedness partition (Figure 2’s right sub-figures), which shows the use of linear context is more suitable for capturing topical relatedness.
3.3 WORD ANALOGY TASK
Word analogy task aims at answering the question like “a is to b as c is to ?”. For example, “London is to UK as Tokyo is to Japan”. We follow the evaluation protocol in Levy & Goldberg
(2014a), answering the questions using both 3CosAdd and 3CosMul. Our experiments show that 3CosMul works consistently better than 3CosAdd, thus only the results of 3CosMul are reported. We follow previous researches, use Google’s analogy dataset (Mikolov et al., 2013a) (with semantic and syntactic partition), MSR’s analogy dataset (Mikolov et al., 2013c), and BATS analogy dataset (Gladkova et al., 2016) in our experiments.
Numerical results are shown in Table 5. We observe that the context representation plays an important role in word analogy task. The choice of context representation (word or bound word) actually has much larger impact than the choice of context type (linear or dependency). The results on Google Syn dataset (Figure 3’s sub-figures in second column) is perhaps most evident. The performance of linear context and dependency-based context with unbound representation is similar.
However, when bound context representation is used, the performance of GSG and GBOW drops more than 30 percent for dependency-based context and around 20 percent for linear context. The main reason for this phenomenon may be that the bound representation already contains syntactic information, thus word embedding models can not learn it from the input word-context pairs. It can also be observed that GloVe is more sensitive to different context representations than Skip-Gram and CBOW, which is probably due to its explicitly defined/optimized objective function.
3.4 SEQUENCE LABELING TASK
Although intrinsic evaluations like word similarity and word analogy tasks could provide direct insights of different context types and representations, the experimental results above cannot be translated to typical uses of word embeddings. For example, these tasks aren’t necessarily correlated with downstream tasks’ accuracy, as shown in (Schnabel et al., 2015; Linzen, 2016; Chiu et al., 2016). More extrinsic tasks should be considered.
In this Subsection, we evaluate the effectiveness of different word embedding models with different contexts on sequence labeling task. Sequence labeling aims at automatically assigning words in texts with labels. Three sub-tasks are considered: Part-of-Speech Tagging (POS), Chunking and Named Entity Recognition (NER) . CoNLL 2000 shared task 7 is used as benchmark for POS and Chunking. CoNLL 2003 shared task 8 is used as benchmark for NER.
Recent advances on sequence labeling task are based on neural networks like Recurrent Neural Network, Convolutional Neural Network, and their combinations with Conditional Random Fields (Collobert et al., 2011; Huang et al., 2015; Ma & Hovy, 2016). These models all require word
7http://www.cnts.ua.ac.be/conll2000/chunking 8http://www.cnts.ua.ac.be/conll2003/ner
embeddings as input. Inspired by the evaluation protocol used in Kiros et al. (2015), we restrict the prediction to simple linear classifier. More precisely, the classifier’s input for predicting the label of word wi is simply the concatenation of vector ~wi−2, ~wi−1, ~wi, ~wi+1, ~wi+2. This ensures the quality of embedding models is directly evaluated, and their strengths and weaknesses are easily observed.
As shown in Figure 4, the overall trend of GSG, GBOW and GloVe is identical except on NER task. Linear context type (red line) works better than dependency-based (blue line) context type when unbound context representation is used. The results are reversed when bound context representation is used. Bound context representation (dotted linear) outperforms unbound context representation (solid linear) on all datasets. These results suggest that linear context type with unbound context representations (as in traditional CSG and CBOW) may not be the best choice of input word vectors for sequence labeling. Dependency-based context with bound context representations should be used instead. Again, similar to that on word analogy task, GloVe is more sensitive to different context representations than Skip-Gram and CBOW on sequence labeling task.
3.5 TEXT CLASSIFICATION TASK
Finally, we evaluate the effectiveness of different word embedding models with different contexts on text classification task. Text classification is one of the most popular and well-studied task in natural language processing. Recently, deep neural networks are dominant on this task (Socher et al., 2013; Kim, 2014; Dai & Le, 2015). They often need pre-trained word embeddings as inputs to improve their performances. Similar to our evaluation of sequence labeling, instead of building complex deep neural networks, we use a simpler classification method called Neural Bag-of-Words to directly evaluate the word embeddings: texts are first represented by the sum of their belonging words’ vectors, then a Logistic Regression Classifier is built upon them for classification.
Different word embedding models are evaluated on 5 text classification datasets. The first 3 datasets are sentence-level: short movie review sentiment (MR) (Pang & Lee, 2005), customer product reviews (CR) (Nakagawa et al., 2010), and subjectivity/objectivity classification (SUBJ) (Pang & Lee, 2004). The other 2 datasets are document-level with multiple sentences: full-length movie review (RT-2k) (Pang & Lee, 2004), and IMDB movie review (IMDB) (Maas et al., 2011).
As shown in Table 6, pre-trained word embeddings outperform random word embeddings by a large margin. This further strengthen previous researches that pre-trained word embeddings are crucial for text classification. Unlike that on previous tasks, different models’ results are actually very similar on text classification task. Overall, models which use bound context representation perform worse than those which use unbound context representation on all datasets except CR. The performances of models that use dependency-based context type and linear context type is comparable. These observations suggest that simple linear context type with unbound context representations (as in traditional CSG and CBOW) is still the best choice of pre-training word embeddings, which is already used in most researches.
4 RELATED WORK
Previously, there are researches which directly compare different word embedding models. ? compares 6 word embedding models using different corpora and hyper-parameters. Levy & Goldberg (2014b) shows the theoretical equivalence of CSG and PPMI matrix factorization. Levy et al. (2015) further discusses the connections between 4 word embedding models (PPMI, PPMI+SVD, CSG, GloVe) and re-evaluates them with the same hyper-parameters. Suzuki & Nagata (2015) investigates different configurations of CSG and Glove, then merges them into a unified form. Yin & Schutze (2016) proposes 4 ensemble methods and shows their effectiveness over individual word embeddings.
There are also researches which focus on evaluating different context types in learning word embeddings. Vulic & Korhonen (2016) compares CSG and dependency-based models on various languages. The results suggest that dependency-based models are able to detect functional similarity on English. However, the advantages of dependency-based context over linear context on other languages is not as promising as that on English. Bansal et al. (2014) investigates different embedding models for parsing task and shows that dependency-based context is more suitable than linear context. Melamud et al. (2016) investigate the performance of CSG, Deps and a substitute-based word embedding models (Yatbaz et al., 2012) 9, which shows that different types of intrinsic task have clear preference to particular types of contexts. On the other hand, for extrinsic task, the optimal context types need to be carefully tuned on specific dataset. However, context representations (word and bound) are not evaluated in these models. Moreover, they focus only on CSG model since it’s more general and intuitive for dependency-based context.
5 CONCLUSION
To the best of our knowledge, this paper provides the first systematical investigation of different context types and representations for learning word embeddings. We evaluate different models on 4 tasks with totally 21 datasets. Experimental results show that:
• Dependency-based context type does not get all the credit for capturing functional similarity. Bound context representation also plays an important role, especially for GBOW.
• Syntactic word analogy benefits less from bound context representation. Bound context representation already contains syntactic information, which makes it difficult to capture this information based on the input word-context pairs.
• Bound context representation is suitable for sequence labeling task, especially when it is used along with dependency-based context.
• On text classification task, different contexts do not affect the final performance much. Nonetheless, the use of pre-trained word embeddings is crucial and linear context type with unbound representation (Skip-Gram) is still the best choice.
• The overall tendency of models with different contexts is similar, especially for Skip-Gram and GloVe. GloVe is more sensitive to different contexts than Skip-Gram and CBOW. CBOW benefits most from linear context type.
In the spirit of transparent and reproducible experiments, the source code is published at https: //github.com/libofang/word2vecPM. We hope researchers will take advantage of our code for further improvements and applications to other tasks.
ACKNOWLEDGMENTS
We thank Omer Levy, Yoav Goldberg, and Ido Dagan for their implementation of word2vecf and evaluation scripts. It systematically investigated the effective of different hyper-parameters on various word embedding models. Both their code and paper have directly influenced us a lot.
This work is supported by National Natural Science Foundation of China (Grant No. 61472428 and No. 71271211), the Fundamental Research Funds for the Central Universities, the Research Funds
9We do not consider this context type in this paper since it performs consistently worse than other two context types, as shown in Melamud et al. (2016); Vulic & Korhonen (2016)
of Renmin University of China No. 14XNLQ06. This work is partially supported by ECNU-RUCInfoSys Joint Data Science Lab and a gift from Tencent.
REFERENCES Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. A
study on similarity and relatedness using distributional and wordnet-based approaches. In NAACL, pp. 19–27. Association for Computational Linguistics, 2009.
Mohit Bansal, Kevin Gimpel, and Karen Livescu. Tailoring continuous word representations for dependency parsing. In ACL, pp. 809–815, 2014.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155, 2003.
Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. Distributional semantics in technicolor. In ACL, pp. 136–145. Association for Computational Linguistics, 2012.
Billy Chiu, Anna Korhonen, and Sampo Pyysalo. Intrinsic evaluation of word vectors fails to predict extrinsic performance. In ACL, pp. 406–414, 2016.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, pp. 160–167. ACM, 2008.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537, 2011.
Andrew M. Dai and Quoc V. Le. Semi-supervised sequence learning. In NIPS, pp. 3079–3087, 2015.
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: The concept revisited. In WWW, pp. 406–414. ACM, 2001.
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy-based detection of morphological and semantic relations with word embeddings: What works and what doesnt. In Proceedings of naacl-hlt, pp. 8–15, 2016.
Zellig Harris. Distributional structure. Word, 10(23):146–162, 1954.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 2016.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.
Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pp. 1746–1751, 2014.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Skip-thought vectors. In NIPS, pp. 3294–3302, 2015.
Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In CoNLL, pp. 171–180, 2014a.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In NIPS, pp. 2177–2185, 2014b.
Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL, pp. 302–308, 2014c.
Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. TACL, 3:211–225, 2015.
Wang Ling, Chris Dyer, Alan W. Black, and Isabel Trancoso. Two/too simple adaptations of word2vec for syntax problems. In HLT-NAACL, pp. 1299–1304, 2015.
Tal Linzen. Issues in evaluating semantic spaces using word analogies. CoRR, abs/1606.07736, 2016.
Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In CoNLL, pp. 104–113, 2013.
Xuezhe Ma and Eduard H. Hovy. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In ACL, 2016.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, pp. 142–150, 2011.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In ACL System Demonstrations, pp. 55–60, 2014.
Oren Melamud, David McClosky, Siddharth Patwardhan, and Mohit Bansal. The role of context types and dimensionality in learning word embeddings. In HLT-NAACL, pp. 1030–1040, 2016.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In NIPS, pp. 3111–3119, 2013b.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, volume 13, pp. 746–751, 2013c.
Andriy Mnih and Geoffrey E. Hinton. Three new graphical models for statistical language modelling. In ICML, pp. 641–648, 2007.
Tetsuji Nakagawa, Kentaro Inui, and Sadao Kurohashi. Dependency tree-based sentiment classification using crfs with hidden variables. In NAACL, pp. 786–794. Association for Computational Linguistics, 2010.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL, pp. 271–278. Association for Computational Linguistics, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pp. 115–124. Association for Computational Linguistics, 2005.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532–1543, 2014.
Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pp. 337–346. ACM, 2011.
Tobias Schnabel, Igor Labutov, David M. Mimno, and Thorsten Joachims. Evaluation methods for unsupervised word embeddings. In EMNLP, pp. 649–657, 2015.
Holger Schwenk. Continuous space language models. Computer Speech & Language, 21(3):492– 518, 2007.
Richard Socher, Cliff Chiung-Yu Lin, Andrew Y. Ng, and Christopher D. Manning. Parsing natural scenes and natural language with recursive neural networks. In ICML, pp. 129–136, 2011.
Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, volume 1631, pp. 1642. Citeseer, 2013.
Jun Suzuki and Masaaki Nagata. A unified learning framework of skip-grams and global vectors. In ACL, pp. 186, 2015.
Ivan Vulic and Anna Korhonen. Is ”universal syntax” universally useful for learning distributed word representations? In ACL, pp. 518, 2016.
Mehmet Ali Yatbaz, Enis Sert, and Deniz Yuret. Learning syntactic categories using paradigmatic representations of word context. In EMNLP-CoNLL, pp. 940–951, 2012.
Wenpeng Yin and Hinrich Schutze. Learning word meta-embeddings. In ACL, pp. 327–332, 2016.
Torsten Zesch, Christof Müller, and Iryna Gurevych. Using wiktionary for computing semantic relatedness. In AAAI, volume 8, pp. 861–866, 2008.
APPENDIX

This paper investigates the issue of whether and how to use syntactic dependencies in unsupervised word representation learning models like CBOW or Skip-Gram, with a focus one the issue of bound (word+dependency type, 'She-nsubj') vs. unbound (word alone, 'She') representations for context at training time. The empirical results are extremely mixed, and no specific novel method consistently outperforms existing methods.
The paper is systematic and I have no major concerns about its soundness. However, I don't think that this paper is of broad interest to the ICLR community. The paper is focused on a fairly narrow detail of representation learning that is entirely specific to NLP, and its results are primarily negative. A short paper at an ACL conference would be a more reasonable target.


Modern convolutional networks, incorporating rectifiers and max-pooling, are neither smooth nor convex. Standard guarantees therefore do not apply. Nevertheless, methods from convex optimization such as gradient descent and Adam are widely used as building blocks for deep learning algorithms. This paper provides the first convergence guarantee applicable to modern convnets. The guarantee matches a lower bound for convex nonsmooth functions. The key technical tool is the neural Taylor approximation – a straightforward application of Taylor expansions to neural networks – and the associated Taylor loss. Experiments on a range of optimizers, layers, and tasks provide evidence that the analysis accurately captures the dynamics of neural optimization. The second half of the paper applies the Taylor approximation to isolate the main difficulty in training rectifier nets: that gradients are shattered. We investigate the hypothesis that, by exploring the space of activation configurations more thoroughly, adaptive optimizers such as RMSProp and Adam are able to converge to better solutions.
1 INTRODUCTION
Deep learning has achieved impressive performance on a range of tasks (LeCun et al., 2015). The workhorse underlying deep learning is gradient descent or backprop. Gradient descent has convergence guarantees in settings that are smooth, convex or both. However, modern convnets are neither smooth nor convex. Every winner of the ImageNet classification challenge since 2012 has used rectifiers which are not smooth (Krizhevsky et al., 2012; Zeiler & Fergus, 2014; Simonyan & Zisserman, 2015; Szegedy et al., 2015; He et al., 2015). Even in convex settings, convergence for nonsmooth functions is lower-bounded by 1/ √ N (Bubeck, 2015).
The paper’s main contribution is the first convergence result for modern convnets, Theorem 2. It applies to any neural net with a loss convex in the output of the net. The idea is simple: backprop constructs linear snapshots (gradients) of a neural net’s landscape; section 2 introduces neural Taylor approximations which
are used to construct Taylor losses as convex snapshots closely related to backprop. The online convex optimization framework (Zinkevich, 2003) then gives 1/ √ N convergence to the Taylor optimum, matching the lower bound in (Bubeck, 2015). Section 2.3 investigates the Taylor optimum and regret terms empirically. We observe that convergence to the Taylor optimum occurs at 1/ √ N in practice.
The nonsmoothness of rectifier nets is perhaps underappreciated. Fig. 1 shows a piecewise-linear (PL) function and its gradient. The gradient is discontinuous or shattered. Shattering is problematic for accelerated and Hessian-based methods which speed up convergence by exploiting the relationship between gradients at nearby points (Sutskever et al., 2013). The success of these methods on rectifier networks requires explanation since gradients at (nearby) points on different sides of a kink are not related. Further, the number of kinks grows exponentially with network depth (Pascanu et al., 2014; Telgarsky, 2016).
Section 3 addresses the success of adaptive optimizers in rectifier nets.1 Adaptive optimizers normalize gradients by their root-mean-square; e.g. AdaGrad, RMSProp, Adam and RadaGrad (Duchi et al., 2011; Hinton et al., 2012; Kingma & Ba, 2015; Krummenacher et al., 2016). Dauphin et al. (2015) argue that RMSProp is successful because it approximates the equilibriation matrix√
diag(H2) which approximates the absolute Hessian |H| (Dauphin et al., 2014). However, the argument is at best part of the story when gradients are shattered.
The only way an optimizer can estimate gradients of a shattered function is to compute them directly. Effective optimizers must therefore explore the space of smooth regions – the bound in theorem 2 is only as good as the optimum over the Taylor losses encountered during backprop. Observations 1 and 2 relate smooth regions in rectifier nets and the Taylor losses to configurations of active neurons. We hypothesize that root-mean-square normalization increases exploration through the set of smooth regions in a rectifier net’s landscape. Experiments in section 3.3 confirm the hypothesis.
1.1 COMPARISON WITH RELATED WORK
Researchers have applied convex techniques to neural networks. Bengio et al. (2006) show that choosing the number of hidden units converts neural optimization into a convex problem, see also Bach (2014). A convex multi-layer architectures are developed in Aslan et al. (2014); Zhang et al. (2016). However, these approaches have not achieved the practical success of convnets. In this work, we analyze convnets as they are rather than proposing a more tractable, but potentially less useful, model. A Taylor decomposition for neural networks was proposed in Montavon et al. (2015). They treat inputs as variable instead of weights and study interpretability instead of convergence.
Our results are closely related to Balduzzi (2016), which uses game-theoretic techniques to prove convergence in rectifier nets. The approach taken here is more direct and holds in greater generality.
2 CONVERGENCE OF NEURAL NETWORKS
Theorem 2 uses techniques and concepts from online convex optimization, which provide worst-case bounds in adversarial settings. We relate adversarial methods to deep learning through the Taylor loss.
Intuitively, backprop searches a vast nonconvex landscape with a linear flashlight (Taylor losses are a more sharply focused convex flashlight, see A.4). The adversary is the landscape: from backprop’s perspective its geometry is an unpredictable external force. The Taylor losses are convex problems that backprop de facto optimizes – the gradients of the actual and Taylor losses are identical. The Taylor optimum improves when, stepping over a kink, backprop shines its light on a new (better) region of the landscape (fig. 2). Regret quantifies the gap between the Taylor optimal loss and the losses incurred during training.
1For simplicity, we restrict to fully connected rectifier (ReLU) nets. The results also apply to convolutions, max-pooling, dropout, dropconnect, maxout, PReLUs and CReLUs (Srivastava et al., 2014; Wan et al., 2013; Goodfellow et al., 2013; He et al., 2015; Shang et al., 2016).
2.1 ONLINE CONVEX OPTIMIZATION
In online convex optimization (Zinkevich, 2003), a learner is given convex loss functions `1, . . . `N . On the nth round, the learner predicts Wn prior to observing `n, and then incurs loss `n(Wn). Since the losses are not known in advance, the performance of the learner is evaluated post hoc via the regret, the difference between the incurred losses and the optimal loss in hindsight:
Regret(N) := N∑ n=1 [ `n(Wn)︸ ︷︷ ︸
losses incurred by learner − `n(V∗)︸ ︷︷ ︸ optimal-in-hindsight
] where V∗ := argmin
V∈H [ N∑ n=1 `n(V) ] .
An algorithm has no-regret if limN→∞ Regret(N)/N → 0 for any sequence of losses with bounded gradients. For example, Kingma & Ba (2015) prove that Adam has no-regret: Theorem 1 (Adam has no-regret, (Kingma & Ba, 2015)). Suppose the the losses `n have bounded gradients ‖∇W `n(W)‖2 ≤ G and ‖∇W `n(W)‖∞ ≤ G for all W ∈ H and suppose that the weights chosen by the algorithm satisfy ‖Wm −Wn‖2 ≤ D and ‖Wm −Wn‖∞ ≤ D for all m,n ∈ {1, . . . , N}. Then Adam satisfies
Regret(N)/N ≤ O ( 1/ √ N ) for all N ≥ 1. (1)
The regret of gradient descent, AdaGrad (Duchi et al., 2011), mirror descent and a variety of related algorithms satisfy (1), albeit with different constant terms that are hidden in the big-O notation. Finally, the 1/ √ N rate is also lower-bound. It cannot be improved without additional assumptions.
2.2 NEURAL TAYLOR APPROXIMATION
Consider a network with L − 1 hidden layers and weight matrices W := {W1, . . . ,WL}. Let x0 denote the input. For hidden layers l ∈ {1, . . . , L − 1}, set al = Wl · xl−1 and xl = s(al) where s(·) is applied coordinatewise. The last layer outputs xL = aL = WL ·xL−1. Let pl denote the size of the lth layer; p0 is the size of the input and pL is the size of the output. Suppose the loss `(f , y) is smooth and convex in the first argument. The training data is (xd, yd)Dd=1. The network is trained on stochastic samples from the training data on a series of rounds n = 1, . . . , N . For simplicity we assume minibatch size 1; the results generalize without difficulty.
We recall backprop using notation from Martens et al. (2012). Let Jab denote the Jacobian matrix of the vector a with respect to the vector b. By the chain rule the gradient decomposes as
∇Wl ` ( fW(x0), y) = J
E L · JLL−1 · · ·Jl+1l︸ ︷︷ ︸
δl
⊗xl−1 = JEL︸︷︷︸ ∇f `(f ,y) · JLl ⊗ xl−1︸ ︷︷ ︸ ∇Wl fW(x0)=:Gl
(2)
where δl = JEl is the backpropagated error computed recursively via δl = δl+1 ·J l+1 l . 2 The middle expression in (2) is the standard representation of backpropagated gradients. The expression on the right factorizes the backpropagated error δl = JEL · JLl into the gradient of the loss JEL and the Jacobian JLl between layers, which describes gradient flow within the network.
The first-order Taylor approximation to a differentiable function f : R → R near a is Ta(x) = f(a) +f ′(a) · (x−a). The neural Taylor approximation for a fully connected network is as follows. Definition 1. The Jacobian tensor of layer l, Gl := JLl ⊗ xl−1, is the gradient of the output of the neural network with respect to the weights of layer l. It is the outer product of a (pL × pl)-matrix with a pl−1-vector, and so is a (pL, pl, pl−1)-tensor.
Given Gl and (pl× pl−1)-matrix V, the expression 〈Gl,V〉 := JLl ·V ·xl−1 is the pL-vector computed via matrix-matrix-vector multiplication. The neural Taylor approximation to f in a neighborhood of Wn, given input xn0 , is the first-order Taylor expansion
Tn(V) := fWn(x n 0 ) + L∑ l=1 〈 Gl,Vl −Wnl 〉 ≈ fV(xn0 ).
Finally, the Taylor loss of the network on round n is T n(V) = `(Tn(V), yn). 2Note: we suppress the dependence of the Jacobians on the round n to simplify notation.
The Taylor approximation to layer l is Tnl (Vl) := fWn(xn0 ) + 〈 Gl,Vl −Wnl 〉 . We can also construct the Taylor approximation to neuron α in layer l. Let the pL-vector JLα := J L l [ :, α] denote the Jacobian with respect to neuron α and let the (pL × pl−1)-matrix Gα := JLα ⊗ xl−1 denote the Jacobian with respect to the weights of neuron α. The Taylor approximation to neuron α is Tnα(Vα) := fWn(x n 0 ) + 〈 Gα,Vα −Wnα 〉 .
The Taylor losses are the simplest non-trivial (i.e. non-affine) convex functions encoding the information generated by backprop, see section A.4.
The following theorem provides convergence guarantees at mutiple spatial scales: network-wise, layer-wise and neuronal. See sections A.2 for a proof of the theorem. It is not currently clear which scale provides the tightest bound. Theorem 2 (no-regret relative to Taylor optimum). Suppose, as in Theorem 1, the Taylor losses have bounded gradients and the weights of the neural network have bounded diameter during training. Suppose the neural net is optimized by an algorithm with Regret(N) ≤ O( √ N) such as gradient descent, AdaGrad, Adam or mirror descent.
• Network guarantee: The running average of the training error of the neural network satisfies
1
N N∑ n=1 ` (fWn(x n 0 ), y
n)︸ ︷︷ ︸ running average of training errors ≤ min V
{ 1
N N∑ n=1 T n(V) } ︸ ︷︷ ︸
Taylor optimum
+ O ( 1√ N ) ︸ ︷︷ ︸ Regret(N)/N . (3)
• Layer-wise / Neuron-wise guarantee: The Taylor loss of [layer-l / neuron-α] is T nl/α(Vl/α) := `(T n l/α(Vl/α), y n). Then,
1
N N∑ n=1 ` (fWn(x n 0 ), y
n)︸ ︷︷ ︸ running average of training errors ≤ min Vl/α
{ 1
N N∑ n=1 T nl/α(Vl/α) } ︸ ︷︷ ︸
layer-wise/neuronal Taylor optimum
+ O ( 1√ N ) ︸ ︷︷ ︸ Regret(N)/N . (4)
The global optima of neural nets are not computationally accessible. Theorem 2 sidesteps the problem by providing a guarantee relative to the Taylor optimum. The bound is path-dependent; it depends on the convex snapshots encountered during backprop. In particular, the theorem does not preclude convergence to poor local optima. If the optimizer encounters Taylor losses with poor solutions then the bound remains high even as regret tends to zero.
2.3 EMPIRICAL ANALYSIS OF ONLINE NEURAL OPTIMIZATION AND REGRET
This section empirically investigates the Taylor optimum and regret terms in theorem 2 on two tasks:
Autoencoder trained on MNIST. Dense layers with architecture 784 → 50 → 30 → 20 → 30 → 50→ 784 and ReLU non-linearities. Trained with MSE loss using minibatches of 64. Convnet trained on CIFAR-10. Three convolutional layers with stack size 64 and 5× 5 receptive fields, ReLU non-linearities and 2 × 2 max-pooling. Followed by a 192 unit fully-connected layer with ReLU before a ten-dimensional fully-connected output layer. Trained with cross-entropy loss using minibatches of 128.
For both tasks we compare the optimization performance of Adam, RMSProp and SGD (figure 6). Learning rates were tuned for optimal performance. Additional parameters for Adam and RMSProp were left at default. For the convnet all three methods perform equally well: achieving a small loss and an accuracy of ≥ 99% on the training set. However, SGD exhibits slightly more variance. For the autoencoder, although it is an extremely simple model, SGD with the best (fixed) learning rate performs significantly worse than the adaptive optimizers.
The neuronal and layer-wise regret are evaluated for each model. At every iteration we record the training error – the left-hand-side of eq. (4). To evaluate the Taylor loss, we record the input to the neuron/layer, it’s weights, the output of the network and the gradient tensor Gl. After training, we
minimize the Taylor loss with respect to V to find the Taylor optimum at each round. The regret is the difference between the observed training loss and the optimal Taylor loss. The figures show cumulative losses and regret. For illustrative purposes we normalize by 1/ √ N : quantities growing at √ N therefore flatten out. Figure 3(a) compares the average regret incurred by neurons in each convolutional layer of the convnet. Shaded regions show one standard deviation. Dashed lines are the regret of individual neurons – importantly the regret behaviour of neurons holds both on average and individually. Figs 3(b) and 3(c) show the regret, cumulative loss incurred by the network, the average loss incurred and the Taylor optimal loss for neurons in layers 1 and 2 respectively.
Fig. 4 compares Adam, RMSProp and SGD. Figure 4(a) shows the layer-wise regret on the convnet. The regret of all of the optimizers scales as √ N for both models, matching the bound in Theorem 2. The additional variance exhibited by SGD explains the difference in regret magnitude. Similar behaviour was observed in the other layers of the networks and also for convnets trained on MNIST. Figure 4(b) shows the same plot for the autoencoder. The regret of all methods scales as √ N (this also holds for the other layers in the network). The gap in performance can be further explained by examining the difference between the observed loss and Taylor optimal loss. Figure 4(c) compares these quantities for each method on the autoencoder. The adaptive optimizers incur lower losses than SGD. Further, the gap between the actually incurred and optimal loss is smaller for adaptive optimizers. This is possibly because adaptive optimizers find better activation configurations of the network, see discussion in section 3. Remarkably, figures 3 and 4 confirm that regret scales as √ N for a variety of optimizers, datasets, models, neurons and layers – verifying the multi-scale guarantee of Theorem 2. A possible explanation for why optimizers match the worst-case (1/ √ N ) regret is that the adversary (that is, the
landscape) keeps revealing Taylor losses with better solutions. The optimizer struggles to keep up with the hindsight optimum on the constantly changing Taylor losses.
3 OPTIMIZATION AND EXPLORATION IN RECTIFIER NETWORKS
Poor optima in rectifier nets are related to shattered gradients: backprop cannot estimate gradients in nearby smooth regions without directly computing them; the flashlight does not shine across kinks. Two recent papers have shown that noise improves the local optima found during training: Neelakantan et al. (2016) introduce noise into gradients whereas Gulcehre et al. (2016) use noisy activations to extract gradient information from across kinks. Intuitively, noise is a mechanism to “peer around kinks” in shattered landscapes.
Introducing noise is not the only way to find better optima. Not only do adaptive optimizers often converge faster than vanilla gradient descent, they often also converge to better local minima.
This section investigates how adaptive optimizers explore shattered landscapes. Section 3.1 shows that smooth regions in rectifier nets correspond to configurations of active neurons and that neural Taylor approximations clamp the configuration of active neurons – i.e. the convex flashlight shines on smooth regions in the landscape. Section 3.2 observes that adaptive optimizers incorporate an exploration bias and hypothesizes that the success of adaptive optimizers derives from exploring the set of smooth regions more extensively than SGD. Section 3.3 evaluates the hypothesis empirically.
3.1 THE ROLE OF ACTIVATION CONFIGURATIONS IN OPTIMIZATION
We describe how configurations of active neurons relate to smooth regions of rectifier networks and to neural Taylor approximations. Recall that the loss of a neural net on its training data is ˆ̀(W) = 1D ∑D d=1 ` ( fW(x d), yd ) . Definition 2. Enumerate the data as [D] = {1, . . . , D} and neurons as [M ]. The activation configuration A(W) is a (D ×M) binary matrix representing the active neurons for each input. The set of all possible activation configurations corresponds to the set of all (D ×M) binary matrices. Observation 1 (activation configurations correspond to smooth regions in rectifier networks). A parameter value exhibits a kink in ˆ̀ iff an infinitesimal change alters the of activation configuration, i.e. ˆ̀ is not smooth at W iff there is a V s.t. A(W) 6= A(W + δV) for all δ > 0.
The neural Taylor approximation to a rectifier network admits a natural description in terms activation configurations. Observation 2 (the Taylor approximation clamps activation configurations in rectifier networks). Suppose datapoint d is sampled on round n. Let 1k := A(Wn)[d,layer k] be the pk-vector given by entries of row d of A(Wn) corresponding to neurons in layer k of a rectifier network. The Taylor approximation Tnl is
Tnl (Vl) =
( l+1∏ k=L Wnk · diag(1k−1) ) ︸ ︷︷ ︸
clamp subsequent layers weights and activations
· (Vl −Wnl ) ·
( 1∏
k=l−1 diag(1k) ·Wnk ) · xn0︸ ︷︷ ︸
clamp preceding layers weights and activations
which clamps the activation configuration, and weights of all layers excluding l.
Observations 1 and 2 connect shattered gradients in rectifier nets to activation configurations and the Taylor loss. The main implication is to conceptually factorize neural optimization into hard (finding “good” smooth regions) and easy (optimizing within a smooth region) subproblems that correspond, roughly, to finding “good” Taylor losses and optimizing them respectively.
3.2 RMS-NORMALIZATION ENCOURAGES EXPLORATION
Adaptive optimizers based on root-mean-square normalization exhibit an up-to-exponential improvement over non-adaptive methods when gradients are sparse (Duchi et al., 2013) or low-rank (Krummenacher et al., 2016) in convex settings. We propose an alternate explanation for the performance of adaptive optimizers in nonconvex nonsmooth settings.
Let ∇` := 1D ∑D d=1∇ `d denote the average gradient over a dataset. RProp replaces the average gradient with its coordinatewise sign (Riedmiller & Braun, 1993). Hinton and Tieleman observed that normalizing by the root-mean-square recovers sign(∇`) =
∑D d=1∇ `d√∑D d=1(∇ `d)2
, where (∇ `d)2 is the square taken coordinatewise. An alternate characterization of the signed-gradient is
Observation 3 (signed-gradient is a maximizer). Suppose none of the coordinates in∇` are zero. The signed-gradient satisfies
sign(∇`) = argmax x∈Bp∞
{ ‖x‖1 : 〈 x,∇` 〉 ≥ 0 } , where Bp∞ = {x ∈ Rp : max
i=1,...,p |xi| ≤ 1}.
The signed-gradient therefore has two key properties. Firstly, small weight updates using the signedgradient decrease the loss since 〈∇`, sign(∇`)〉 > 0. Secondly, the signed-gradient is the update that, subject to an `∞ constraint, has the largest impact on the most coordinates. Viewing the signedgradient as changing weights – or exploring – maximally suggests the following hypothesis:
Hypothesis 1 (RMS-normalization encourages exploration over activation configurations). Gradient descent with RMS-normalized updates (or running average of RMS) performs a broader search through the space of activation configurations than vanilla gradient descent.
3.3 EMPIRICAL ANALYSIS OF EXPLORATION BY ADAPTIVE OPTIMIZERS
Motivated by hypothesis 1, we investigate how RMSProp and SGD explore the space of activation configurations on the tasks from section 2.3; see implementation details in A.6.
For a fixed parameters W, the activation configuration of a neural net with M neurons and D datapoints is represented as a (D × M) binary matrix, recall definition 2. The set of activation configurations encountered by a network over N rounds of training is represented by an (N,D,M) binary tensor denoted A where An := A[n, :, :] := A(Wn). Figure 5 quantifies exploration in the space of activation configurations in three ways:
5(a): Hamming distance plots mink<n ‖An − Ak‖2F , the minimum Hamming distance between the current activation configuration and all previous configurations. It indicates the novelty of the current activation configuration.
5(b): Activation-state switches plots 1tot ∑N−1 n=1 ∥∥An[d, :] − An−1[d, :]∥∥2F , the total number of times each data point (sorted) switches its activation state across all neurons and epochs as a proportion of possible switches. It indicates the variability of the network response.
5(c): Log-product of singular values. The matrix A[ :, :,m] specifies the rounds and datapoints that activate neuronm. The right column plots the log-product of A[ :, :,m]’s first 50 singular values for each neuron (sorted).3 It indicates the (log-)volume of configuration space covered by each neuron. Note that values reaching the bottom of the plot indicate singular values near 0.
We observe the following.
RMSProp explores the space of activation configurations far more than SGD. The result holds on both tasks, across all three measures, and for multiple learning rates for SGD (including the optimally tuned rate). The finding provides evidence for hypothesis 1.
RMSProp converges to a significantly better local optimum on the autoencoder, Fig. 6. We observe no difference on CIFAR-10. We hypothesize that RMSProp finds a better optimum through more extensive exploration through the space of activation configurations. CIFAR is an easier problem and possibly requires less exploration.
Adam performs targeted exploration. Adam achieves the best performance on the autoencoder. Surprisingly, it explores substantially less than RMSProp according to the Hamming distance and activation-switches, although still more than SGD. The singular values provide a higher-resolution analysis: the ±40 most exploratory neurons match the behavior of RMSProp, with a sharp dropoff
3The time-average is subtracted from each column of A[ :, :,m]. If the response of neuron m to datapoint d is constant over all rounds, then column A[ :, d,m] maps to (0, . . . , 0) and does not contribute to the volume.
from neuron 60 onwards. A possible explanation is that momentum encourages targeted exploration by rapidly discarding avenues that are not promising.
4 DISCUSSION
Rectifier convnets are the dominant technology in computer vision and a host of related applications. Our main contribution is the first convergence result applicable to convnets as they are used in practice, including rectifier nets, max-pooling, dropout and related methods. The key analytical tool is the neural Taylor approximation, the first-order approximation to the output of a neural net. The Taylor loss – the loss on the neural Taylor approximation – is a convex approximation to the loss of the network. Remarkably, the convergence rate matches known lower bounds on convex nonsmooth functions (Bubeck, 2015). Experiments in section 2.3 show the regret matches the theoretical analysis under a wide range of settings.
The bound in theorem 2 contains an easy term to optimize (the regret) and a hard term (finding “good” Taylor losses). Section 3.1 observes that the Taylor losses speak directly to the fundamental difficulty of optimizing nonsmooth functions: that gradients are shattered – the gradient at a point is not a reliable estimate of nearby gradients.
Smooth regions of rectifier nets correspond to activation configurations. Gradients in one smooth region cannot be used to estimate gradients in another. Exploring the set of activation configurations may therefore be crucial for optimizers to find better local minima in shattered landscapes. Empirical results in section 3.3 suggest that the improved performance of RMSProp over SGD can be explained in part by a carefully tuned exploration bias.
Finally, the paper raises several questions:
1. To what extent is exploration necessary for good performance?
2. Can exploration/exploitation tradeoffs in nonsmooth neural nets be quantified?
3. There are exponentially more kinks in early layers (near the input) compared to later layers. Should optimizers explore more aggressively in early layers?
4. Can exploring activation configurations help design better optimizers?
The Taylor decomposition provides a useful tool for separating the convex and nonconvex aspects of neural optimization, and may also prove useful when tackling exploration in neural nets.
Acknowledgements. We thank L. Helminger and T. Vogels for useful discussions and help with TensorFlow. Some experiments were performed using a Tesla K80 kindly donated by NVidia.
A APPENDIX
A.1 BACKGROUND ON CONVEX OPTIMIZATION
A continuous function f is smooth if there exists a β > 0 such that ‖∇ f(x) − ∇ f(y)‖2 ≤ β · ‖x− y‖2 for all x and y in the domain. Rectifiers are not smooth for any value of β. Nonsmooth convex functions. Let X ⊂ Rp be a convex set contained in a ball of radius R. Let ` : X → R be a convex function. Section 3.1 of (Bubeck, 2015) shows that projected gradient descent has convergence guarantee
`
( 1
N N∑ n=1 wn
) − `(w∗) ≤ O ( 1√ N ) where wn are generated by gradient descent and w∗ := argminw∈X `(w) is the minimizer of `. It is also shown, section 3.5, that
min 1≤n≤N
`(wn)− `(w∗) ≥ Ω (
1√ N ) where the weights are in the span of the previously observed gradients: wn ∈ span{∇ `(w1), . . . ,∇ `(wn−1)} for all n ∈ {1, . . . , N}. The gradient of a convex function increases monotonically. That is〈
∇ `(w)−∇ `(v),w − v 〉 ≥ 0
for all points w,v where the gradient exists. Gradients at one point of a nonsmooth convex function therefore do contain information about other points, although not as much information as in the smooth case. In contrast, the gradients of nonsmooth nonconvex functions can vary wildly as shown in Fig. 1.
Smooth convex functions. In the smooth setting, gradient descent converges at rate 1N . The lower bound for convergence is even better, 1N2 . The lower bound is achieved by Nesterov’s accelerated gradient descent method.
A.2 PROOF OF THEOREM 2
Proof. We prove the network case; the others are similar. The Taylor loss has three key properties by construction:
T1. The Taylor loss T n coincides with the loss at Wn: `(fWn(x n 0 ), y n) = T n(V)|V=Wn
T2. The Taylor loss gradient T n coincides with the loss gradient at Wn: ∇W `(fWn(xn0 ), yn) = ∇V T n(V)|V=Wn
T3. The Taylor losses are convex functions of V because `(f , y) is convex in its first argument and convexity is invariant under affine maps. If ` is a convex function, then so is g(x) = `(Ax+ b), where A ∈ Rm×n and b ∈ Rm.
By T1, the training loss, i.e. the left-hand side of (3), exactly coincides with the Taylor losses. By T2, the gradients of the Taylor losses exactly coincide with the errors computed by backpropagation on the training losses. That is, the training loss over n rounds is indistinguishable from the Taylor losses to the first order:
losses: 1
N N∑ n=1 ` (fWn(x n 0 ), y n) = 1 N N∑ n=1 T n (Wn)
gradients: ∇W
( 1
N N∑ n=1 ` (fWn(x n 0 ), y n)
) = ∇W ( 1
N N∑ n=1 T n (Wn) ) We can therefore substitute the Taylor losses in place of the training loss (fW(x0), y) without altering either the losses incurred during training or the dynamics of backpropagation (or any first-order method).
Since the Taylor losses are convex, the bound holds for any no-regret optimizer following (Zinkevich, 2003).
A.3 PROOF OF OBSERVATIONS IN SECTION 3
Proof of observation 1.
Proof. The loss `(f , y) is a smooth function of the network’s output f by assumption. Kinks in ˆ̀(W) = 1D ∑D d=1 ` ( fW(x d), yd )
can therefore only arise when a rectifier changes its activation for at least one element of the training data.
Proof of observation 2. Note that the rectifier is ρ(a) = max(0, a) with derivative ρ′(a) = 1 if a > 0 and ρ′(a) = 0 if a < 0.
Proof. Recall that the Taylor approximation to layer l is
Tnl (Vl) := fWn(x n 0 ) + 〈 Gl,Vl −Wnl 〉 = fWn(x n 0 ) + J L l · (Vl −Wnl ) · xnl−1
= fWn(x n 0 ) + ( l+1∏ k=L Jkk−1 ) · (Vl −Wnl ) · xnl−1
The Jacobian of layer k is the function Jk+1k (ak) = Wk+1 · diag ( s′(ak) ) which in general varies nonlinearly with ak. The Taylor approximation clamps the Jacobian by setting it as constant.
For a layer of rectifiers, s(·) = ρ(·), the Jacobian Jk+1k = Wk+1 · diag ( ρ′(ak) ) is constructed by zeroing out the rows of Wk+1 corresponding to inactive neurons in layer k. It follows that the
Taylor loss can be written as
Tnl (Vl) = ( l+1∏ k=L Wnk · diag(1k−1) ) · (Vl −Wnl ) · xnl−1
Finally, observe that
xnl−1 =
( 1∏
k=l−1 diag(1k) ·Wnk
) · xn0
since diag(1k) ·Wnk · xnk−1 = diag ( ρ′(ak) ) ·Wnkxnk−1 = ρ(Wnk · xnk−1).
Proof of observation 3.
Proof. Immediate.
A.4 COMPARISON OF TAYLOR LOSS WITH TAYLOR APPROXIMATION TO LOSS
It is instructive to compare the Taylor loss in definition 1 with the Taylor approximation to the loss. The Taylor loss is
` ( Tn(V), yn ) = ` ( fWn(x n 0 ) + L∑ l=1 〈 Gl,Vl −Wnl 〉 , yn ) In contrast, the Taylor approximation to the loss is
Tn` (V) = ` ( fWn(x n 0 ), y n )︸ ︷︷ ︸
loss incurred on round n
+ L∑ l=1 〈 JEL ·Gl︸ ︷︷ ︸
δl
,Vl −Wnl 〉 = `n + L∑ l=1 〈 ∇Wl `,Vl −Wnl 〉 .
The constant term is the loss incurred on round n; the linear coefficients are the backpropagated errors.
It is easy to see that the two expressions have the same gradient. Why not work directly with the Taylor approximation to the loss? The problem is that the Taylor approximation to the loss is affine, and so decreases without bound. Upgrading to a second order Taylor approximation is no help since it is not convex.
A.5 DETAILS ON EXPERIMENTS ON REGRET
See section 2.3 for the architecture of the autoencoder and convnet used. The hyperparameters used for different optimizers are as follows: the autoencoder uses learning rate η = 0.001 for RMSprop and η = 0.01 for Adam, while the convnet uses learning rate η = 0.0005 for RMSprop and η = 0.0002 for Adam. All other hyperparameters are kept at their literature-standard values.
Fig. 6 shows the training losses obtained by the convnet on CIFAR-10 and the autoencoder on MNIST.
The gradient tensor Gl is not computed explicitly by TensorFlow. Instead, it is necessary to compute the gradient of each component of the output layer (e.g. 10 in total for a network trained on CIFAR10, 784 for an autoencoder trained on MNIST) with respect to Wl and then assemble the gradients into a tensor. When the loss is the squared error, the Taylor optimal at round n can be computed in closed form. Otherwise we use SGD.
A.6 DETAILS ON EXPERIMENTS ON EXPLORATION
Given matrix or vector A or a, the squared Frobenius norm is
‖A‖2F = M,N∑ m,n=1 A2m,n and ‖a‖2F = N∑ n=1 a2n.
The Hamming distance between two binary vectors a and b can be computed as ‖a− b‖2F . For tractability in the convnet, we only record activations for 1% of the CIFAR dataset, and at most 10000 units of each convolutional layer. We record the full network state on all inputs for the autoencoder. The singular value plots in figure 5 are calculated only on the first 50 epochs.

This paper develops a theoretical guarantee for the convergence of the training error. The result is quite general that covers the training of a wide range of neural network models. The key idea of the paper is approximate the training loss by its linear approximation. Since its linearity in the variables (thus convex), the authors plug in results that has been developed in the literature of online learning.
This paper has good novelty in using the Taylor approximation thus greatly simplifying the analysis of the behaviour of the model. However, there are two problems about the main result of this paper, Theorem 2.
1. It is not clear if the Taylor optimum would converge or not.
As noticed by the authors, the upper bound is path dependent. Appendix 3 tries to claim that this Taylor optimum indeed converges, but the proof is buggy. In the proof of Lemma 2, it is proved that the difference between two sequential Taylor optimum is approaching 0. Note that this is actually weaker than being Cauchy sequence and insufficient to guarantee convergence.
2. The lefthand side of Equation (3) (I will denote it by L3 in this review) is not equivalent to training error.
An upper bound on this average error is not sufficient to guarantee the convergence of the training error neither. Take the gradient descent for example (thus each minibatch x_0^n is the whole training set), the convergence of the training error should be lim_{n -> \infty} l(f_{w^n}(x_0^n), y^n). The convergence of L3 is necessary but not sufficient to imply the convergence of the training error.
Another concern about Theorem 2 (but it is minor compared to the two problems mentioned above) is that to achieve the O(1/\sqrt{n}) rate, the algorithm has to pick a particular learning rate. Larger or smaller learning rate (in the order of n) will lead to significantly worse regret. But in the experiments of the paper, the learning rates are not picked according to the theorem.
Overall, this paper has a good motivation and good novelty. It could be further developed into a good paper. But due to the two problems and a buggy proof mentioned above, I think it is not ready for publish yet.