FINE-GRAINED ANALYSIS OF SENTENCE EMBEDDINGS USING AUXILIARY PREDICTION TASKS
There is a lot of research interest in encoding variable length sentences into fixed length vectors, in a way that preserves the sentence meanings. Two common methods include representations based on averaging word vectors, and representations based on the hidden states of recurrent neural networks such as LSTMs. The sentence vectors are used as features for subsequent machine learning tasks or for pre-training in the context of deep learning. However, not much is known about the properties that are encoded in these sentence representations and about the language information they capture. We propose a framework that facilitates better understanding of the encoded representations. We define prediction tasks around isolated aspects of sentence structure (namely sentence length, word content, and word order), and score representations by the ability to train a classifier to solve each prediction task when using the representation as input. We demonstrate the potential contribution of the approach by analyzing different sentence representation mechanisms. The analysis sheds light on the relative strengths of different sentence embedding methods with respect to these low level prediction tasks, and on the effect of the encoded vector’s dimensionality on the resulting representations.
1 INTRODUCTION
While sentence embeddings or sentence representations play a central role in recent deep learning approaches to NLP, little is known about the information that is captured by different sentence embedding learning mechanisms. We propose a methodology facilitating fine-grained measurement of some of the information encoded in sentence embeddings, as well as performing fine-grained comparison of different sentence embedding methods.
In sentence embeddings, sentences, which are variable-length sequences of discrete symbols, are encoded into fixed length continuous vectors that are then used for further prediction tasks. A simple and common approach is producing word-level vectors using, e.g., word2vec (Mikolov et al., 2013a;b), and summing or averaging the vectors of the words participating in the sentence. This continuous-bag-of-words (CBOW) approach disregards the word order in the sentence.1
Another approach is the encoder-decoder architecture, producing models also known as sequenceto-sequence models (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014, inter alia). In this architecture, an encoder network (e.g. an LSTM) is used to produce a vector representation of the sentence, which is then fed as input into a decoder network that uses it to perform some prediction task (e.g. recreate the sentence, or produce a translation of it). The encoder and decoder networks are trained jointly in order to perform the final task.
1We use the term CBOW to refer to a sentence representation that is composed of an average of the vectors of the words in the sentence, not to be confused with the training method by the same name which is used in the word2vec algorithm.
Some systems (for example in machine translation) train the system end-to-end, and use the trained system for prediction (Bahdanau et al., 2014). Such systems do not generally care about the encoded vectors, which are used merely as intermediate values. However, another common case is to train an encoder-decoder network and then throw away the decoder and use the trained encoder as a general mechanism for obtaining sentence representations. For example, an encoder-decoder network can be trained as an auto-encoder, where the encoder creates a vector representation, and the decoder attempts to recreate the original sentence (Li et al., 2015). Similarly, Kiros et al. (2015) train a network to encode a sentence such that the decoder can recreate its neighboring sentences in the text. Such networks do not require specially labeled data, and can be trained on large amounts of unannotated text. As the decoder needs information about the sentence in order to perform well, it is clear that the encoded vectors capture a non-trivial amount of information about the sentence, making the encoder appealing to use as a general purpose, stand-alone sentence encoding mechanism. The sentence encodings can then be used as input for other prediction tasks for which less training data is available (Dai & Le, 2015). In this work we focus on these “general purpose” sentence encodings.
The resulting sentence representations are opaque, and there is currently no good way of comparing different representations short of using them as input for different high-level semantic tasks (e.g. sentiment classification, entailment recognition, document retrieval, question answering, sentence similarity, etc.) and measuring how well they perform on these tasks. This is the approach taken by Li et al. (2015), Hill et al. (2016) and Kiros et al. (2015). This method of comparing sentence embeddings leaves a lot to be desired: the comparison is at a very coarse-grained level, does not tell us much about the kind of information that is encoded in the representation, and does not help us form generalizable conclusions.
Our Contribution We take a first step towards opening the black box of vector embeddings for sentences. We propose a methodology that facilitates comparing sentence embeddings on a much finer-grained level, and demonstrate its use by analyzing and comparing different sentence representations. We analyze sentence representation methods that are based on LSTM auto-encoders and the simple CBOW representation produced by averaging word2vec word embeddings. For each of CBOW and LSTM auto-encoder, we compare different numbers of dimensions, exploring the effect of the dimensionality on the resulting representation. We also provide some comparison to the skip-thought embeddings of Kiros et al. (2015).
In this work, we focus on what are arguably the three most basic characteristics of a sequence: its length, the items within it, and their order. We investigate different sentence representations based on the capacity to which they encode these aspects. Our analysis of these low-level properties leads to interesting, actionable insights, exposing relative strengths and weaknesses of the different representations.
Limitations Focusing on low-level sentence properties also has limitations: The tasks focus on measuring the preservation of surface aspects of the sentence and do not measure syntactic and semantic generalization abilities; the tasks are not directly related to any specific downstream application (although the properties we test are important factors in many tasks – knowing that a model is good at predicting length and word order is likely advantageous for syntactic parsing, while models that excel at word content are good for text classification tasks). Dealing with these limitations requires a complementary set of auxiliary tasks, which is outside the scope of this study and is left for future work.
The study also suffers from the general limitations of empirical work: we do not prove general theorems but rather measure behaviors on several data points and attempt to draw conclusions from these measurements. There is always the risk that our conclusions only hold for the datasets on which we measured, and will not generalize. However, we do consider our large sample of sentences from Wikipedia to be representative of the English language, at least in terms of the three basic sentence properties that we study.
Summary of Findings Our analysis reveals the following insights regarding the different sentence embedding methods:
• Sentence representations based on averaged word vectors are surprisingly effective, and encode a non-trivial amount of information regarding sentence length. The information they contain
can also be used to reconstruct a non-trivial amount of the original word order in a probabilistic manner (due to regularities in the natural language data).
• LSTM auto-encoders are very effective at encoding word order and word content. • Increasing the number of dimensions benefits some tasks more than others. • Adding more hidden units sometimes degrades the encoders’ ability to encode word content. This
degradation is not correlated with the BLEU scores of the decoder, suggesting that BLEU over the decoder output is sub-optimal for evaluating the encoders’ quality.
• LSTM encoders trained as auto-encoders do not rely on ordering patterns in the training sentences when encoding novel sentences, while the skip-thought encoders do rely on such patterns.
2 RELATED WORK
Word-level distributed representations have been analyzed rather extensively, both empirically and theoretically, for example by Baroni et al. (2014), Levy & Goldberg (2014) and Levy et al. (2015). In contrast, the analysis of sentence-level representations has been much more limited. Commonly used approaches is to either compare the performance of the sentence embeddings on down-stream tasks (Hill et al., 2016), or to analyze models, specifically trained for predefined task (Schmaltz et al., 2016; Sutskever et al., 2011).
While the resulting analysis reveals differences in performance of different models, it does not adequately explain what kind of linguistic properties of the sentence they capture. Other studies analyze the hidden units learned by neural networks when training a sentence representation model (Elman, 1991; Karpathy et al., 2015; Kádár et al., 2016). This approach often associates certain linguistic aspects with certain hidden units. Kádár et al. (2016) propose a methodology for quantifying the contribution of each input word to a resulting GRU-based encoding. These methods depend on the specific learning model and cannot be applied to arbitrary representations. Moreover, it is still not clear what is captured by the final sentence embeddings.
Our work is orthogonal and complementary to the previous efforts: we analyze the resulting sentence embeddings by devising auxiliary prediction tasks for core sentence properties. The methodology we purpose is general and can be applied to any sentence representation model.
3 APPROACH
We aim to inspect and compare encoded sentence vectors in a task-independent manner. The main idea of our method is to focus on isolated aspects of sentence structure, and design experiments to measure to what extent each aspect is captured in a given representation.
In each experiment, we formulate a prediction task. Given a sentence representation method, we create training data and train a classifier to predict a specific sentence property (e.g. their length) based on their vector representations. We then measure how well we can train a model to perform the task. The basic premise is that if we cannot train a classifier to predict some property of a sentence based on its vector representation, then this property is not encoded in the representation (or rather, not encoded in a useful way, considering how the representation is likely to be used).
The experiments in this work focus on low-level properties of sentences – the sentence length, the identities of words in a sentence, and the order of the words. We consider these to be the core elements of sentence structure. Generalizing the approach to higher-level semantic and syntactic properties holds great potential, which we hope will be explored in future work, by us or by others.
3.1 THE PREDICTION TASKS
We now turn to describe the specific prediction tasks. We use lower case italics (s, w) to refer to sentences and words, and boldface to refer to their corresponding vector representations (s, w). When more than one element is considered, they are distinguished by indices (w1, w2, w1, w2).
Our underlying corpus for generating the classification instances consists of 200,000 Wikipedia sentences, where 150,000 sentences are used to generate training examples, and 25,000 sentences
are used for each of the test and development examples. These sentences are a subset of the training set that was used to train the original sentence encoders. The idea behind this setup is to test the models on what are presumably their best embeddings. Length Task This task measures to what extent the sentence representation encodes its length. Given a sentence representation s ∈ Rk, the goal of the classifier is to predict the length (number of words) in the original sentence s. The task is formulated as multiclass classification, with eight output classes corresponding to binned lengths.2 The resulting dataset is reasonably balanced, with a majority class (lengths 5-8 words) of 5,182 test instances and a minority class (34-70) of 1,084 test instances. Predicting the majority class results in classification accuracy of 20.1%. Word-content Task This task measures to what extent the sentence representation encodes the identities of words within it. Given a sentence representation s ∈ Rk and a word representation w ∈ Rd, the goal of the classifier is to determine whether w appears in the s, with access to neither w nor s. This is formulated as a binary classification task, where the input is the concatenation of s and w.
To create a dataset for this task, we need to provide positive and negative examples. Obtaining positive examples is straightforward: we simply pick a random word from each sentence. For negative examples, we could pick a random word from the entire corpus. However, we found that such a dataset tends to push models to memorize words as either positive or negative words, instead of finding their relation to the sentence representation. Therefore, for each sentence we pick as a negative example a word that appears as a positive example somewhere in our dataset, but does not appear in the given sentence. This forces the models to learn a relationship between word and sentence representations. We generate one positive and one negative example from each sentence. The dataset is balanced, with a baseline accuracy of 50%. Word-order Task This task measures to what extent the sentence representation encodes word order. Given a sentence representation s ∈ Rk and the representations of two words that appear in the sentence, w1,w2 ∈ Rd, the goal of the classifier is to predict whether w1 appears before or after w2 in the original sentence s. Again, the model has no access to the original sentence and the two words. This is formulated as a binary classification task, where the input is a concatenation of the three vectors s, w1 and w2.
For each sentence in the corpus, we simply pick two random words from the sentence as a positive example. For negative examples, we flip the order of the words. We generate one positive and one negative example from each sentence. The dataset is balanced, with a baseline accuracy of 50%.
4 SENTENCE REPRESENTATION MODELS
Given a sentence s = {w1, w2, ..., wN} we aim to find a sentence representation s using an encoder: ENC : s = {w1, w2, ..., wN} 7→ s ∈ Rk
The encoding process usually assumes a vector representation wi ∈ Rd for each word in the vocabulary. In general, the word and sentence embedding dimensions, d and k, need not be the same. The word vectors can be learned together with other encoder parameters or pre-trained. Below we describe different instantiations of ENC.
Continuous Bag-of-words (CBOW) This simple yet effective text representation consists of performing element-wise averaging of word vectors that are obtained using a word-embedding method such as word2vec.
Despite its obliviousness to word order, CBOW has proven useful in different tasks (Hill et al., 2016) and is easy to compute, making it an important model class to consider.
Encoder-Decoder (ED) The encoder-decoder framework has been successfully used in a number of sequence-to-sequence learning tasks (Sutskever et al., 2014; Bahdanau et al., 2014; Dai & Le, 2015; Li et al., 2015). After the encoding phase, a decoder maps the sentence representation back to the sequence of words:
DEC : s ∈ Rk 7→ s = {w1, w2, ..., wN} 2We use the bins (5-8), (9-12), (13-16), (17-20), (21-25), (26-29), (30-33), (34-70).
100 300 500 750 1000
Representation dimensions
10
20
30
40
50
60
70
80
90
L e n
g th
p re
d ic
ti o n
a c c u
ra c y
0
5
10
15
20
25
30
35
B L E U
ED CBOW ED BLEU
(a) Length test.
100 300 500 750 1000
Representation dimensions
50
55
60
65
70
75
80
85
90
C o n
te n
t p
re d
ic ti
o n
a c c u
ra c y
0
5
10
15
20
25
30
35
B L E U
ED CBOW ED BLEU
(b) Content test.
100 300 500 750 1000
Representation dimensions
50
60
70
80
90
O rd
e r
p re
d ic
ti o n
a c c u
ra c y
0
5
10
15
20
25
30
35
B L E U
ED CBOW ED BLEU
(c) Order test.
Figure 1: Task accuracy vs. embedding size for different models; ED BLEU scores given for reference.
Here we investigate the specific case of an auto-encoder, where the entire encoding-decoding process can be trained end-to-end from a corpus of raw texts. The sentence representation is the final output vector of the encoder. We use a long short-term memory (LSTM) recurrent neural network (Hochreiter & Schmidhuber, 1997; Graves et al., 2013) for both encoder and decoder. The LSTM decoder is similar to the LSTM encoder but with different weights.
5 EXPERIMENTAL SETUP
The bag-of-words (CBOW) and encoder-decoder models are trained on 1 million sentences from a 2012 Wikipedia dump with vocabulary size of 50,000 tokens. We use NLTK (Bird, 2006) for tokenization, and constrain sentence lengths to be between 5 and 70 words. For both models we control the embedding size k and train word and sentence vectors of sizes k ∈ {100, 300, 500, 750, 1000}. More details about the experimental setup are available in the Appendix.
6 RESULTS
In this section we provide a detailed description of our experimental results along with their analysis. For each of the three main tests – length, content and order – we investigate the performance of different sentence representation models across embedding size.
6.1 LENGTH EXPERIMENTS
We begin by investigating how well the different representations encode sentence length. Figure 1a shows the performance of the different models on the length task, as well as the BLEU obtained by the LSTM encoder-decoder (ED).
With enough dimensions, the LSTM embeddings are very good at capturing sentence length, obtaining accuracies between 82% and 87%. Length prediction ability is not perfectly correlated with BLEU scores: from 300 dimensions onward the length prediction accuracies of the LSTM remain relatively stable, while the BLEU score of the encoder-decoder model increases as more dimensions are added.
Somewhat surprisingly, the CBOW model also encodes a fair amount of length information, with length prediction accuracies of 45% to 65%, way above the 20% baseline. This is remarkable, as the CBOW representation consists of averaged word vectors, and we did not expect it to encode length at all. We return to CBOW’s exceptional performance in Section 7.
6.2 WORD CONTENT EXPERIMENTS
To what extent do the different sentence representations encode the identities of the words in the sentence? Figure 1b visualizes the performance of our models on the word content test.
All the representations encode some amount of word information, and clearly outperform the random baseline of 50%. Some trends are worth noting. While the capacity of the LSTM encoder to preserve word identities generally increases when adding dimensions, the performance peaks at 750 dimensions and drops afterwards. This stands in contrast to the BLEU score of the respective
encoder-decoder models. We hypothesize that this occurs because a sizable part of the auto-encoder performance comes from the decoder, which also improves as we add more dimensions. At 1000 dimensions, the decoder’s language model may be strong enough to allow the representation produced by the encoder to be less informative with regard to word content.
CBOW representations with low dimensional vectors (100 and 300 dimensions) perform exceptionally well, outperforming the more complex, sequence-aware models by a wide margin. If your task requires access to word identities, it is worth considering this simple representation. Interestingly, CBOW scores drop at higher dimensions.
6.3 WORD ORDER EXPERIMENTS
Figure 1c shows the performance of the different models on the order test. The LSTM encoders are very capable of encoding word order, with LSTM-1000 allowing the recovery of word order in 91% of the cases. Similar to the length test, LSTM order prediction accuracy is only loosely correlated with BLEU scores. It is worth noting that increasing the representation size helps the LSTM-encoder to better encode order information.
Surprisingly, the CBOW encodings manage to reach an accuracy of 70% on the word order task, 20% above the baseline. This is remarkable as, by definition, the CBOW encoder does not attempt to preserve word order information. One way to explain this is by considering distribution patterns of words in natural language sentences: some words tend to appear before others. In the next section we analyze the effect of natural language on the different models.
7 IMPORTANCE OF “NATURAL LANGUAGENESS”
Natural language imposes many constraints on sentence structure. To what extent do the different encoders rely on specific properties of word distributions in natural language sentences when encoding sentences?
To account for this, we perform additional experiments in which we attempt to control for the effect of natural language.
How can CBOW encode sentence length? Is the ability of CBOW embeddings to encode length related to specific words being indicative of longer or shorter sentences? To control for this, we created a synthetic dataset where each word in each sentence is replaced by a random word from the dictionary and re-ran the length test for the CBOW embeddings using this dataset. As Figure 2a shows, this only leads to a slight decrease in accuracy, indicating that the identity of the words is not the main component in CBOW’s success at predicting length.
100 300 500 750 1000
Representation dimensions
35
40
45
50
55
60
65
L e n
g th
p re
d ic
ti o n
a c c u
ra c y
CBOW CBOW syn sent
(a) Length accuracy for different CBOW sizes on natural and synthetic (random words) sentences.
5 10 15 20 25 30 35
Sentence length
0.35
0.40
0.45
0.50
0.55
N o rm
(b) Average embedding norm vs. sentence length for CBOW with an embedding size of 300.
An alternative explanation for CBOW’s ability to encode sentence length is given by considering the norms of the sentence embeddings. Indeed, Figure 2b shows that the embedding norm decreases as sentences grow longer. We believe this is one of the main reasons for the strong CBOW results.
While the correlation between the number of averaged vectors and the resulting norm surprised us, in retrospect it is an expected behavior that has sound mathematical foundations. To understand the behavior, consider the different word vectors to be random variables, with the values in each
dimension centered roughly around zero. Both central limit theorem and Hoeffding‘s inequality tell us that as we add more samples, the expected average of the values will better approximate the true mean, causing the norm of the average vector to decrease. We expect the correlation between the sentence length and its norm to be more pronounced with shorter sentences (above some number of samples we will already be very close to the true mean, and the norm will not decrease further), a behavior which we indeed observe in practice.
How does CBOW encode word order? The surprisingly strong performance of the CBOW model on the order task made us hypothesize that much of the word order information is captured in general natural language word order statistics.
To investigate this, we re-run the word order tests, but this time drop the sentence embedding in training and testing time, learning from the word-pairs alone. In other words, we feed the network as input two word embeddings and ask which word comes first in the sentence. This test isolates general word order statistics of language from information that is contained in the sentence embedding (Fig. 3).
The difference between including and removing the sentence embeddings when using the CBOW model is minor, while the LSTM-ED suffers a significant drop. Clearly, the LSTMED model encodes word order, while the prediction ability of CBOW is mostly explained by general language statistics. However, CBOW does benefit from the sentence to some extent: we observe a gain of ∼3% accuracy points when the CBOW tests are allowed access to the sentence representation. This may be explained by higher order statistics of correlation between word order patterns and the occurrences of specific words.
How important is English word order for encoding sentences? To what extent are the models trained to rely on natural language word order when encoding sentences? To control for this, we create a synthetic dataset, PERMUTED, in which the word order in each sentence is randomly permuted. Then, we repeat the length, content and order experiments using the PERMUTED dataset (we still use the original sentence encoders that are trained on non-permuted sentences). While the permuted sentence representation is the same for CBOW, it is completely different when generated by the encoder-decoder.
Results are presented in Fig. 4. When considering CBOW embeddings, word order accuracy drops to chance level, as expected, while results on the other tests remain the same. Moving to the LSTM encoder-decoder, the results on all three tests are comparable to the ones using non-permuted sentences. These results are somewhat surprising since the models were originally trained on “real”, non-permuted sentences. This indicates that the LSTM encoder-decoder is a general-purpose sequence encoder that for the most part does not rely on word ordering properties of natural language when encoding sentences. The small and consistent drop in word order accuracy on the permuted sentences can be attributed to the encoder relying on natural language word order to some extent, but can also be explained by the word order prediction task becoming harder due to the inability to
use general word order statistics. The results suggest that a trained encoder will transfer well across different natural language domains, as long as the vocabularies remain stable. When considering the decoder’s BLEU score on the permuted dataset (not shown), we do see a dramatic decrease in accuracy. For example, LSTM encoder-decoder with 1000 dimensions drops from 32.5 to 8.2 BLEU score. These results suggest that the decoder, which is thrown away, contains most of the language-specific information.
8 SKIP-THOUGHT VECTORS
In addition to the experiments on CBOW and LSTM-encoders, we also experiment with the skipthought vectors model (Kiros et al., 2015). This model extends the idea of the auto-encoder to neighboring sentences.
Given a sentence si, it first encodes it using an RNN, similar to the auto-encoder model. However, instead of predicting the original sentence, skip-thought predicts the preceding and following sentences, si−1 and si+1. The encoder and decoder are implemented with gated recurrent units (Cho et al., 2014).
Here, we deviate from the controlled environment and use the author’s provided model3 with the recommended embeddings size of 4800. This makes the direct comparison of the models “unfair”. However, our aim is not to decide which is the “best” model but rather to show how our method can be used to measure the kinds of information captured by different representations.
Table 1 summarizes the performance of the skip-thought embeddings in each of the prediction tasks on both the PERMUTED and original dataset.
The performance of the skip-thought embeddings is well above the baselines and roughly similar for all tasks. Its performance is similar to the higher-dimensional encoder-decoder models, except in the order task where it lags somewhat behind. However, we note that the results are not directly comparable as skip-thought was trained on a different corpus.
The more interesting finding is its performance on the PERMUTED sentences. In this setting we see a large drop. In contrast to the LSTM encoder-decoder, skip-thought’s ability to predict length and word content does degrade significantly on the permuted sentences, suggesting that the encoding process of the skip-thought model is indeed specialized towards natural language texts.
9 CONCLUSION
We presented a methodology for performing fine-grained analysis of sentence embeddings using auxiliary prediction tasks. Our analysis reveals some properties of sentence embedding methods:
• CBOW is surprisingly effective – in addition to being very strong at content, it is also predictive of length, and can be used to reconstruct a non-trivial amount of the original word order. 300 dimensions perform best, with greatly degraded word-content prediction performance on higher dimensions.
• With enough dimensions, LSTM auto-encoders are very effective at encoding word order and word content information. Increasing the dimensionality of the LSTM encoder does not significantly improve its ability to encode length, but does increase its ability to encode content and order information. 500 dimensional embeddings are already quite effective for encoding word order, with little gains beyond that. Word content accuracy peaks at 750 dimensions and drops at 1000, suggesting that larger is not always better.
3https://github.com/ryankiros/skip-thoughts
• The trained LSTM encoder (when trained with an auto-encoder objective) does not rely on ordering patterns in the training sentences when encoding novel sequences. In contrast, the skip-thought encoder does rely on such patterns. Its performance on the other tasks is similar to the higher-dimensional LSTM encoder, which is impressive considering it was trained on a different corpus.
• Finally, the encoder-decoder’s ability to recreate sentences (BLEU) is not entirely indicative of the quality of the encoder at representing aspects such as word identity and order. This suggests that BLEU is sub-optimal for model selection.
10 ADDITIONAL EXPERIMENTS - CONTENT TASK
How well do the models preserve content when we increase the sentence length? In Fig. 5 we plot content prediction accuracy vs. sentence length for different models.
As expected, all models suffer a drop in content accuracy on longer sentences. The degradation is roughly linear in the sentence length. For the encoder-decoder, models with fewer dimensions seem to degrade slower.
APPENDIX III: SIGNIFICANCE TESTS
In this section we report the significance tests we conduct in order to evaluate our findings. In order to do so, we use the paired t-test (Rubin, 1973).
All the results reported in the summery of findings are highly significant (p-value 0.0001). The ones we found to be not significant (p-value 0.03) are the ones which their accuracy does not have much of a difference, i.e ED with size 500 and ED with size 750 tested on the word order task (p-value=0.11), or CBOW with dimensions 750 and 1000 (p-value=0.3).

The authors present a methodology for analyzing sentence embedding techniques by checking how much the embeddings preserve information about sentence length, word content, and word order. They examine several popular embedding methods including autoencoding LSTMs, averaged word vectors, and skip-thought vectors. The experiments are thorough and provide interesting insights into the representational power of common sentence embedding strategies, such as the fact that word ordering is surprisingly low-entropy conditioned on word content.
Exploring what sort of information is encoded in representation learning methods for NLP is an important and under-researched area. For example, the tide of word-embeddings research was mostly stemmed after a thread of careful experimental results showing most embeddings to be essentially equivalent, culminating in "Improving Distributional Similarity with Lessons Learned from Word Embeddings" by Levy, Goldberg, and Dagan. As representation learning becomes even more important in NLP this sort of research will be even more important.
While this paper makes a valuable contribution in setting out and exploring a methodology for evaluating sentence embeddings, the evaluations themselves are quite simple and do not necessarily correlate with real-world desiderata for sentence embeddings (as the authors note in other comments, performance on these tasks is not a normative measure of embedding quality). For example, as the authors note, the ability of the averaged vector to encode sentence length is trivially to be expected given the central limit theorem (or more accurately, concentration inequalities like Hoeffding's inequality).
The word-order experiments were interesting. A relevant citation for this sort of conditional ordering procedure is "Generating Text with Recurrent Neural Networks" by Sutskever, Martens, and Hinton, who refer to the conversion of a bag of words into a sentence as "debagging."
Although this is just a first step in better understanding of sentence embeddings, it is an important one and I recommend this paper for publication.


NONPARAMETRICALLY LEARNING ACTIVATION FUNCTIONS IN DEEP NEURAL NETS
We provide a principled framework for nonparametrically learning activation functions in deep neural networks. Currently, state-of-the-art deep networks treat choice of activation function as a hyper-parameter before training. By allowing activation functions to be estimated as part of the training procedure, we expand the class of functions that each node in the network can learn. We also provide a theoretical justification for our choice of nonparametric activation functions and demonstrate that networks with our nonparametric activation functions generalize well. To demonstrate the power of our novel techniques, we test them on image recognition datasets and achieve up to a 15% relative increase in test performance compared to the baseline.
1 INTRODUCTION
Deep learning techniques have proven particularly useful in the classification setting, recently surpassing even human performance on some image-classification tasks. We seek to advance the state of the art by learning not only weights between neurons in the network but also part of the network structure itself – the activation functions. Current deep learning literature largely focuses on improving architectures and adding regularization to the training process.
By contrast, the realm of learning the network structure itself is relatively unexplored. Current best practice often treats network size, shape and choice of activation function as a hyper-parameter to be chosen empirically. Instead, we propose to learn the activation functions through nonparametric estimation. We introduce a class of nonparametric models for activation functions. Importantly, we ensure that our technique can be incorporated into the back-propagation framework. This is crucial because it means our method can be easily added to current practice. To this end, we propose learning functions via basis expansion. In particular, we find that using a Fourier basis works well in practice and offers improved performance over the baseline on several benchmark datasets. We see relative improvements in test error rates of up to 15%. Nonparametric activation functions in dropout nets are especially successful.
We also introduce a two-stage training process. First, a network without nonparametric activation functions is trained. Then, the learned network is used as an initialization for an identical network with nonparametric activation functions. This method of initializing the network yields considerable improvements in performance for convolution neural networks on image classification tasks.
Lastly, we consider the algorithmic stability approach to accessing generalization bounds. We use this to demonstrate that feed-forward networks with our method of nonparametrically estimating activation functions generalize well.
To summarize, our contributions are the following:
• A theoretically justified framework for learning activation functions in a neural network,
• Provable bounds on the generalization error of networks where the activation functions are learned, and
• An optional two-stage training process that can greatly improve results for certain network architectures, specifically those with convolution layers.
2 RELATED WORKS
Recent work from Agostinelli et al. (2015) describes piecewise linear activation functions. Theoretically speaking, our approach improves upon this by fully exploring the space of possible activation functions – not merely considering those which are piecewise linear.
Learning activation functions, in conjunction with the weights in the layer transformations, allows for the fitting of an arbitrary function of the output of the previous layer. A similar idea is Network in Network, due to Lin et al. (2013). There the authors propose to replace standard convolution layers with what they call mlpconv layers. Traditional convolution layers, if using a sigmoid activation, are essentially learning a generalized linear model of the previous layer’s output. Their innovation is to use a more general nonlinear function estimator as the filter – a multilayer perceptron. Thus, their scheme fits in the back-propagation framework and so is easy to implement. Our approach is different because though we are learning arbitrary functions of the previous layer, our activation function approach can be used in any layer of the network and in networks without convolution layers, which theirs cannot.
Maclaurin et al. (2015) propose to learn the hyper-parameters of a neural net through reverse gradient descent. This is similar to the work we present here in that they also provide a method for learning parameters of the network.
3 NONPARAMETRIC ACTIVATION FUNCTIONS
Here we describe our main contribution: the nonparametric activation function model. We also provide justification for a weight tying scheme to use nonparametric activations in conjunction with convolution layers, a regularization which proves powerful empirically (see Section 5).
3.1 FOURIER BASIS EXPANSION FOR ACTIVATION FUNCTIONS
For the nonparametric estimation we use a Fourier series basis expansion. One concern, however, with Fourier basis expansion is that function approximation over a fixed interval often exhibits poor tail behavior. To combat that, we fit the approximation over an interval [−L,L] and then use it over truncation of the interval, [−L+T, L+T ]. The designation NPF stands for Nonparametric Fourier Basis Expansion for Activation Functions.
Definition 3.1. Given some sample size n, an interval [−L,L] and tail truncation T where 0 ≤ T ≤ L, the activation function NPF(L, T ) is parametrized by (a0, . . . , ak) and (b1, . . . , bk) given by
f(x) =  a0 + ∑k i=1 ai cos((−L+ T )iπx/L) + bi sin((−L+ T )iπx/L) x < −L+ T a0 + ∑k i=1 ai cos(iπx/L) + bi sin(iπx/L) −L+ T ≤ x ≤ L− T
a0 + ∑k i=1 ai cos((L− T )iπx/L) + bi sin((L− T )iπx/L) x > L− T
k grows with sample size n and is given by k = dn1/7e.
We choose to present basis expansion of arbitrary functions on the interval [−L,L] because it is thought to be advantageous in practice to train neural networks with activation functions that are antisymmetric through the origin (LeCun et al., 1998). Though the nonparametric activation functions learned may not be symmetric about the origin, this approach will hopefully result in them being as close to symmetric as possible.
Each node in the fully connected layers estimates a potentially different activation function. However, in the convolution layers one activation function is learned per filter. The power of a convolution net is the weight tying scheme in the convolution layer, and we preserve that by only estimating one activation function per filter. In Section 5 the effectiveness of this approach can be seen.
NPFC(L, T ) is the notation for the nonparametrically estimated activation functions in convolution layers. Figure 1 shows some examples of activation functions learned on the CIFAR-10 dataset.
3.2 TRAINING WITH LEARNED ACTIVATION UNITS
A practical difficulty is that training nonparametric deep nets often proved unstable, specifically for networks with convolution layers. To remedy this our two-stage procedure is given as a generic description in Algorithm 1.
Algorithm 1 Generic Two Stage Training for Deep Convolutional Neural Networks Input: A network architecture, hyper parameters T ,L
1: Instantiate a network with ReLU activations in convolution layers and NPF(T, L) in all others. 2: Run training algorithm of choice. 3: Instantiate a new network with NPFC(T, L) activations in convolution layers and NPF(T, L) in
all others. Instantiate all weights from the trained network. Instantiate NPFC(T, L) as desired. 4: Run training algorithm of choice. 5: return Network weights resulting from the second training.
One point that is not immediately clear is how to initialize NPF(L, T ) activation functions. A good initialization, of any parameter in the network, is crucial to the success of fitting the model. We choose to initialize them to the Fourier series approximation of the tanh function. This works quite well as will be seen in Section 5.
Also left to choice is how to perform both stages of training. Of particular interest is the training method used for the second stage. We explored the following two approaches:
• In stage 2, train only the nonparametric activation functions, holding the remainder of the network constant.
• In stage 2, train all weights in the network together.
We discovered that both methods are successful, but allowing for the activations and weights in the network to be trained together gave slightly better results empirically, so it is those that are reported in Section 5.
Figure 2a shows the training path for a convolution network on the CIFAR-10 dataset. The network has three convolution layers followed by two fully connected layers. It is initially trained with rectified linear units as activations in the convolution layers and with NPF(4, 1) activations in the fully connected layers. The stage two network, instead, has NPFC(4, 1) activations in the convolution layer.
0 20 40 60 80 100
Epochs Trained
0%
10%
20%
30%
40%
50%
60%
70%
80%
E rr
o r
R a te
Begin Stage Two
Test and Training Error Rate on CIFAR-10 for Two Stage Training
Training Error
Test Error
(a) The training and testing error paths during training. Notice the jump in error rate after stage two begins, but that ultimately it settles into better generalization performance. This is for a convolutional neural net with three convolution and two fully connected layers trained on CIFAR-10.
0 5000 10000 15000 20000
Number of Weight Updates
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
E rr
o r
R a te
Test Error Rate on CIFAR-10
Single-Stage Training with NPF(4, 1) Activations
Baseline CIFAR-10
(b) Test errors for a convolution neural net trained on CIFAR-10. Each line represents one network trained with random initialization. Five random initializations and training paths for each network type are shown.
4 ALGORITHMIC STABILITY APPROACH TO GENERALIZATION BOUNDS
For our model to be theoretically sound, its generalization error should vanish with a sufficiently large sample size. This is true for standard neural networks, and such generalization bounds are well known (for example via VC bounds (Bartlett & Anthony, 1999)). However, because we expand the function class, we can not naively apply the traditional results. Instead of the VC bound route, we take the algorithmic stability approach.
In recent work from Hardt et al. (2015) the authors explore a class of bounds for stochastic gradient methods. They build on the definition of stability given in Bousquet & Elisseeff (2002). Denote by f(w, z) some loss function of the output of a model, where w indicates the parameters of the learned model and z is an example from X×Y , the space of data and labels. A is an algorithm and D is the training data. Denote w = A(D) as the parameters of the model learned by the algorithm A. We take the following definition from Hardt et al. (2015).
Definition 4.1. An algorithm is -uniformly stable if for all data sets D,D′ ∈ (X × Y )n such that D,D′ differ in at most one example, we have
sup z
EA[f(A(D), z)− f(A(D′), z)] ≤ .
The expectation is taken with respect to any randomness present in the algorithm itself. By stab(A,n) be the infimum over all for which the statement holds.
Both Bousquet & Elisseeff (2002); Hardt et al. (2015) present Theorem 4.2 regarding the generalization ability of -uniformly stable algorithms is given
Theorem 4.2. Take some algorithm -uniformly stable algorithm A. Then |ED,A[RD[A(D)]−R[A(D)]]| ≤ .
The notation RD[w] signifies the empirical risk when the loss function depends on w in addition to z. Likewise for the notation RD[A(D)]. Lastly, we have Theorem 4.5 also from Hardt et al. (2015) that will allow us to derive a generalization bound for our model trained with stochastic gradient descent.
Definition 4.3. A function f is L-Lipschitz if for all points u, v ∈ dom(f), |f(u)− f(v)| ≤ L||u− v||.
Definition 4.4. A function f : Ω→ R is β-smooth if for all u, v ∈ Ω, ||∇f(u)−∇f(v)|| ≤ β||u− v||.
Theorem 4.5. Assume that f(,̇z) ∈ [0, 1] is an L−Lipschitz and β-smooth loss function for every z. Suppose that we run SGM for T steps with monotonically non increasing step sizes αt ≤ c/t. Then SGM has uniform stability with
stab ≤ 1 + 1/βc
n− 1 (2cL2)
1 βc+1T βc βc+1 .
In particular, up to some constant factors, the following relationship approximately holds
stab ≤ T 1−1/(βc+1)
n .
(Hardt et al., 2015)
In order to apply the theorem, we demonstrate our network architecture and choice of loss function meets this requirement. Once -stability is established, generalization follows by Theorem 4.2.
4.1 FEED-FORWARD NETWORKS WITH NPF(T, L) ACTIVATIONS
For the full proofs of results in this section, see Appendix A. Assume a network architecture of Q fully connected hidden layers (affine transformation layers) fed into a softmax layer which classifies into one of c classes. The weight vector for the network w should be viewed as the all parameters of the affine transformations and activation functions.
The loss function is the negative log-likelihood of the network output, which is a softmax over c classes. We treat the softmax layer as two layers for the purposes of demonstrating the generalization
bound. One layer is an affine transformation of the output of the lower layers in the network to c nodes. We will denote the result of this computation at each node i in the affine transformation layer within the softmax layer as gQ+1(w, x)i. The second layer maps the outputs at these c nodes to c nodes – this will be the network output. It computes at the the ith node
oi(w, x) = exp gQ+1(w, x)i∑c i=1 exp gQ+1(w, x)i .
Thus, on some training example (x, y), the negative log-likelihood of the network output is
f(w, (x, y)) = − log(oi(w, x)) = −gQ+1(w, x)y + log ( c∑ i=1 exp gQ+1(w, x)c ) .
If we can demonstrate that this f has the desired properties, then the generalization bound given in Section 4 will apply.
One point to note is that in order to control the gradients of the various transformations in the layers in the network, we need to control the norm of the weight matrices W and biases b in the affine layers and the coefficients of the activations NPF(L, T ). This is a common technique and is compatible with the framework in Hardt et al. (2015), as it only makes updates less expansive. Note that though f does not have range [0, 1], it is bounded due to the max-norm constraints and can be rescaled. Some smoothing is assumed in order achieve the bound, but in practice the smoothing is not a concern as it can be assumed to take place in an -neighborhood of the transition from one piece of the activation function to another. The smoothed nonparametric activation functions are denoted NPF (L, T ). Recall that k = dn1/7e. Theorem 4.6. Consider a fully connected network with a softmax output layer and NPF (L, T ) activation functions. Assume we use SGD with a max-norm constraint on the weight matrix and biases in the affine layers and on the coefficients of the activation functions to train the network. Then there exists some K such that the loss function as given in Section 4.1 is K-Lipschitz. Specifically, for any such network where the number of hidden layers Q ≥ 1 and there are p nodes per layer,
K = max { O (√ Q(pQ+1k3Q+p(2Q+1)/2k3Q−1L) Q ) ,O (√ Q(pQ+1k3Q+p(2Q+1)/2k3Q−1L) QLQ )} .
Proof. The full proof is in Appendix A. The proof idea is to bound the first partial derivatives of the loss function f with respect to each parameter in the network. Then we can proceed recursively down through the network to obtain a bound. From there the gradient can be bounded, implying the loss function f is Lipschitz.
Theorem 4.7. Consider a fully connected network with a softmax output layer and NPF (L, T ) activation functions. Assume we use SGD with a max-norm constraint on the weight matrix and biases in the affine layers and on the coefficients of the activation functions to train the network. Then there exists some K such that the loss function as given in Section 4.1 is β-smooth. Specifically, for any such network where the number of hidden layers Q ≥ 1 and there are p nodes per layer,
β = max { O ( Q p3Q+5/2k9Q−2
L2Q 3Q−1
) ,O ( Q p2 Q+Q+2k5×2 Q+3QL2 Q−1+1
2Q+Q
)} .
Proof. The full proof is in Appendix A. The proof idea is to bound the second partial derivatives of f with respect to each of the parameters in the network. Then we can bound |λmax(∇2f)| and thus the largest singular value. This gives an upper bound of β on ||∇2f ||2, implying f is β-smooth.
Theorem 4.8. Consider a fully connected network with a softmax output layer and NPF (L, T ) activation functions. Assume we use SGD with a max-norm constraint on the weight matrix and biases in the affine layers and on the coefficients of the activation functions to train the network. Then, for all practical values of L,T , ,Q and p, the resulting Lipschitz constant K and smoothness constant β are sufficiently large that
stab ≤ T
n− 1 .
Thus if T = O( √ n), stab → 0 as sample size n→∞.
Proof. By Theorems 4.5, 4.6, and 4.7 the result is immediate.
5 EXPERIMENTAL RESULTS
In this section we provide empirical results for our novel nonparametric deep learning model. The goal of these experiments is to demonstrate its efficacy when used with several different architectures and on several different datasets, not to beat the current best result on either dataset. Crucially, our experiments provide an apples to apples comparison between networks with identical architectures and employing the same regularization techniques where the only difference is choice of activation function. The generalization results from the Section 4 are directly relevant to the experimental results here. In all experiments listed in this section, the reported testing error is an estimation of the generalization error because the final training error was zero in all cases.
5.1 IMPLEMENTATION, REGULARIZATION AND ARCHITECTURES
To implement the various deep architectures, we use the Python CPU and GPU computing library Theano (Bergstra et al., 2010). We use Theano as an interface to the powerful CUDA and CuDNN libraries (John Nickolls, 2008; Chetlur et al., 2014) enabling fast training for our neural networks. We ran our simulations on Princeton’s SMILE server and TIGER computing cluster. The SMILE server was equipped with a single Tesla K40c GPU, while the TIGER cluster has 200 K20 GPUs.
We use early stopping and dropout as regularization techniques (Srivastava et al., 2014; Hardt et al., 2015). The error rate reported for each experiment is the lowest testing error seen during training.
A common deep network architecture consists 3 stacked convolutional layers with 2 fully connected layers on top (Srivastava et al., 2014; Agostinelli et al., 2015). The convolutional layers have 96, 128, and 256 filters each and each layer has a 5 × 5 filter size that is applied with stride 1 in both directions. The max pooling layers pool 3 × 3 fields and are applied with a stride of 2 in both directions. We use this architecture for the more challenging CIFAR-10 dataset.
5.2 MNIST
This dataset consists of 60,000 training and 10,000 testing images. Images are 28 × 28 rasterized grayscale images – that is they have one channel. All data points are normalized before training by taking the entire combined dataset, finding the maximum and minimum entry, then centering and rescaling. The dataset is from Lecun & Cortes (1999).
The experiments investigate the effects of our techniques on standard multilayer neural networks as well as convolutional neural networks. Specifically, for the standard neural networks we use networks with three fully connected hidden layers with 2048 units in each layer fed into a 10-way softmax.
The convolutional neural networks in these experiments have two convolutional layers with 32 and 64 filters respectively. Filters are 5 × 5 and these layers are followed by a max pooling layer with 2×2 pool size. The feed-forward networks consist of 3 fully connected layers with 2048 units each. A mini-batch size of 250 was used for all experiments.
Table 1 contains the results from the MNIST baseline experiments.
For the experiments with NPF(L, T ) activations, we use the same fully connected and convolutional architectures as in the MNIST baseline experiments. Table 2 contains the results of the experiments.
Learning activation functions via Fourier basis expansion offers sizable improvements on the MNIST dataset as can be seen in Table 2. No experiments were done using our two-stage tech-
nique, as it proved unnecessary on this comparatively easy to learn dataset. Notice that the weight tying in the NPFC activation makes a considerable difference for the convolution nets, especially when dropout is used in conjunction with learning the activation functions. The biggest improvement is in learning activations for both convolution and fully connected layers and using dropout – a relative 15% improvement over the baseline. Using ReLU activations in the convolution layers with nonparametric activations in the fully connected layers also offered sizable performance improvements over the baseline.
In feedforward networks with three fully connected layers there also is an improvement in the test error rate. Here again the improvement is more pronounced with dropout, from 1.35% to 1.10% versus 1.66% to 1.60%.
5.3 CIFAR-10
The CIFAR-10 dataset is due to Krizhevsky (2009). This dataset consists of 50,000 training and 10,000 test images. The images are three channel color images with dimension 32× 32. Thus each image can be viewed either as a 3× 32× 32 tensor or a vector of length 3072. These images belong to one of ten classes. We apply ZCA whitening and global contrast normalization to the dataset as in Srivastava et al. (2014). The architecture described at the beginning of Section 5 is used for all experiments. For dropout nets we follow Srivastava et al. (2014) and use a dropout of 0.9 on the input, 0.75 in the convolution layers and 0.5 in the fully connected layers. The baseline results can be found in Table 3. Mini-batch sizes between 125 and 250 are used, depending upon memory constraints.
In Table 4, one-stage training corresponds to standard back-propagation as in the baseline experiments. The two-stage procedure is the one we introduce to train convolution nets with learned activation functions described in Section 3.2. As can be seen in Table 4, one stage training does not work when learning nonparametric activations in convolution layers – test error is worse than the baseline result. However, by using the two-stage process we see a relative 5% boost in performance without dropout and a relative 9% boost with dropout! In absolute terms we achieve up to a 1.3% improvement in generalization performance using learned activations.
Also in Table 4, we can see that we achieved improved performance both with and without dropout when using ReLU activations for convolution layers and NPF(4, 1) for fully connected layers. Furthermore, Figure 2b shows that in terms of number of weight updates required, adding nonparametric activations did not slow training.
6 DISCUSSION AND CONCLUSIONS
As can be seen in Section 5, nonparametric activation functions offer a meaningful improvement in the generalization performance of deep neural nets in practice. We achieve relative improvements in performance of up to 15% and absolute improvements of up to 1.3% on two benchmark datasets. Equally importantly, networks with the activation functions NPF(L, T ) and NPFC(L, T ) can be trained as fast as their baseline comparison. To realize these improvements on more challenging datasets such as CIFAR-10 we introduced a two-stage training procedure. Our nonparametric activation functions are principled in that they have the necessary properties to guarantee vanishing generalization error in the sense of Bousquet & Elisseeff (2002). Specifically, it was shown in Theorem 4.8, from Section 4.1, that for any feed-forward network architecture using NPF(L, T ) activation functions, we achieve vanishing generalization error as sample size increases.
One interesting direction for future work is to investigate why Fourier basis expansion is successful where other methods, such as polynomial basis expansion (which we also explored), were not. Both can be theoretically justified, yet only one works well in practice. Further study of how to eliminate the two-stage process is needed.
Ultimately, we improve upon other approaches of expanding the function class which can be learned by deep neural networks. An important previous work on learning more general function approximations at each node is that of Lin et al. (2013). Their result offers a new type of convolution filter that can learn a broader function class, and in practice their technique also works well. It is limited however in that its use is restricted to convolution layers in convolution neural networks. Ours is applicable to any network architecture and, in Section 5, we demonstrate its success on multiple datasets.
A GENERALIZATION BOUND FOR FULLY CONNECTED NETWORKS WITH FOURIER BASIS ACTIVATION FUNCTIONS
This is an extension of Section 4.1. For the purposes of this section assume that all data x are vectors in [−1, 1]m, which is permissible because we classify input that is bounded. Furthermore, we assume that the network architecture is a series of Q fully connected hidden layers (affine transformation layers) fed into a softmax layer which classifies into one of c classes. The weight vector for the network w should be viewed as the all parameters of the affine transformations and activation functions.
The loss function is the negative log-likelihood of the network output, which is a softmax over c classes. The softmax layer in our framework should be treated as two layers for the purposes of demonstrating the generalization bound. One layer is an affine transformation of the output of the lower layers in the network to c nodes. We will denote the result of this computation at each node i in the affine transformation layer within the softmax layer as gQ+1(w, x)i. The second layer maps the outputs at these c nodes to c nodes – this will be the network output. It computes at the the ith node
oi(w, x) = exp gQ+1(w, x)i∑c i=1 exp gQ+1(w, x)i .
Thus, on some training example (x, y), the negative log-likelihood of the network output is
f(w, (x, y)) = − log(oi(w, x)) = −gQ+1(w, x)y + log ( c∑ i=1 exp gQ+1(w, x)c ) .
If we can demonstrate that this f has the desired properties, then the generalization bound given in Section 4 will apply.
For clarity, we refer to network of stacked affine transformation layers as a fully connected network. These layers compute Wx + b, where x is either the input layer (so x ∈ Rm) or it is the output of the previous layer in the network (in which case x ∈ Rp, p being the number of nodes in that layer). Regarding notation:
• w is a vector denoting all the parameters in the network, so it consists of all parameters noted below.
• Wi denotes a portion of the parameter vector, corresponding to the affine transformation of the pi−1-dimensional output of the (i− 1)th layer to the pi-dimensional input to the ith layer in the network.
• Wi,j denotes the jth row of the matrix Wi.
• Wi,j,k refers to the weight of the edge from node j in the (i − 1)th layer to edge c in the ith layer.
• bi denotes the constant term in the affine transformation from the (i− 1)th layer to the ith layer.
• bi,j denotes the jth component of the vector bi.
• ai,j0 , (a i,j 1 , . . . , a i,j k ) and (b i,j 1 , . . . , b i,j k ) denote the parameters of activation function of the
jth node in the ith layer.
The 0th layer is the input layer. Let Q be the number of layers in the network; if i = Q+ 1, it refers to weights on edges in the affine transformation contained within the softmax layer.
Intuitively, our choice of network topology and use of NPF(L, T ) activation function should have the necessary properties to apply Theorem 4.5. The definition of NPF(L, T ) is repeated below for clarity.
Definition A.1. Given some sample size n, an interval [−L,L] and tail truncation T where 0 ≤ T ≤ L, the activation function NPF(L, T ) is parametrized by (a0, . . . , ak) and (b1, . . . , bk) given
by
f(x) =  a0 + ∑k i=1 ai cos((−L+ T )iπ/L) + bi sin((−L+ T )iπ/L) x < −L+ T a0 + ∑k i=1 ai cos(iπx/L) + bi sin(iπx/L) −L+ T ≤ x ≤ L− T
a0 + ∑k i=1 ai cos((L− T )iπ/L) + bi sin((L− T )iπ/L) x > L− T
k grows with sample size n and is given by k = dn1/7e.
We need to control the norm of the weight matrices W and biases b in the affine transformations. In addition, we also need to control the magnitude of the coefficients in the basis expansion. This does not affect the applicability of the results from Hardt et al. (2015), as it at most reduces the expansivity of the updates.
Furthermore, for the analysis of their properties we assume a slightly modified version of the activation functions – in an epsilon neighborhood about the points where the pieces of the function connect we smooth their joint by taking a convex combination of the two pieces to smoothly transition from one to the other. To this end define the following functions
L(x) = a0 + k∑ i=1 ai cos((−L+ T )iπ/L) + bi sin((−L+ T )iπ/L),
M(x) = a0 + k∑ i=1 ai cos(iπx/L) + bi sin(iπx/L),
U(x) = a0 + k∑ i=1 ai cos((L− T )iπ/L) + bi sin((L− T )iπ/L).
for given L, T, k, (a0, . . . , ak), and (b1, . . . , bk). Then the -neighborhood smoothed nonparametric activation functions are given by Definition A.2.
Definition A.2. Given some sample size n, an interval [−L,L] and tail truncation T where 0 ≤ T ≤ L, the activation function NPF (L, T ) is parametrized by (a0, . . . , ak) and (b1, . . . , bk) given by
f(x) =  L(x) x < −L+ T − [Case 1] x+L−T+ 2 M(x) + −L+T+ −x 2 L(x) −L+ T − ≤ x ≤ −L+ T + [Case 2] M(x) −L+ T + ≤ x ≤ L− T − [Case 3] L−T+ −x 2 M(x) + x−L+T+
2 U(x) L− T − ≤ x ≤ L− T + [Case 4] U(x) x > L− T + [Case 5]
k grows with sample size n and is given by k = dn1/7e.
Since we are concerned with behavior as is made small – because we are approximating NPF(T, L) with NPF (T, L) – assume without loss of generality that ≤ 1.
A.1 LIPSCHITZ CONSTANT
We proceed layer by layer to obtain bounds on the first partial derivatives of the loss function – which will allow us to bound the gradient. Lemma A.3 gives bounds for the first layer. The notation a(w, x) denotes the output of a layer-wide activation transformation. g(w, x) denotes the output of a layer-wide affine transformation.
Lemma A.3. Assume x ∈ [−a, a]m. Now, consider the affine transformation g1(w) = W1x + b1 from Rm → Rp, thus W1 ∈ Rm×p and b ∈ Rp. Denoting by g1(w, x)i = W1,ix + b1,i, it follows that g1 has first partial derivatives bounded by max{1, a}.
Proof. Clearly,∇g1(w, x)i = [1, . . . , 1;x; 0, . . . , 0]. Thus it has first partial derivatives bounded by max{1, a}.
What if we have instead as input to the affine transformation a function of x rather than x itself? Then we have the Lemma A.4.
Lemma A.4. Let ai−1(w, x) ∈ [−a, a]pi−1 , the output of the (i − 1)th layer in the network. Now, consider the affine transformation
gi(w, x) = Wiai−1(w, x) + bi.
from Rpi−1 → Rpi . bi ∈ Rd. If we have the following assumptions:
• ai(w, x)q has first partial derivatives bounded by B, and
• ||Wi,j ||2 ≤ N .
Denoting by gi(w, x)j = Wi,jai−1(w, x) + bi,j , it follows that gi(w, x)j has first partial derivatives bounded by max{1, a, √ nNL}.
Proof. We need to first examine
∂gi(w, x)j ∂bi,j = ∂bi,j ∂bi,j + ∂ ∂bi,j ( n∑ q=1 Wi,j,qai−1(w, x)q ) = 1 + 0 = 1.
Next we have for any m,
∂gi(w, x)j ∂Wi,j,m = ∂bi,j ∂Wi,j,m +
∂
∂Wi,j,m ( n∑ q=1 Wi,j,qai−1(w, x)q )
= 0 + ai−1(w, x)m + ∑ q 6=m Wi,j,q ∂ai−1(w, x)q ∂Wi,j,m
= ai−1(w, x)m.
Partial derivatives with respect to all other parameters from the ith layer are clearly 0. If l > i, partial derivatives with respect to all parameters from the lth layer are clearly 0. If l < i,∣∣∣∣∂gi(w, x)j∂Wl,m,q ∣∣∣∣ = ∣∣∣∣∣ ∂bi,j∂Wl,m,q + ∂∂Wl,m,q ( pi−1∑ r=1 Wi,j,rai−1(w, x)r )∣∣∣∣∣ =
∣∣∣∣∣0 + pi−1∑ r=1 Wi,j,r ∂ai−1(w, x)r ∂Wl,m,q ∣∣∣∣∣ ≤ N
√√√√pi−1∑ r=1 ( ∂ai−1(w, x)r ∂Wl,m,q )2 (Cauchy-Schwartz)
≤ √pi−1NB. Likewise,
|∂gi(w, x)j ∂bl,q | ≤ √pi−1NB
and for the partial derivatives with respect to activation function parameters. It follows that the transformations gi(w, x)j have first partial derivatives bounded by max{1, a, √ pi−1NB}.
Lemma A.5 provides a bound on the range of affine transformations of outputs of NPF (T, L) activations.
Lemma A.5. Consider the affine transformation in layer i, gi(w, x). Assume
• All layers in the network have NPF (L, T ) activations and that coefficients in these activations are bounded by M ,
• For all i and j, ||Wi,j ||2 ≤ N , and
• For all i and j, |bi,j | ≤ O.
Then, for all j, |gi(w, x)j | ≤ √ pi−1(2k + 1)NM +O = O( √ pi−1k).
Proof. Recall that gi(w, x)j = Wi,jai−1(w, x)+bi,j , thus by Cauchy-Schwartz and that im(sin) = im(cos) = [−1, 1],
|gi(w, x)j | ≤ √ pi−1(2k + 1)NM +O = O( √ pi−1k).
Next, we bound the component functions of NPF (L, T ). In addition, to derive asymptotic bounds, we assume N ,M , and O are constants as they are part of the training procedure and not the network structure. We are more concerned with how the smoothness and continuity properties change as n,k,L,T ,Q and pi change. Also note that for the activation function NPF (L, T ) to be well defined, it must be that ≤ L. We also assume ≤ 1. Lemma A.6. Assume gi(w, x) ∈ [−a, a]pi and that gi(w, x)j has first partial derivatives bounded by B. Further assume that |ad| ≤ M and define f(w, x) = ad cos ( dπgi(w,x)j
L
) . Then f(w, x) has
first partial derivatives bounded by max{dπMBL , 1}.
Proof. Taking the derivative of f with respect to ad yields∣∣∣∣ ∂f∂ad ∣∣∣∣ = ∣∣∣∣cos(dπgi(w, x)jL )∣∣∣∣ ≤ 1. Next if we take the derivative of f with respect to any weight from elsewhere in the network – some z – we get
∂f ∂z = −dπad L ∂gi(w, x)j ∂z sin
( dπgi(w, x)j
L
) ,
and therefore that ∣∣∣∣∂f∂z ∣∣∣∣ ≤ ∣∣∣∣dπadL ∂gi(w, x)j∂z ∣∣∣∣ ≤ dπMBL , and so the partial derivatives of f are bounded by max{dπMBL , 1}.
Immediately from the proof to Lemma A.6 we see that an identical result holds if we replaced cos with sin.
Lemma A.7. Let gi(w, x) ∈ [−a, a]pi−1 and assume
• gi(w, x)j has first partial derivatives bounded by B,
• All layers in the network have NPF (L, T ) activations and that coefficients in these activations are bounded by M ,
• For all i and j, ||Wi,j ||2 ≤ N , and
• For all i and j, |bi,j | ≤ O.
Consider the layer-wide activation transformation ai(w, x) where ai(w, x)j is given by a function of the type NPF (L, T ). ai(w, x)j then has first partial derivatives with bound
B′ = O ( max {√ pi−1k 3B + k2L
L ,
√ pi−1k 3B + k2L }) .
Proof. To bound the partial derivatives, first consider Cases 1 and 5, clearly the derivative with respect to some parameter z is 0. For Case 3, by Lemma A.6,∣∣∣∣∂M(gi(w, x)j)∂z ∣∣∣∣ ≤ (k2 + k)MBπL + (k2 + k) = (k2 + k)(MB + L)πL = O ( k2 + k2B L ) .
Next consider Case 2, we have ∂
∂z
( gi(w, x)j + L− T +
2 M(gi(w, x)j) + −L+ T + − gi(w, x)j 2 L(gi(w, x)j) ) = ∂
∂z
( gi(w, x)j + L− T +
2 M(gi(w, x)j)
) − ∂gi(w, x)j
∂z
1
2 L(gi(w, x)j)
= 1
2 ∂gi(w, x)j ∂z M(gi(w, x)j) +
( gi(w, x)j + L− T +
2
) ∂M(gi(w, x)j)
∂z − ∂gi(w, x)j ∂z 1 2 L(gi(w, x)j)
=
( M(gi(w, x)j)− L(gi(w, x)j)
2
) ∂gi(w, x)j
∂z +
( gi(w, x)j + L− T +
2
) ∂M(gi(w, x)j)
∂z .
Both M(x) and L(x) are trivially bounded above by (k2 + k + 1)M . Therefore∣∣∣∣ ∂∂z ( gi(w, x)j + L− T + 2 M(gi(w, x)j) + −L+ T + − gi(w, x)j 2 L(gi(w, x)j) )∣∣∣∣ = ∣∣∣∣(M(gi(w, x)j)− L(gi(w, x)j)2 ) ∂gi(w, x)j ∂z + ( gi(w, x)j + L− T + 2 ) ∂M(gi(w, x)j) ∂z
∣∣∣∣ ≤ (k 2 + k + 1)BM + (√ pi−1(2k + 1)NM +O + L+
2
)( (k2 + k)(MB + L)π
L ) = O ( max {√ pi−1k 3B + k2L
L ,
√ pi−1k 3B + k2L }) .
By symmetry, we have the same bound for Case 4. Thus overall, the bound on the first partial derivatives of ai(w, x)j is
O(max {√ pi−1k 3B + k2L
L ,
√ pi−1k 3B + k2L } ),
as for all values of k,B,L, and pi−1 the bound for Cases 2 and 4 dominates the bound for Case 3.
Finally, we turn to the top layer of the network.
Lemma A.8. With the notation established in Section A, assume there are c classes, the network has Q layers of affine transformations, and each gL(w, x)j has first partial derivatives bounded by B. Then the loss function f is 2B √ (pl + 1)c+ ∑Q i=1 pi(pi−1 + 1)−Lipschitz. If all layers have
p nodes, then f is 2B √
(Q− 1)p2 + (c+m+Q)p+ c-Lipschitz. In asymptotic notation, it is K-Lipschitz with K = O(Bp √ Q).
Proof. Consider the partial derivative of f with respect to an arbitrary parameter z
∂f ∂z = ∂gL(w, x)y ∂z + 1∑c j=1 exp(gL(w, x)j) c∑ j=1 ∂gL(w, x)j ∂z exp(gL(w, x)j).
By the assumption |∂gL(w,x)y∂z | ≤ Q. Therefore the following holds∣∣∣∣∂f∂z ∣∣∣∣ = ∣∣∣∣∣∣∂gL(w, x)y∂z + 1∑cj=1 exp(gL(w, x)j) c∑ j=1 ∂gL(w, x)j ∂z exp(gj) ∣∣∣∣∣∣ ≤ ∣∣∣∣∂gL(w, x)y∂z ∣∣∣∣+ ∣∣∣∣∣∣ 1∑cj=1 exp(gL(w, x)j) k∑ j=1 ∂gL(w, x)j ∂z exp(gL(w, x)j)
∣∣∣∣∣∣ ≤ B + ∣∣∣∣maxj ∂gL(w, x)j∂z ∣∣∣∣
≤ 2B. If layer i has pi nodes there are pi × pi−1 + pi = pi(pi−1 + 1) parameters corresponding to the affine transformation from layer i − 1 to layer i. Recall layer 0 is the input layer, so p0 = m.
Furthermore, the affine transformation in the softmax has pl × c + c parameters. Thus the total number of parameters in the network is
(pl + 1)c+ Q∑ i=1 pi(pi−1 + 1),
which implies that f is 2B √ (pl + 1)c+ ∑Q i=1 pi(pi−1 + 1)−Lipschitz. If all layers have the same number of nodes, then the total number of parameters in the network is (Q− 1)p2 + (c+m+Q)p+ c
and so in this case f is 2B √
(Q− 1)p2 + (c+m+Q)p+ c-Lipschitz. In asymptotic notation, it is K-Lipschitz with K = O(Bp √ Q).
Combining all the results above we have Theorem A.9.
Theorem A.9. Consider a fully connected network with a softmax output layer and NPF (L, T ) activation functions. Assume we use SGD with a max-norm constraint on the weight matrix and biases in the affine layers and on the coefficients of the activation functions to train the network. Then there exists some K such that the loss function as given in Section A is K-Lipschitz. Specifically, for any such network, where the number of hidden layers Q ≥ 1 and there are p nodes per layer,
K = max { O (√ Q(pQ+1k3Q + p(2Q+1)/2k3Q−1L)
Q
) ,O (√ Q(pQ+1k3Q + p(2Q+1)/2k3Q−1L)
QLQ
)} .
Proof. By Lemmas A.3, A.4, A.7 and A.8 such an K must exist. Note that the output of a NPF (T, L) activation is in [−(k2 +k+ 1)N, (k2 +k+ 1)N ]. Then, the computation is as follows, using the aforementioned theorems:
• g1 – B = 1 • a1 – B = max { O (√ pk3+k2L ) ,O (√ pk3+k2L
L )} • g2 – B = max { O ( pk3+ √ pk2L ) ,O ( pk3+ √ pk2L
L )} • a2 – B = max { O ( p3/2k6+pk5L 2 ) ,O ( p3/2k6+pk5L
2L2 )} • g3 – B = max { O ( p2k6+p3/2k5L 2 ) ,O ( p2k6+p3/2k5L
2L2 )} • a3 – B = max { O ( p5/2k9+p2k8L 3 ) ,O ( p5/2k9+p2k8L
3L3 )} Above the transformations are listed with the bounds on their first partial derivatives. A simple inductive argument yields for i ≥ 2
• gi –
B = max { O ( pi−1k3(i−1) + p(2i−3)/2k3i−4L
i−1
) ,O ( pi−1k3(i−1) + p(2i−3)/2k3i−4L
i−1Li−1 )} • ai –
B = max { O ( p(2i−1)/2k3i + pi−1k3i−1
i
) ,O ( p(2i−1)/2k3i + pi−1k3i−1
iLi )} Using Lemma A.8 gives
K = max { O (√ Q(pQ+1k3Q + p(2Q+1)/2k3Q−1L)
Q
) ,O (√ Q(pQ+1k3Q + p(2Q+1)/2k3Q−1L)
QLQ
)} .
concluding the proof.
A.2 β-SMOOTHNESS
To derive the smoothness constant, we begin at the top of the network and work down, bounding partial second derivatives of transformations in the network. Lemma A.10. With the notation established in Section A, assume there are c classes and each gj has first partial derivatives bounded by B1 and second partial derivatives bounded by B2. Then the loss function f is 2((pl + 1)c+ ∑Q i=1 pi(pi−1 + 1))(B 2 1 +B2)-smooth. If all layers have p nodes, then f is 2((Q− 1)p2 + (c+m+Q)p+ c)(B21 +B2)-smooth.
Proof. We begin by recalling from the proof of Lemma A.8 that for some arbitrary parameter z,
∂f ∂z = ∂gL(w, x)y ∂z + 1∑c l=1 exp(gj) c∑ l=1 ∂gL(w, x)l ∂z exp(gL(w, x)l).
Therefore differentiating again with respect to another parameter x yields
∂2f ∂z∂x = ∂2gL(w, x)y ∂z∂x − [ c∑ l=1 ∂gL(w, x)l ∂z exp(gL(w, x)l) ][ c∑ l=1 ∂gL(w, x)l ∂x exp(gL(w, x)l) ] ×
[ c∑ l=1 exp(gL(w, x)l) ]−2 + [ c∑ l=1 ( ∂2gL(w, x)l ∂z∂x + ∂gL(w, x)l ∂z ∂gL(w, x)l ∂x ) exp(gL(w, x)l) ] ×
[ c∑ l=1 exp(gL(w, x)l) ]−1 .
If we take the absolute value of both sides we can see that∣∣∣∣ ∂2f∂z∂x ∣∣∣∣ ≤ 2B21 + 2B2.
And therefore by Theorem C.2 it follows that
|λmax(∇2f)| ≤ 2((pl + 1)c+ Q∑ i=1 pi(pi−1 + 1))(B 2 1 +B2).
Because∇2f is symmetric, its singular values are just the absolute values of its eigenvalues. Therefore ||∇2f ||2 ≤ 2n(B21 + B2). By Lemma D.2, we have that f is 2(2(pl + 1)c+ ∑Q i=1 pi(pi−1 + 1))(B21 +B2)-smooth. If all layers have the same number of nodes, then |λmax(∇2f)| ≤ 2((Q− 1)p2 + (c+m+Q)p+ c)(B21 +B2) and so f is 2((Q− 1)p2 + (c+m+Q)p+ c)(B21 +B2)-smooth.
Next, it is necessary to control the second partial derivatives of the affine transformations as the first derivatives are bounded in Lemma A.4. This leads to Lemma A.11. Lemma A.11. Let ai−1(w, x) ∈ [−a, a]pi−1 , the output of the (i− 1)th layer in the network. Now, consider the affine transformation
gi(w, x) = Wiai−1(w, x) + bi,
from Rpi−1 → Rpi . bi ∈ Rd. If we have the following assumptions:
• ai(w, x)q has first partial derivatives bounded by B1,
• ai(w, x)q has second partial derivatives bounded by B2, and
• ||Wi,j ||2 ≤ N .
Denoting by gi(w, x)j = Wi,jai−1(w, x) + bi,j , it follows that gi(w, x)j has second partial derivatives bounded by max{B1, N √ pi−1B2}.
Proof. From the proof of Lemma A.4 we have that ∂gi(w, x)j ∂bi,j = 1
and for any m ∂gi(w, x)j ∂Wi,j,m = ai−1(w, x)m.
For some m 6= j and any q, ∂gi(w, x)j ∂Wi,m,q = 0.
And, lastly, that the partial derivatives with respect to any parameter z in layer l < i
∂gi(w, x)j ∂z = pi−1∑ q=1 Wi,j,q ∂ai−1(w, x)q ∂z .
We see then that for the first and third cases the second partial derivatives are zero. For the second case above, the second partial derivatives are either 0 or bounded by B1. What remains then is to check the fourth case. If we differentiate with respect to bi,j ,
∂2gi(w, x)j ∂z∂bi,j = 0.
If we differentiate with respect to Wi,m,q , for m 6= j, ∂2gi(w, x)j ∂z∂Wi,m,q = 0.
If we differentiate with respect to Wi,j,q , ∂2gi(w, x)j ∂z∂Wi,j,q = ∂ai−1(w, x)q ∂z . Thus the only remaining case is if we differentiate with respect to some parameter z′ from a layer l < i:
∂2gi(w, x)j ∂z∂z′ = pi−1∑ q=1 ∂ ∂z′ ( Wi,j,q ∂ai−1(w, x)q ∂z )
= pi−1∑ q=1 Wi,j,q ∂2ai−1(w, x)q ∂z∂z′ .
If we take the absolute value of both sides we get that∣∣∣∣∂2gi(w, x)j∂z∂z′ ∣∣∣∣ = ∣∣∣∣∣ pi−1∑ q=1 Wi,j,q ∂2ai−1(w, x)q ∂z∂z′ ∣∣∣∣∣ ≤ N
√√√√pi−1∑ q=1 ( ∂2ai−1(w, x)q ∂z∂z′ )2 (Cauchy-Schwartz)
≤ N√pi−1B2 And therefore gi(w, x)j has second partial derivatives bounded by max{B1, N √ pi−1B2}.
We also consider affine transformations on the input – although this is covered by Lemma A.11, we can obtain a better bound directly.
Lemma A.12. Assume x ∈ [−a, a]m. Now, consider the affine transformation g1(w) = W1x+ b1 from Rm → Rp, thus W1 ∈ Rm×p and b ∈ Rp. Denoting by g1(w, x)i = W1,ix + b1,i, it follows that g1 has second partial derivatives bounded by 0.
Proof. Recall that from the proof of Lemma A.3 ∇g1(w, x)i = [1, . . . , 1;x; 0, . . . , 0]. Thus its second derivatives are all 0.
Finally, a bound is needed for the second partial derivatives of the NPF (L, T ) activations.
Lemma A.13. Assume gi(w, x) ∈ [−a, a]pi and that gi(w, x)j has first partial derivatives bounded by B1 and second partial derivatives bounded by B2. Further assume that |ad| ≤ M and define f(w, x) = ad cos ( dπgi(w,x)j
L
) . Then f(w, x) has second partial derivatives bounded by
max { dπMLB2+d
2π2MB21 L2 , dπB1 L
} .
Proof. Recall from the proof of Lemma A.6 that taking the derivative of f(w, x) with respect to ad yields
∂f(w, x)
∂ad = cos
( dπgi(w, x)j
L
) ,
and that if we take the derivative of f(w, x) with respect to any weight from elsewhere in the network – some z – we get
∂f(w, x) ∂z = −dπad L ∂gi(w, x)j ∂z sin
( dπgi(w, x)j
L ) Now, if we take the derivative of ∂f(w,x)∂ad with respect to ad we get 0. If we take the derivative of ∂f(w,x) ∂ad with respect to some z we get
∂2f(w, x)
∂ad∂z = − sin
( dπgi(w, x)j
L
)( dπ
L ∂gi(w, x)j ∂z ) and we can bound this as ∣∣∣∣∂2f(w, x)∂ad∂z
∣∣∣∣ ≤ dπB1L . Next if we take the derivative of ∂f(w,x)∂z with respect to ad we get the same as above. Lastly we take the derivative of ∂f(w,x)∂z with respect to some z ′ which gives ∂2f(w, x)
∂z∂z′ = −dπad L ∂2gi(w, x)j ∂z∂z′ sin
( dπgi(w, x)j
L
) −d
2π2ad L2 ∂gi(w, x)j ∂z ∂gi(w, x)j ∂z′ cos
( dπgi(w, x)j
L
) ,
which we can bound as∣∣∣∣∂2f(w, x)∂z∂z′ ∣∣∣∣ ≤ dπMB2L + d2π2MB21L2 = dπMLB2 + d2π2MB21L2 .
Therefore the second partial derivatives of f are bounded by max { dπMLB2+d
2π2MB21 L2 , dπB1 L
} .
Lemma A.14. Let gi(w, x) ∈ [−a, a]pi−1 and assume
• gi(w, x)j has first partial derivatives bounded by B1,
• gi(w, x)j has first partial derivatives bounded by B2,
• All layers in the network have NPF (L, T ) activations and that coefficients in these activations are bounded by M ,
• For all i and j, ||Wi,j ||2 ≤ N , and
• For all i and j, |bi,j | ≤ O.
Consider the layer-wide activation transformation ai(w, x) where ai(w, x)j is given by a function of the type NPF (L, T ). ai(w, x)j then has second partial derivatives bounded by
B′ = O ( B22k 2 + B1k 2 + √ pi−1k 3(B1 +B2)
L + k4B21 L2 + k3B21 L
) .
Proof. Without loss of generality we can assume that if Bi 6= 0 that Bi ≥ 1, as these are upper bounds. To bound the second partial derivatives, first note that Cases 1 and 5 have partial second
derivatives of 0 with respect to all parameters in the network. Next, in Case 3 we see from Lemma A.13 that∣∣∣∣∂2M(gi(w, x)j)∂z∂z′ ∣∣∣∣ ≤ 2 k∑ d=1 ( dπMLB2 + d 2π2MB21 L2 + dπB1 L ) =
(k2 + k)πMLB2 L2 +
( k3
3 + k2 2 + k 6 ) π2MB21 L2 + (k2 + k)πB1 L
= O ( k2(B1 +B2)
L + k3B21 L2 ) for any two parameters z and z′. Denote
c1 = L− T + and c2 = −L+ T + . In Case 5 then, we can compute
∂2ai(w, x)j ∂z∂z′ = ∂ ∂z′
( ∂gi(w, x)j
∂z′ M(gi(w, x)j) 2 + ∂M(gi(w, x)j) ∂z′ gi(w, x)j + c1 2
−∂gi(w, x)j ∂z′ L(gi(w, x)j) 2 ) = ∂2gi(w, x)j ∂z∂z′ M(gi(w, x)j) 2 + ∂gi(w, x)j ∂z′ ∂M(gi(w, x)j) ∂z 1 2
+ ∂gi(w, x)j
∂z
∂M(gi(w, x)j) ∂z′ 1 2 + ∂2M(gi(w, x)j) ∂z∂z′ gi(w, x)j + c1 2
− ∂ 2gi(w, x)j ∂z∂z′ L(gi(w, x)j)
2 and then bound it, using B3 and B4 as bounds on the first and second partial derivatives of M , respectively,∣∣∣∣∂2ai(w, x)j∂z∂z′
∣∣∣∣ ≤ B22 (k2 + k + 1)M + B1B3 +B4 a+ c12 = O ( B22k 2 + B1k 2 + √ pi−1k 3(B1 +B2)
L + k4B21 L2 + k3B21 L + k2(B1 +B2) L
+ k3B21 L2 ) = O ( B22k 2 + B1k 2 + √ pi−1k 3(B1 +B2)
L + k4B21 L2 + k3B21 L
) .
Since this bound dominates the one for Case 3, we are done.
Combining all the results above we have Theorem A.15. Theorem A.15. Consider a fully connected network with a softmax output layer and NPF (L, T ) activation functions. Assume we use SGD with a max-norm constraint on the weight matrix and biases in the affine layers and on the coefficients of the activation functions to train the network. Then there exists someK such that the loss function as given in Section A is β-smooth. Specifically, for any such network where the number of hidden layers Q ≥ 1 and and there are p nodes per layer,
β = max { O ( Q p3Q+5/2k9Q−2
L2Q 3Q−1
) ,O ( Q p2 Q+Q+2k5×2 Q+3QL2 Q−1+1
2Q+Q
)} .
Proof. By Lemmas A.3, A.4, A.7, A.10, A.11, A.14 and A.12 such a β must exist.
To compute this β, the aforementioned results can be used as well as the computations from the proof of Theorem A.9. The computation is as follows, using the aforementioned results,
• g1 – B = 0 • a1 – B = max { O (√ pk3 L + k4 L2 ) ,O ( k2 + √ pk3 L + k4 L2 )}
• g2 – B = max { O ( pk3 L + √ pk4 L2 ) ,O (√ pk3 + k2L + pk3 L + √ pk4 L2 )} • a2 – B = max { O ( p2k10 L4 3 ) ,O ( p2k10L2 2
)} • g3 – B = max { O ( p4k16 L6 5 ) ,O ( p5/2k10L2 2
)} Above, next to each transformation, is the bound B on its second partial derivatives. The bounds are computed with respect to L ≤ 1 and L > 1, and then the maximum is taken over both cases. By a simple inductive argument for i ≥ 2,
• gi –
B = max { O ( p2i−3/2k6i−8
L2i−2 2i−3
) ,O ( p2 i−1−1/2k5×2 i−1 L2 i−2
2i−1
)}
• ai –
B = max { O ( p2i−2k6i−2
L2i 2i−1
) ,O ( p2 i−1k5×2 i L2 i−1
2i
)}
Using the bounds derived above, the bounds from Theorem A.9 and Lemma A.10 gives
β = max { O ( Q p3Q+5/2k9Q−2
L2Q 3Q−1
) ,O ( Q p2 Q+Q+2k5×2 Q+3QL2 Q−1+1
2Q+Q
)} .
A.3 GENERALIZABILITY
With the necessary properties demonstrated, we arrive at Theorem A.16. By p denote the number of nodes per layer and Q the number of hidden layers.
Theorem A.16. Consider a fully connected network with a softmax output layer and NPF (L, T ) activation functions. Assume we use SGD with a max-norm constraint on the weight matrix and biases in the affine layers and on the coefficients of the activation functions to train the network. Then, for all practical values of L,T , ,Q and p, the resulting Lipschitz constant K and smoothness constant β are sufficiently large that
stab ≤ T
n− 1 .
Thus if T = O( √ n), stab → 0 as sample size n→∞.
Proof. By Theorems 4.5, A.9, and A.15 the result is immediate.
B RESULTS FOR FUNCTIONS OVER Rn
Without proof we give the mean value inequality for vector valued function in Theorem B.1.
Theorem B.1. Let f be a differentiable vector valued function, that is f : A ⊂ Rn → Rm. Then for x, y ∈ A, we have that
||f(x)− f(y)|| ≤ ||Df(z)(x− y)||,
where z is some point on the line segment connecting x, y. HereDf represents the Frechet derivative of f , which can be represented in matrix form by the Jacobian.
C RESULTS FOR MATRICES
When we want to bound the largest eigenvalue of the matrix, we can get a loose bound by the sum of the entries of the matrix as in Theorems C.1 and C.2. Theorem C.1. For some matrix norm || · ||, the spectral radius of the matrix A is upper bounded by ||A||.
Proof. Take the eigenvector corresponding to the largest eigenvalue in magnitude of A. Denote this pair by x, λ. Next define the matrix X = [x| · · · |x]. It follows then that AX = λX . Therefore
|λ|||X|| = ||λX|| = ||AX|| ≤ ||A||||X|| because matrix norms are sub-multiplicative, demonstrating |λ| ≤ ||A||.
Theorem C.2. Let A be a symmetric n× n matrix. Let the entries of the matrix be bounded above by some constant k. It follows then that maxk |λk(A)| ≤ nk.
Proof. By Theorem C.1 we can see that
max k |λk(A)| ≤ √∑ i,j (Aij)2 ≤ √ n2k2 = nk.
D RESULTS FOR LIPSCHITZ AND SMOOTH FUNCTIONS
First note that if ∇f(x) is well defined everywhere and if ||∇f(x)|| ≤ L over the entire domain, it is L-Lipschitz by the mean value theorem and an application of Cauchy-Schwartz. In the convex case, the reverse is also true. Lemma D.1. Take g to be a function from Rn → Rd. If g is L-Lipschitz in each of its components, g is √ nL-Lipschitz.
Proof. Consider any x, y ∈ Rn. By the assumption we know that for i = 1, . . . , n |gi(x)−gi(y)|2 ≤ L2||x− y||22. Therefore
n∑ i=1 |gi(x)− gi(y)|2 ≤ nL2||x− y||22,
and, thus, the desired result that
||g(x)− g(y)||2 = √√√√ n∑ i=1 |gi(x)− gi(y)|2 ≤ √ nL||x− y||2.
Directly from the definition it is difficult to prove a gradient to be Lipschitz continuous. However we can utilize Lemma D.2. Lemma D.2. For some twice differentiable function f : A ⊂ Rn → R, f is β-smooth if its Hessian is bounded by β with respect to the euclidean norm.
Proof. Take the gradient ∇f which is a mapping A ⊂ Rn → Rn. We can apply the mean value inequality for vector valued functions, Theorem B.1, which gives us
||∇f(x)−∇f(y)||2 ≤ ||∇2f(z)(x− y)||2, where z is on the line segment connecting x and y. Because of how the induced euclidean norm on a matrix is defined it follows that
||∇2f(z)(x− y)||2 ≤ ||∇2f(z)||2||x− y||2 ≤ β||x− y||2. Combining the two equations above gives the desired result,
||∇f(x)−∇f(y)||2 ≤ β||x− y||2.
The insight to take this approach to demonstrating β-smoothness is from Bach & Moulines (2011).

This paper describes an approach to learning the non-linear activation function in deep neural nets.  This is achieved by representing the activation function in a basis of non-linear functions and learning the coefficients.  Authors use Fourier basis in the paper.  A theoretical analysis of the proposed approach is also presented, using algorithmic stability arguments, to demonstrate good generalization behavior (vanishing generalization error with large data sets) of networks with learnt non-linearities.
The main question I have about this paper is that writing a non-linear activation function as a linear or affine combination of other non-linear basis functions is equivalent to making a larger network whose nodes have the basis functions as non-linearities and whose weights have certain constraints on them.  Thus, the value of the proposed approach of learning non-linearities over optimizing network capacity for a given task (with fixed non-linearities) is not clear to me.  Or could it be argued that the constrained implied by learnt non-linearity approach are somehow good thing to do?
Another question - In the two stage training process for CNNs, when ReLU activation is replaced by NPFC(L,T), is the NPFC(L,T) activation initialized to approximate ReLU, or is it initialized using random coefficients?
Few minor corrections/questions:
- Pg 2. “ … the interval [-L+T, L+T] …” should be “ … the interval [-L+T, L-T] … “ ?
- Pg 2., Equation for f(x), should it be “ (-L+T) i \pi x / L “ in both sin and cos terms, or without “ x “ ?
- Theorem 4.2 “ … some algorithm \eps-uniformly stable …” remove the word “algorithm”
- Theorem 4.5.  SGM undefined