Mini-batch based Stochastic Gradient Descent(SGD) has been widely used to train deep neural networks efficiently. In this paper, we design a general framework to automatically and adaptively select training data for SGD. The framework is based on neural networks and we call it Neural Data Filter (NDF). In Neural Data Filter, the whole training process of the original neural network is monitored and supervised by a deep reinforcement network, which controls whether to filter some data in sequentially arrived mini-batches so as to maximize future accumulative reward (e.g., validation accuracy). The SGD process accompanied with NDF is able to use less data and converge faster while achieving comparable accuracy as the standard SGD trained on the full dataset. Our experiments show that NDF bootstraps SGD training for different neural network models including Multi Layer Perceptron Network and Recurrent Neural Network trained on various types of tasks including image classification and text understanding.
1 INTRODUCTION
With large amount of training data as its fuel, deep neural networks (DNN) have achieved stateof-art performances in multiple tasks. Examples include deep convolutional neural network (CNN) for image understanding (Krizhevsky et al., 2012; Ioffe & Szegedy, 2015; He et al., 2015; Ren et al., 2015) and recurrent neural networks (RNN) for natural language processing (Cho et al., 2014; Kiros et al., 2015; Dai & Le, 2015; Shang et al., 2015). To effectively train DNN with large scale of data, typically mini-batch based Stochastic Gradient Descent (SGD) (and its variants such as Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012) and Adam (Kingma & Ba, 2014)) is used. The mini-batch based SGD training is a sequential process, in which mini-batches of data D = {D1, · · ·Dt, . . . , DT } arrive sequentially in a random order. Here Dt = (d1, · · · , dM ) is the mini-batch of data arriving at the t-th time step and consisting of M training instances. After receivingDt at t-th step, the loss and gradient w.r.t. current model parametersWt are Lt = 1M l(dm) and gt = ∂Lt∂Wt , based on which the neural network model gets updated:
Wt+1 =Wt − ηtgt. (1) Here l(·) is the loss function specified by the neural network and ηt is the learning rate at t-th step. With the sequential execution of SGD training, the neural network evolves constantly from a raw state to a fairly mature state, rendering different views even for the same training data. For example, as imposed by the spirit of Curriculum Learning (CL) (Bengio et al., 2009) and Self-Paced Learning (SPL) (Kumar et al., 2010), at the baby stage of the neural network, easy examples play important roles whereas hard examples are comparatively negligible. In contrast, at the adult age, the neural ∗Works done when Yang Fan is an intern at Microsoft Research Asia.
network tends to favor harder training examples, since easy ones bring minor changes. It remains an important question that, how to optimally and dynamically allocate training data at different stages of SGD training?
A possible approach is to solve this problem in an active manner: at each time step t, the minibatch data Dt is chosen from all the left untrained data (Tsvetkov et al., 2016; Sachan & Xing, 2016). However, this typically requires a feed-forward pass over the whole remaining dataset at each training step, making it computationally expensive. We therefore consider a passive way in this paper, in which the random ordering of all the mini-batches is pre-given and maintained during the training process. What actually do is, after receiving the mini-batch Dt of M training instances, we dynamically determine which instances in Dt are used for training and which are filtered, based on the features extracted from the feedforward pass only on Dt. Acting in this way avoids unnecessary computational steps on those filtered data and thus speeds-up the training process.
Previous works such as curriculum learning (CL) and self-paced learning (SPL) can be leveraged to fulfill such a data filtration task. However, they are typically based on simple heuristic rules, such as shuffling the sequence length to train language model (Bengio et al., 2009), or abandoning training instances whose loss values are larger than a human-defined threshold (Kumar et al., 2010; Jiang et al., 2014a).
In this work, we propose a Neural Data Filter (NDF) framework from a more principled and selfadaptive view. In this framework, as illustrated in Figure 1, the SGD training for DNN is naturally casted into a Markov Decision Process (MDP) (Sutton & Barto, 1998) and data filtration strategy is fully controlled through deep reinforcement learning (Mnih et al., 2013; Lillicrap et al., 2015b; Mnih et al., 2016). In such an MDP, a state (namely s1, · · · , st, · · ·) is composed of two parts: the minibatch of data arrived and the parameters of the current neural network model, i.e, st = {Dt,Wt}. In each time step t, NDF receives a representation f(st) for current state from SGD, outputs the action at specifying which instances in Dt will be filtered according to its policy At. Afterwards, the remaining data determined by at will be used by SGD to update the neural network state and generate a reward rt (such as validation accuracy), which will be leveraged by NDF as the feedback for updating its own policy.
From another view, while SGD acts as the trainer for base model, i.e., DNN, it meanwhile is the trainee of reinforcement learning module. In other words, reinforcement learning acts at the teacher module while SGD for DNN is the student. Speaking more ambitiously, such a teacher-student framework based on reinforcement learning goes far beyond data filtration for neural network training: On one hand, the base model the can be benefitted is not limited to neural networks; on the other, the action space in reinforcement learning teacher module covers any strategies in machine learning process, such as hyper-parameter tuning and distributed scheduling. Through carefully designed interaction between the two modules, the training process of general machine learning models can be more elaborately controlled.
The rest of the paper is organized as follows: in the next section 2, we will introduce the details of Neural Data Filter (NDF), including the MDP language to model Stochastic Gradient Descent training, and the policy gradient algorithms to learn NDF. Then in section 3, the empirical results
of training LSTM RNN will be shown to verify the effectiveness of NDF. We discuss related work in subsequent section 4 and conclude the paper in the last section 5.
2 NEURAL DATA FILTER
We introduce the mathematical details of Neural Data Filter (NDF) for SGD training in this section. As a summary, NDF aims to filter certain amount of training data within a mini-batch, in order to achieve better convergence speed for SGD training. To achieve that, as introduced in last section and Figure 1, we cast Stochastic Gradient Descent training for DNN as a Markov Decision Process (MDP), termed as SGD-MDP.
SGD-MDP: As traditional MDP, SGD-MDP is composed of the tuple < s, a,P, r, γ >, illustrated as:
• s is the state, corresponding to the mini-batch data arrived and current neural network state: st = (Dt,Wt).
• a represents the actions space and for data filtration task, we have a = {am}Mm=1 ∈ {0, 1}M , where M is the batch size and am ∈ {0, 1} denotes whether to filter the mth data instance in Dt or not1. Those filtered instances will have no effects to neural network training.
• Pass′ = P (s′|s, a) is the state transition probability, determined by two factors: 1) The uniform distribution of sequentially arrived training batch data; 2) The optimization process specified by Gradient Descent principle (c.f. equation 1). The randomness comes from stochastic factors in training, such as dropout (Srivastava et al., 2014).
• r = r(s, a) is the reward, set to be any signal indicating how well the training goes, such as validation accuracy, or the lost gap for current mini-batch data before/after model update.
• Furthermore future reward r is discounted by a discounting factor γ ∈ [0, 1] into the cumulative reward.
NDF samples the action a by its policy function A = PΘ(a|s) with parameters Θ to be learnt. For example, NDF policy A can be set as logistic regression:
A(s, a; Θ) = PΘ(a|s) = aσ(θf(s) + b) + (1− a)(1− σ(θf(s) + b)), (2) where σ(x) = 1/(1 + exp(−x)) is sigmoid function, Θ = {θ, b}. f(s) is the feature vector to effectively represent state s, discussed as below.
State Features: The aim of designing state feature vector f(s) is to effectively and efficiently represent SGD-MDP state. Since state s includes both arrived training data and current neural network state, we adopt three categories features to compose f(s):
• Data features, contains information for data instance, such as its label category (we use 1 of |Y | representations), the length of sentence, or linguistic features for text segments (Tsvetkov et al., 2016). Data features are commonly used in Curriculum Learning (Bengio et al., 2009; Tsvetkov et al., 2016).
• Neural network features, include the signals reflecting how well current neural network is trained. We collect several simple features, such as passed mini-batch number (i.e., iteration), the average historical training loss and current validation accuracy. They are proven to be effective enough to represent current neural network status.
• Features to represent the combination of both data and model. By using these features, we target to represent how important the arrived training data is for current neural network. We mainly use three parts of such signals in our classification tasks: 1) the predicted probabilities of each class; 2)the cross-entropy loss, which appears frequently in Self-Paced
1We consider data instances within the same mini-batch are independent with each other, therefore for statement simplicity, when the context is clear, a will be used to denote the remain/filter decision for single data instance, i.e., a ∈ {0, 1}. Similarly, the notation s will sometimes represent the state for only one training instance.
Learning algorithms (Kumar et al., 2010; Jiang et al., 2014a; Sachan & Xing, 2016); 3) the margin value 2.
The state features f(s) are computed once each mini-batch training data arrives.
The whole process for training neural networks is listed in Algorithm 1. In particular, we take the similar generalization framework proposed in (Andrychowicz et al., 2016), in which we use part of training data to train the policy of NDF (Step 1 and 2), and apply the data filtration model to the training process on the whole dataset (Step 3). The detailed algorithm to train NDF policy will be introduced in the next subsection.
Algorithm 1 Training Neural Networks with Neural Data Filter. Input: Training Data D. 1. Sample part of NDF training data D′ from D. 2. Optimize NDF policy network A(s; Θ) (c.f. equation 2) based on D′ by policy gradient. 3. Apply A(s; Θ) to full dataset D to train neural network model by SGD. Output: The Neural Network Model.
2.1 TRAINING ALGORITHM FOR NDF POLICY
Policy gradient methods are adopted to learn NDF policy A. In particular, according to different policy gradient methods, we designed two algorithms: NDF-REINFORCE and NDF-ActorCritic.
NDF-REINFORCE. NDF-REINFORCE is based on REINFORCE algorithm (Williams, 1992), an elegant Monto-Carlo based policy gradient method which favors action with high sampled reward. The algorithm details are listed in Algorithm 2. Particularly, as indicated in equation 3, NDFREINFORCE will support data filtration policy leading to higher cumulative reward vt.
Algorithm 2 NDF-REINFORCE algorithm to train NDF policy. Input: Training data D′. Episode number L. Mini-batch size M . Discount factor γ ∈ [0, 1]. for each episode l = 1, 2, · · · , L do
Initialize the base neural network model. Shuffle D′ to get the mini-batches sequence D′ = {D1, D2, · · · , DT }. for t = 1, · · · , T do
Sample data filtration action for each data instance in Dt = {d1, · · · , dM}: a = {am}Mm=1, am ∝ A(sm, a; Θ), sm is the state corresponding to the dm
Update neural network model by Gradient Descent based on the selected data in Dt. Receive reward rt.
end for for t = 1, · · · , T do
Compute cumulative reward vt = rt + γrt+1 + · · ·+ γT−trT . Update policy parameter Θ:
Θ← Θ + αvt ∑ m ∂ logA(s, am; Θ) ∂Θ (3)
end for end for Output: The NDF policy network A(s, a; Θ).
NDF-ActorCritic.
The gradient estimator in REINFORCE poses high variance given its Monto-Carlo nature. Furthermore, it is quite inefficient to update policy network only once in each episode. We therefore design NDF-ActorCritic algorithm based on value function estimation. In NDF-ActorCritic, a parametric value function estimator Q(s, a;W ) (i.e., a critic) with parameters W for estimating state-action
2The margin for a training instance (x, y) is defined as P (y|x)−maxy′ 6=y P (y′|x) (Cortes et al., 2013)
value function is leveraged to avoid the high variance of vt from Monto-Carlo sampling in NDFREINFORCE. It remains an open and challenging question that how to define optimal value function estimator Q(s, a;W ) for SGD-MDP. Particularly in this work, as a preliminary attempt, the following function is used as the critic:
Q(s, a;W ) = σ(wT0 relu(f(s)W1a) + b), (4)
where f(s) = (f(s1); f(s2); · · · , f(sM )) is a matrix with M rows and each row f(sm) represents state features for the corresponding training instance dm. W = {w0,W1, b} is the parameter set to be learnt by Temporal-Difference algorithm. Base on such a formulation, the details of NDFActorCritic is listed in Algorithm 3.
Algorithm 3 NDF-ActorCritic algorithm to train NDF policy. Input: Training data D′. Episode number L. Mini-batch size M . Discount factor γ ∈ [0, 1]. for each episode l = 1, 2, · · · , L do
Initialize the base neural network model. Shuffle D′ to get the mini-batches sequence D′ = {D1, D2, · · · , DT }. for t = 1, · · · , T do
Sample data filtration action for each data instance in Dt = {d1, · · · , dM}: a = {am}Mm=1, am ∝ A(sm, a; Θ), sm is the state corresponding to the dm . s = {sm}Mm=1.
Update neural network model by Gradient Descent based on the selected data. Receive reward rt. Update policy(actor) parameter Θ: Θ← Θ + αQ(s, a;W ) ∑ m ∂ logA(s,am;Θ)
∂Θ . Update critic parameter W :
q = rt−1 + γQ(s, a;W )−Q(s′, a′;W ), W = W − βq ∂Q(s′, a′;W )
∂W (5)
a′ ← a, s′ ← s end for
end for Output: The NDF policy network A(s, a; Θ).
3 EXPERIMENTS
3.1 EXPERIMENTS SETUP
We conduct experiments on two different tasks/models: IMDB movie review sentiment classification (with Recurrent Neural Network) and MNIST digital image classification (with Multilayer Perceptron Network). Different data filtration strategies we applied to SGD training include:
• Unfiltered SGD. The SGD training algorithm without any data filtration. Here rather than vanilla sgd (c.f. equation 1), we use its advanced variants such as Adadelta (Zeiler, 2012) or Adam (Kingma & Ba, 2014) to each of the task.
• Self-Paced Learning (SPL) (Kumar et al., 2010). It refers to filtering training data by its ‘hardness’, as reflected by loss value. Mathematically speaking, those training data d satisfying l(d) > η will be filtered out, where the threshold η grows from smaller to larger during training process. In our implementation, to improve the robustness of SPL, following the widely used trick (Jiang et al., 2014b), we filter data using its loss rank in one mini-batch, rather than the absolute loss value. That is to say, we filter data instances with topK largest training losses within a M -sized mini-batch, where K linearly drops from M − 1 to 0 during training.
• NDF-REINFORCE. The policy trained with NDF-REINFORCE, as shown in Algorithm 2. We use a signal to indicate training speed as reward. To be concrete, we set an accuracy threshold τ ∈ [0, 1] and record the first mini-batch index iτ in which validation accuracy
exceeds τ , then the reward is set as rT = − log(τ/T ). Note here only terminal reward exists (i.e., rt = 0,∀t < T ). • NDF-ActorCritic. The policy trained with NDF-ActorCritic, as shown in Algorithm 3.
Discount factor is set as γ = 0.95. Since actor-critic algorithm makes it possible to update policy per time step, rather than per episode, different with the terminal reward set in NDF-REINFORCE, validation accuracy is used as the immediate reward for each time step. To save time cost, only part of validation set is extracted to compute validation accuracy.
• Randomly Drop. To conduct more comprehensive comparison, for NDF-REINFORCE and NDF-ActorCritic, we record the ratio of filtered data instances per epoch, and then randomly filter data in each mini-batch according to the logged ratio. In this way we form two more baselines, referred to as RandDropREINFORCE and RandDropActorCritic respectively.
For all strategies other than Plain SGD, we make sure that the base neural network model will not be updated until M un-trained, yet selected data instances are accumulated. In that way we make sure that the batch size are the same for every strategies (i.e., M ), thus convergence speed is only determined by the effectiveness of data filtration strategies, not by different batch size led by different number of filtered data. For NDF strategies, we initialize b = 2 (c.f. equation 2), with the goal of maintaining training data at the early age, and use Adam (Kingma & Ba, 2014) to optimize the policy. The model is implemented with Theano (Theano Development Team, 2016) and run on one Telsa K40 GPU.
3.2 IMDB SENTIMENT CLASSIFICATION
IMDB movie review dataset3 is a binary sentiment classification dataset consisting of 50k movie review comments with positive/negative sentiment labels (Maas et al., 2011). We apply LSTM (Hochreiter & Schmidhuber, 1997) RNN to each sentence, and the last hidden state of LSTM is fed into a logistic regression classifier to predict the sentiment label (Dai & Le, 2015). The model size (i.e., word embedding size × hidden state size) is 256× 512 and mini-batch size is set as M = 16. Adadelta (Zeiler, 2012) is used to perform LSTM model training.
The IMDB dataset contains 25k training sentences and 25k test sentences. For NDF-REINFORCE and NDF-ActorCritic, from all the training data we randomly sample 10k and 5k as the training/validation set to learn data filtration policy. For NDF-REINFORCE, the validation accuracy threshold is set as τ = 0.8. For NDF-ActorCritic, the size of sub validation set to compute immediate reward is 1k. The episode number is set as L = 30. Early stop on validation set is used to control training process in each episode.
The detailed results are shown in Figure 2, whose x-axis represents the number of effective training instances and y-axis denotes the accuracy on test dataset. All the curves are results of 5 repeated runs. From the figure we have the following observations:
• NDF (shown by the two solid lines) significantly boosts the convergence of SGD training for LSTM. With much less data, NDF achieves satisfactory classification accuracy. For example, NDF-REINFORCE achieves 80% test accuracy with only roughly half training data (about 40k) of Plain SGD consumes (about 80k). Furthermore, NDF significantly outperforms the two Randomly Drop baselines, demonstrating the effectiveness of learnt policies.
• Self-Paced Learning (shown by the red dashed line) helps for the initialization of LSTM, however, it delays training after the middle phrase.
• For the two variants of NDF, NDF-REINFORCE performs better than NDF-ActorCritic. Our conjecture for the reason is: 1) For NDF-REINFORCE, we use a terminal reward fully devoted to indicate training convergence; 2) The critic function (c.f., equation 4) may not be expressive enough to approximate true state-action value functions. Deep critic function should be the next step.
3http://ai.stanford.edu/˜amaas/data/sentiment/
To better understand the learnt policies of NDF, in Figure 3 we plot the ratio of filtered data instances per every certain number of iterations. It can be observed that more and more training data are kept during the training process, which are consistent with the intuition of Curriculum Learning and SelfPaced Learning. Furthermore, the learnt feature weights for NDF policies (i.e. θ in equation 2) are listed in Table 1. From the table, we can observe:
• Longer movie reviews, with positive sentiments are likely to be kept. • Margin plays critical value in determining the importance of data. As reflected by its fairly
large positive weights, training data with large margin is likely to be kept.
• Note that the feature − log py is the training loss, its negative weights mean that training instances with larger loss values tend to be filtered, thus more and more data will be kept since loss values get smaller and smaller during training, which is consistent with the curves
in Figure 3. However, such a trend is diminished by the negative weight values for neural network features, i.e., historical training accuracy and normalized iteration.
3.3 IMAGE CLASSIFICATION ON CORRUPTED-MNIST
We further test different data filtration strategies for multilayer perceptron network training on image recognition task. The dataset we used is MNIST, which consists of 60k training and 10k testing images of handwritten digits from 10 categories (i.e., 0, · · ·, 9). To further demonstrate the effectiveness of the proposed neural data filter in automatically choosing important instances for training, we manually corrupt the original MNIST dataset by injecting some noises to the original pictures as follows: We randomly split 60k training images into ten folds, and flip (i−1)×10% randomly chosen pixels of each image in the i-th fold, i = 1, 2, · · · , 10. The 10k test set are remained unchanged. Flipping a pixel means setting its value r as r = 1.0 − r. Such a corrupted dataset is named as C-MNIST. Some sampled images from C-MNIST are shown in Figure 4.
A three-layer feedforward neural network with size 784×300×10 is used to classify the C-MNIST dataset. For data filtration policy, different from the single-layer logistic regression in equation 2, in this task, NDF-REINFORCE and NDF-ActorCritic leverage a three-layer neural network with model size 24 × 12 × 1 as policy network, where the first layer node number 24 is the dimension of state features fs 4, and sigmoid function is used as the activation function for the middle layer. 10k randomly selected images out of 60k training set acts as validation set to provide reward signals to NDF-REINFORCE and NDF-ActorCritic. For NDF-REINFORCE, the validation accuracy threshold is set as τ = 0.90. For NDF-ActorCritic, the immediate reward is computed on the whole validation set. The episode number for policy training is set as L = 50 and we control training in each episode by early stopping based on validation set accuracy. We use Adam (Kingma & Ba, 2014) to optimize policy network.
The test set accuracy curves (averaged over five repeated runs) of different data filtration strategies are demonstrated in Figure 5. From Figure 5 we can observe:
• Similar to the result in IMDB sentiment classification, NDF-REINFORCE achieves the best convergence speed;
• The performance of NDF-ActorCritic is inferior to NDF-REINFORCE. In fact, NDFActorCritic acts similar to sgd training without any data filtration. This further shows although Actor-Critic reduces variance compared with REINFORCE, the difficulty in designing/training better critic functions hurts its performance.
4 RELATED WORK
Plenty of previous works talk about data scheduling (e.g., filtration and ordering) strategies for machine learning. A remarkable example is Curriculum Learning (CL) (Bengio et al., 2009) showing that a data order from easy instances to hard ones, a.k.a., a curriculum, benefits learning process.
4fs is similar to the features in Table 1, except that (y0, y1) and (log p0, log p1) are switched into (y0, · · · , y9) and (log p0, · · · , log p9) respectively, given there are ten target classes in mnist classification.
The measure of hardness in CL is typically determined by heuristic understandings of data (Bengio et al., 2009; Spitkovsky et al., 2010; Tsvetkov et al., 2016). As a comparison, Self-Paced Learning (SPL) (Kumar et al., 2010; Jiang et al., 2014a;b; Supancic & Ramanan, 2013) quantifies the hardness by the loss on data. In SPL, those training instances with loss values larger than a threshold η will be neglected and η gradually increases in the training process such that finally all training instances will play effects. Apparently SPL can be viewed as a data filtration strategy considered in this paper.
Recently researchers have noticed the importance of data scheduling for training Deep Neural Network models. For example, in (Loshchilov & Hutter, 2015), a simple batch selection strategy based on the loss values of training data is proposed for speed up neural networks training. (Tsvetkov et al., 2016) leverages Bayesian Optimization to optimize a curriculum function for training distributed word representations. The authors of (Sachan & Xing, 2016) investigated several handcrafted criteria for data ordering in solving Question Answering tasks based on DNN. Our works differs significantly with these works in that 1) We aim to filter data in randomly arrived mini-batches in training process to save computational efforts, rather than actively select mini-batch; 2) We leverage reinforcement learning to automatically derive the optimal policy according to the feedback of training process, rather than use naive and heuristic rules.
The proposed Neural Data Filter (NDL) for data filtration is based on deep reinforcement learning (DRL) (Mnih et al., 2013; 2016; Lillicrap et al., 2015a; Silver et al., 2016), which applies deep neural networks to reinforcement learning (Sutton & Barto, 1998). In particular, NDL belongs to policy based reinforcement learning, seeking to search directly for optimal control policy. REINFORCE (Williams, 1992) and actor-critic (Konda & Tsitsiklis, 1999) are two representative policy gradient algorithms, with the difference that actor-critic adopts value function approximation to reduce the high variance of policy gradient estimator in REINFORCE.
5 CONCLUSION
In this paper we introduce Neural Data Filter (NDF), a reinforcement learning framework to select/filter data for training deep neural network. Experiments on several deep neural networks training demonstrate that NDF boosts the convergence of Stochastic Gradient Descent. Going beyond data filtration, the proposed framework is able to supervise any sequential training process, thus opens a new view for self-adaptively tuning/controlling machine learning process.
As to future work, on one aspect, we aim to test NDF to more tasks and models, such as Convolutional Neural Network (CNN) for image classification. We would also plan to give clearer explanation on the behavior of NDF, such as what data is dropped at different phrases of training, and whether the proposed critic function is good enough. On the other aspect, we aim to apply such a reinforcement learning based teacher-student framework to other strategy design problems for machine learning, such as hyper-parameter tuning, structure learning and distributed scheduling, with the hope of providing better guidance for controlled training process.

This work proposes to augment normal gradient descent algorithms with a "Data Filter", that acts as a curriculum teacher by selecting which examples the trained target network should see to learn optimally. Such a filter is learned simultaneously to the target network, and trained via Reinforcement Learning algorithms receiving rewards based on the state of training with respect to some pseudo-validation set.
Stylistic comment, please use the more common style of "(Author, year)" rather than "Author (year)" when the Author is *not* referred to or used in the sentence.
E.g. "and its variants such as Adagrad Duchi et al. (2011)" should be "such as Adagrad (Duchi et al., 2011)", and  "proposed in Andrychowicz et al. (2016)," should remain so.
I think the paragraph containing "What we need to do is, after seeing the mini-batch Dt of M training instances, we dynamically determine which instances in Dt are used for training and which are filtered." should be clarified. What is "seeing"? That is, you should mention explicitly that you do the forward-pass first, then compute features from that, and then decide for which examples to perform the backwards pass.
There are a few choices in this work which I do not understand:
Why wait until the end of the episode to update your reinforce policy (algorithm 2), but train your actor critic at each step (algorithm 3)? You say REINFORCE has high variance, which is true, but does not mean it cannot be trained at each step (unless you have some experiments that suggest otherwise, and if so they should be included or mentionned in the paper).
Similarly, why not train REINFORCE with the same reward as your Actor-Critic model? And vice-versa? You claim several times that a limitation of REINFORCE is that you need to wait for the episode to be over, but considering your data is i.i.d., you can make your episode be anything from a single training step, one D_t, to the whole multi-epoch training procedure.
I have a few qualms with the experimental setting:
- is Figure 2 obtained from a single (i.e. one per setup) experiment? From different initial weights? If so, there is no proper way of knowing whether results are chance or not! This is a serious concern for me.
- with most state-of-the-art work using optimization methods such as Adam and RMSProp, is it surprising that they were not experimented with.
- it is not clear what the learning rates are; how fast should the RL part adapt to the SL part? Its not clear that this was experimented with at all.
- the environment, i.e. the target network being trained, is not stationnary at all. It would have been interesting to measure how much the policy changes as a function of time. Figure 3, could both be the result of the policy adapting, or of the policy remaining fixed and the features changing (which could indicate a failure of the policy to adapt).
- in fact it is not really adressed in the paper that the environment is non-stationary, given the current setup, the distribution of features will change as the target network progresses. This has an impact on optimization.
- how is the "pseudo-validation" data, target to the policy, chosen? It should be a subset of the training data. The second paragraph of section 3.2 suggests something of the sort, but then your algorithms suggest that the same data is used to train both the policies and the networks, so I am unsure of which is what.
Overall the idea is novel and interesting, the paper is well written for the most part, but the methodology has some flaws. Clearer explanations and either more justification of the experimental choices or more experiments are needed to make this paper complete. Unless the authors convince me otherwise, I think it would be worth waiting for more experiments and submitting a very strong paper rather than presenting this (potentially powerful!) idea with weak results.


LEARNING PYTHON CODE SUGGESTION WITH A SPARSE POINTER NETWORK
To enhance developer productivity, all modern integrated development environments (IDEs) include code suggestion functionality that proposes likely next tokens at the cursor. While current IDEs work well for statically-typed languages, their reliance on type annotations means that they do not provide the same level of support for dynamic programming languages as for statically-typed languages. Moreover, suggestion engines in modern IDEs do not propose expressions or multi-statement idiomatic code. Recent work has shown that language models can improve code suggestion systems by learning from software repositories. This paper introduces a neural language model with a sparse pointer network aimed at capturing very longrange dependencies. We release a large-scale code suggestion corpus of 41M lines of Python code crawled from GitHub. On this corpus, we found standard neural language models to perform well at suggesting local phenomena, but struggle to refer to identifiers that are introduced many tokens in the past. By augmenting a neural language model with a pointer network specialized in referring to predefined classes of identifiers, we obtain a much lower perplexity and a 5 percentage points increase in accuracy for code suggestion compared to an LSTM baseline. In fact, this increase in code suggestion accuracy is due to a 13 times more accurate prediction of identifiers. Furthermore, a qualitative analysis shows this model indeed captures interesting long-range dependencies, like referring to a class member defined over 60 tokens in the past.
1 INTRODUCTION
Integrated development environments (IDEs) are essential tools for programmers. Especially when a developer is new to a codebase, one of their most useful features is code suggestion: given a piece of code as context, suggest a likely sequence of next tokens. Typically, the IDE suggests an identifier or a function call, including API calls. While extensive support exists for statically-typed languages such as Java, code suggestion for dynamic languages like Python is harder and less well supported because of the lack of type annotations. Moreover, suggestion engines in modern IDEs do not propose expressions or multi-statement idiomatic code.
Recently, methods from statistical natural language processing (NLP) have been used to train code suggestion systems from code usage in large code repositories (Hindle et al., 2012; Allamanis & Sutton, 2013; Tu et al., 2014). To this end, usually an n-gram language model is trained to score possible completions. Neural language models for code suggestion (White et al., 2015; Das & Shah, 2015) have extended this line of work to capture more long-range dependencies. Yet, these standard neural language models are limited by the so-called hidden state bottleneck, i.e., all context information has to be stored in a fixed-dimensional internal vector representation. This limitation restricts such models to local phenomena and does not capture very long-range semantic relationships like suggesting calling a function that has been defined many tokens before.
To address these issues, we create a large corpus of 41M lines of Python code by using a heuristic for crawling high-quality code repositories from GitHub. We investigate, for the first time, the use of attention (Bahdanau et al., 2014) for code suggestion and find that, despite a substantial improvement
in accuracy, it still makes avoidable mistakes. Hence, we introduce a model that leverages long-range Python dependencies by selectively attending over the introduction of identifiers as determined by examining the Abstract Syntax Tree. The model is a form of pointer network (Vinyals et al., 2015a), and learns to dynamically choose between syntax-aware pointing for modeling long-range dependencies and free form generation to deal with local phenomena, based on the current context.
Our contributions are threefold: (i) We release a code suggestion corpus of 41M lines of Python code crawled from GitHub, (ii) We introduce a sparse attention mechanism that captures very long-range dependencies for code suggestion of this dynamic programming language efficiently, and (iii) We provide a qualitative analysis demonstrating that this model is indeed able to learn such long-range dependencies.
2 METHODS
We first revisit neural language models, before briefly describing how to extend such a language model with an attention mechanism. Then we introduce a sparse attention mechanism for a pointer network that can exploit the Python abstract syntax tree of the current context for code suggestion.
2.1 NEURAL LANGUAGE MODEL
Code suggestion can be approached by a language model that measures the probability of observing a sequence of tokens in a Python program. For example, for the sequence S = a1, . . . , aN , the joint probability of S factorizes according to
Pθ(S) = Pθ(a1) · N∏ t=2 Pθ(at | at−1, . . . , a1) (1)
where the parameters θ are estimated from a training corpus. Given a sequence of Python tokens, we seek to predict the next M tokens at+1, . . . , at+M that maximize Equation 1
argmax at+1, ..., at+M Pθ(a1, . . . , at, at+1, . . . , at+M ). (2)
In this work, we build upon neural language models using Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM, Hochreiter & Schmidhuber, 1997). This neural language model estimates the probabilities in Equation 1 using the output vector of an LSTM at time step t (denoted ht here) according to
Pθ(at = τ | at−1, . . . , a1) = exp (vTτ ht + bτ )∑ τ ′ exp (v T τ ′ht + bτ ′)
(3)
where vτ is a parameter vector associated with token τ in the vocabulary.
Neural language models can, in theory, capture long-term dependencies in token sequences through their internal memory. However, as this internal memory has fixed dimension and can be updated at every time step, such models often only capture local phenomena. In contrast, we are interested in very long-range dependencies like referring to a function identifier introduced many tokens in the past. For example, a function identifier may be introduced at the top of a file and only used near the bottom. In the following, we investigate various external memory architectures for neural code suggestion.
2.2 ATTENTION
A straight-forward approach to capturing long-range dependencies is to use a neural attention mechanism (Bahdanau et al., 2014) on the previous K output vectors of the language model. Attention mechanisms have been successfully applied to sequence-to-sequence tasks such as machine translation (Bahdanau et al., 2014), question-answering (Hermann et al., 2015), syntactic parsing (Vinyals et al., 2015b), as well as dual-sequence modeling like recognizing textual entailment (Rocktäschel et al., 2016). The idea is to overcome the hidden-state bottleneck by allowing referral back to previous output vectors. Recently, these mechanisms were applied to language modelling by Cheng et al. (2016) and Tran et al. (2016).
Formally, an attention mechanism with a fixed memory Mt ∈ Rk×K of K vectors mi ∈ Rk for i ∈ [1,K], produces an attention distribution αt ∈ RK and context vector ct ∈ Rk at each time step t according to Equations 4 to 7. Furthermore, WM ,W h ∈ Rk×k and w ∈ Rk are trainable parameters. Finally, note that 1K represents a K-dimensional vector of ones.
Mt = [m1 . . . mK ] ∈ Rk×K (4) Gt = tanh(WMMt + 1TK(W
hht)) ∈ Rk×K (5) αt = softmax(wTGt) ∈ R1×K (6) ct =Mtα T t ∈ Rk (7)
For language modeling, we populate Mt with a fixed window of the previous K LSTM output vectors. To obtain a distribution over the next token we combine the context vector ct of the attention mechanism with the output vector ht of the LSTM using a trainable projection matrixWA ∈ Rk×2k. The resulting final output vector nt ∈ Rk encodes the next-word distribution and is projected to the size of the vocabulary |V |. Subsequently, we apply a softmax to arrive at a probability distribution yt ∈ R|V | over the next token. This process is presented in Equation 9 where WV ∈ R|V |×k and bV ∈ R|V | are trainable parameters.
nt = tanh ( WA [ ht ct ]) ∈ Rk (8)
yt = softmax(W V nt + b V ) ∈ R|V | (9)
The problem of the attention mechanism above is that it quickly becomes computationally expensive for large K. Moreover, attending over many memories can make training hard as a lot of noise is introduced in early stages of optimization where the LSTM outputs (and thus the memoryMt) are more or less random. To alleviate these problems we now turn to pointer networks and a simple heuristic for populating Mt that permits the efficient retrieval of identifiers in a large history of Python code.
2.3 SPARSE POINTER NETWORK
We develop an attention mechanism that provides a filtered view of a large history of Python tokens. At any given time step, the memory consists of context representations of the previous K identifiers introduced in the history. This allows us to model long-range dependencies found in identifier usage. For instance, a class identifier may be declared hundreds of lines of code before it is used. Given a history of Python tokens, we obtain a next-word distribution from a weighed average of the sparse pointer network for identifier reference and a standard neural language model. The weighting of the two is determined by a controller.
Formally, at time-step t, the sparse pointer network operates on a memoryMt ∈ Rk×K of only the K previous identifier representations (e.g. function identifiers, class identifiers and so on). In addition, we maintain a vectormt = [id1, . . . , idK ] ∈ NK of symbol ids for these identifier representations (i.e. pointers into the large global vocabulary).
As before, we calculate a context vector ct using the attention mechanism (Equation 7), but on a memoryMt only containing representations of identifiers that were declared in the history. Next, we obtain a pseudo-sparse distribution over the global vocabulary from
st[i] = { αt[j] ifmt[j] = i −C otherwise (10)
it = softmax(st) ∈ R|V | (11)
where−C is a large negative constant (e.g. −1000). In addition, we calculate a next-word distribution from a standard neural language model
yt = softmax(W V ht + bV ) ∈ R|V | (12)
and we use a controller to calculate a distribution λt ∈ R2 over the language model and pointer network for the final weighted next-word distribution y∗t via
hλt = [ ht xt ct ] ∈ R3k (13)
λt = softmax(W λhλt + b λ) ∈ R2 (14) y∗t = [yt it]λt ∈ R|V | (15)
Here, xt is the representation of the input token, and W λ ∈ R2×3k and bλ ∈ R2 a trainable weight matrix and bias respectively. This controller is conditioned on the input, output and context representations. This means for deciding whether to refer to an identifier or generate from the global vocabulary, the controller has access to information from the encoded next-word distribution ht of the standard neural language model, as well as the attention-weighted identifier representations ct from the current history.
Figure 1 overviews this process. In it, the identifier base_path appears twice, once as an argument to a function and once as a member of a class (denoted by *). Each appearance has a different id in the vocabulary and obtains a different probability from the model. In the example, the model correctly chooses to refer to the member of the class instead of the out-of-scope function argument, although, from a user point-of-view, the suggestion would be the same in both cases.
3 LARGE-SCALE PYTHON CORPUS
Previous work on code suggestion either focused on statically-typed languages (particularly Java) or trained on very small corpora. Thus, we decided to collect a new large-scale corpus of the dynamic programming language Python. According to the programming language popularity website Pypl (Carbonnelle, 2016), Python is the second most popular language after Java. It is also the 3rd most common language in terms of number of repositories on the open-source code repository GitHub, after JavaScript and Java (Zapponi, 2016).
We collected a corpus of 41M lines of Python code from GitHub projects. Ideally, we would like this corpus to only contain high-quality Python code, as our language model learns to suggest code from how users write code. However, it is difficult to automatically assess what constitutes high-quality code. Thus, we resort to the heuristic that popular code projects tend to be of good quality, There are
two metrics on GitHub that we can use for this purpose, namely stars (similar to bookmarks) and forks (copies of a repository that allow users to freely experiment with changes without affecting the original repository). Similar to Allamanis & Sutton (2013) and Allamanis et al. (2014), we select Python projects with more than 100 stars, sort by the number of forks descending, and take the top 1000 projects. We then removed projects that did not compile with Python3, leaving us with 949 projects. We split the corpus on the project level into train, dev, and test. Table 1 presents the corpus statistics.
3.1 NORMALIZATION OF IDENTIFIERS
Unsurprisingly, the long tail of words in the vocabulary consists of rare identifiers. To improve generalization, we normalize identifiers before feeding the resulting token stream to our models. That is, we replace every identifier name with an anonymous identifier indicating the identifier group (class, variable, argument, attribute or function) concatenated with a random number that makes the identifier unique in its scope. Note that we only replace novel identifiers defined within a file. Identifier references to external APIs and libraries are left untouched. Consistent with previous corpus creation for code suggestion (e.g. Khanh Dam et al., 2016; White et al., 2015), we replace numerical constant tokens with $NUM$, remove comments, reformat the code, and replace tokens appearing less than five times with an $OOV$ (out of vocabulary) token.
4 EXPERIMENTS
Although previous work by White et al. (2015) already established that a simple neural language model outperforms an n-gram model for code suggestion, we include a number of n-gram baselines to confirm this observation. Specifically, we use n-gram models for n ∈ {3, 4, 5, 6} with Modified Kneser-Ney smoothing (Kneser & Ney, 1995) from the Kyoto Language Modelling Toolkit (Neubig, 2012).
We train the sparse pointer network using mini-batch SGD with a batch size of 30 and truncated backpropagation through time (Werbos, 1990) with a history of 20 identifier representations. We use
an initial learning rate of 0.7 and decay it by 0.9 after every epoch. As additional baselines, we test a neural language model with LSTM units with and without attention. For the attention language models, we experiment with a fixed-window attention memory of the previous 20 and 50 tokens respectively, and a batch size of 75. We found during testing that the baseline models performed worse with the same batch size as the sparse pointer network of 30. We therefore chose to report the stronger results obtained with a batch size of 75.
All neural language models were developed in TensorFlow (Abadi et al., 2016) and trained using cross-entropy loss. While processing a Python source code file, the last recurrent state of the RNN is fed as the initial state of the subsequent sequence of the same file and reset between files. All models use an input and hidden size of 200, an LSTM forget gate bias of 1 (Jozefowicz et al., 2015), gradient norm clipping of 5 (Pascanu et al., 2013), and randomly initialized parameters in the interval (−0.05, 0.05). As regularizer, we use a dropout of 0.1 on the input representations. Furthermore, we use a sampled softmax (Jean et al., 2015) with a log-uniform sampling distribution and a sample size of 1000.
5 RESULTS
We evaluate all models using perplexity (PP), as well as accuracy of the top prediction (Acc) and the top five predictions (Acc@5). The results are summarized in Table 2.
We can confirm that for code suggestion neural models outperform n-gram language models by a large margin. Furthermore, adding attention improves the results substantially (2.3 lower perplexity and 3.4 percentage points increased accuracy). Interestingly, this increase can be attributed to a superior prediction of identifiers, which increased from an accuracy of 2.1% to 21.4%. An LSTM with an attention window of 50 gives us the best accuracy for the top prediction. We achieve further improvements for perplexity and accuracy of the top five predictions by using a sparse pointer network that uses a smaller memory of the past 20 identifier representations.
5.1 QUALITATIVE ANALYSIS
Figures 3a-d show a code suggestion example involving an identifier usage. While the LSTM baseline is uncertain about the next token, we get a sensible prediction by using attention or the sparse pointer network. The sparse pointer network provides more reasonable alternative suggestions beyond the correct top suggestion.
Figures 3e-h show the use-case referring to a class attribute declared 67 tokens in the past. Only the Sparse Pointer Network makes a good suggestion. Furthermore, the attention weights in 3i demonstrate that this model distinguished attributes from other groups of identifiers. We give a full example of a token-by-token suggestion of the Sparse Pointer Network in Figure 4 in the Appendix.
(a) Code snippet for referencing variable.
(b) LSTM Model. (c) LSTM w/ Attention 50. (d) Sparse Pointer Network.
6 RELATED WORK
Previous code suggestion work using methods from statistical NLP has mostly focused on n-gram models. Much of this work is inspired by Hindle et al. (2012) who argued that real programs fall into a much smaller space than the flexibility of programming languages allows. They were able to capture the repetitiveness and predictable statistical properties of real programs using language models. Subsequently, Tu et al. (2014) improved upon Hindle et al.’s work by adding a cache mechanism that allowed them to exploit locality stemming from the specialisation and decoupling of program modules. Tu et al.’s idea of adding a cache mechanism to the language model is specifically designed to exploit the properties of source code, and thus follows the same aim as the sparse attention mechanism introduced in this paper.
While the majority of preceding work trained on small corpora, Allamanis & Sutton (2013) created a corpus of 352M lines of Java code which they analysed with n-gram language models. The size of the corpus allowed them to train a single language model that was effective across multiple different project domains. White et al. (2015) later demonstrated that neural language models outperform n-gram models for code suggestion. They compared various n-gram models (up to nine grams), including Tu et al.’s cache model, with a basic RNN neural language model. Khanh Dam et al. (2016) compared White et al.’s basic RNN with LSTMs and found that the latter are better at code suggestion due to their improved ability to learn long-range dependencies found in source code. Our paper extends this line of work by introducing a sparse attention model that captures even longer dependencies.
The combination of lagged attention mechanisms with language modelling is inspired by Cheng et al. (2016) who equipped LSTM cells with a fixed-length memory tape rather than a single memory cell. They achieved promising results on the standard Penn Treebank benchmark corpus (Marcus et al., 1993). Similarly, Tran et al. added a memory block to LSTMs for language modelling of English, German and Italian and outperformed both n-gram and neural language models. Their memory encompasses representations of all possible words in the vocabulary rather than providing a sparse view as we do. Attention mechanisms were previously applied to the study of source code by Allamanis et al. who used a convolutional neural network combined with an attention mechanism to generate method names from bodies.
An alternative to our purely lexical approach to code suggestion involves the use of probabilistic context-free grammars (PCFGs) which exploit the formal grammar specifications and well-defined, deterministic parsers available for source code. These were used by Allamanis & Sutton (2014) to extract idiomatic patterns from source code. A weakness of PCFGs is their inability to model context-dependent rules of programming languages such as that variables need to be declared before being used. Maddison & Tarlow (2014) added context-aware variables to their PCFG model in order to capture such rules.
Ling et al. (2016) recently used a pointer network to generate code from natural language descriptions. Our use of a controller for deciding whether to generate from a language model or copy an identifier using a sparse pointer network is inspired by their latent code predictor. However, their inputs (textual descriptions) are short whereas code suggestion requires capturing very long-range dependencies that we addressed by a filtered view on the memory of previous identifier representations.
7 CONCLUSIONS AND FUTURE WORK
In this paper, we investigated neural language models for code suggestion of the dynamically-typed programming language Python. We released a corpus of 41M lines of Python crawled from GitHub and compared n-gram, standard neural language models, and attention. By using attention, we observed an order of magnitude more accurate prediction of identifiers. Furthermore, we proposed a sparse pointer network that can efficiently capture long-range dependencies by only operating on a filtered view of a memory of previous identifier representations. This model achieves the lowest perplexity and best accuracy among the top five predictions. The Python corpus and the code for our models is released at https://github.com/uclmr/pycodesuggest.
The presented methods were only tested for code suggestion within the same Python file. We are interested in scaling the approach to the level of entire code projects and collections thereof, as well as integrating a trained code suggestion model into an existing IDE. Furthermore, we plan to work on code completion, i.e., models that provide a likely continuation of a partial token, using character language models (Graves, 2013).
ACKNOWLEDGMENTS
This work was supported by Microsoft Research through its PhD Scholarship Programme, an Allen Distinguished Investigator Award, and a Marie Curie Career Integration Award.

This paper presents an improved neural language models designed for selected long-term dependency, i.e., to predict more accurately the next identifier for the dynamic programming language such as Python. The improvements are obtained by:
1) replacing the fixed-widow attention with a pointer network, in which the memory only consists of context representation of the previous K identifies introduced for the entire history.
2) a conventional neural LSTM-based language model is combined with such a sparse pointer network with a controller, which linearly combines the prediction of both components using a dynamic weights, decided by the input, hidden state, and the context representations at the time stamp.
Such a model avoids the the need of large window size of the attention to predict next identifier, which usually requires a long-term dependency in the programming language. This is partly validated by the python codebase (which is another contribution of this paper) experiments in the paper.
While the paper still misses some critical information that I would like to see, including how the sparse pointer network performance chances with different size of K, and how computationally efficient it is for both training and inference time compared to LSTM w/ attention of various window size, and ablation experiments about how much (1) and (2) contribute respectively, it might be of interest to the ICLR community to see it accepted.


We provide a theoretical explanation for the great performance of ResNet via the study of deep linear networks and some nonlinear variants. We show that with or without nonlinearities, by adding shortcuts that have depth two, the condition number of the Hessian of the loss function at the zero initial point is depthinvariant, which makes training very deep models no more difficult than shallow ones. Shortcuts of higher depth result in an extremely flat (high-order) stationary point initially, from which the optimization algorithm is hard to escape. The 1shortcut, however, is essentially equivalent to no shortcuts. Extensive experiments are provided accompanying our theoretical results. We show that initializing the network to small weights with 2-shortcuts achieves significantly better results than random Gaussian (Xavier) initialization, orthogonal initialization, and shortcuts of deeper depth, from various perspectives ranging from final loss, learning dynamics and stability, to the behavior of the Hessian along the learning process.
1 INTRODUCTION
Residual network (ResNet) was first proposed in He et al. (2015a) and extended in He et al. (2016). It followed a principled approach to add shortcut connections every two layers to a VGG-style network (Simonyan & Zisserman, 2014). The new network becomes easier to train, and achieves both lower training and test errors. Using the new structure, He et al. (2015a) managed to train a network with 1001 layers, which was virtually impossible before. Unlike Highway Network (Srivastava et al., 2015a;b) which not only has shortcut paths but also borrows the idea of gates from LSTM (Sainath et al., 2013), ResNet does not have gates. Later He et al. (2016) found that by keeping a clean shortcut path, residual networks will perform even better.
Many attempts have been made to improve ResNet to a further extent. “ResNet in ResNet” (Targ et al., 2016) adds more convolution layers and data paths to each layer, making it capable of representing several types of residual units. “ResNets of ResNets” (Zhang et al., 2016) construct multilevel shortcut connections, which means there exist shortcuts that skip multiple residual units. Wide Residual Networks (Zagoruyko & Komodakis, 2016) makes the residual network shorter but wider, and achieves state of the art results on several datasets while using a shallower network. Moreover, some existing models are also reported to be improved by shortcut connections, including Inceptionv4 (Szegedy et al., 2016), in which shortcut connections make the deep network easier to train.
Why are residual networks so easy to train? He et al. (2015a) suggests that layers in residual networks are learning residual mappings, making them easier to represent identity mappings, which prevents the networks from degradation when the depths of the networks increase. However, Veit et al. (2016) claims that ResNets are actually ensembles of shallow networks, which means they do not solve the problem of training deep networks completely.
We propose a theoretical explanation for the great performance of ResNet. We concur with He et al. (2015a) that the key contribution of ResNet should be some special structure of the loss function that makes training very deep models no more difficult than shallow ones. Analysis, however, seems non-trivial. Quoting He et al. (2015a):
“But if F has only a single layer, Eqn.(1) is similar to a linear layer: y = W1x + x, for which we have not observed advantages. ”
“Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs. ”
Their empirical observations are inspiring. First, the 1-shortcuts mentioned in the first paragraph do not work. Second, noting that the non-bottleneck ResNets have 2-shortcuts, but the bottleneck ResNets use 3-shortcuts, one sees that shortcuts with depth three also do not work. Hence, a reasonable theoretical explanation must be able to distinguish the 2-shortcut from shortcuts of other depths, and clearly demonstrate why the 2-shortcuts are special and are able to ease the optimization process so significantly for deep models, while shortcuts of other depths may not do the job.
FIXAiming at explaining the performance of 2-shortcuts, we need to eliminate other variables that maycontribute to the success of ResNet. Indeed, one may argue that the deep structure of ResNet may give it better representation power (lower approximation error), which contributes to lower training errors. To eliminate this effect, we focus on deep linear networks, where deeper models do not have better approximation properties. The special role of 2-shortcuts naturally arises in the study.
2 MAIN RESULTS
Our work reveals that non-degenerate depth-invariant initial condition numbers, a unique property of residual networks with 2-shortcuts, contributed to the success of ResNet. In fact, in a linear network that will be defined rigorously later, the condition number of Hessian of the Frobenius loss function at the zero initial point is
cond(H) = √ cond((ΣXX − ΣY X)T (ΣXX − ΣY X)), (1)
which is independent of the number of layers. Here ΣXX and ΣY X denote the input-input and the output-input correlation matrices, defined in Section 3.3. The condition number of a possibly non-PSD matrix is defined as: Definition 1. The condition number of a matrix A is defined as
cond(A) = σmax(A)
σmin(A) , (2)
where σmax(A) and σmin(A) are the maximum and minimum of singular values of A. In particular, if A is normal, i.e. ATA = AAT , the definition can be simplified to 1
cond(A) = |λ(A)|max |λ(A)|min , (3)
where |λ(A)|max and |λ(A)|min are the maximum and minimum of the absolute values of eigenvalues of A.
Moreover, the zero initial point for ResNet with 2-shortcuts is in fact a so-called strict saddle point (Ge et al., 2015), which are proved to be easy to escape from.
Why shortcuts of other depths do not work? We show that the Hessian at the zero initial point for the 1-shortcut ResNet has condition number growing unboundedly for deep nets. As is well known in convex optimization theory, large condition numbers can have enormous adversarial impact on the convergence of first order methods (Nemirovski, 2005). Hence, it is quite clear that starting training at a point with a huge condition number would make the algorithm very difficult to escape from the initial point, making 1-shortcut ResNet no better than conventional approaches.
1The equivalence of Equation (2) and Equation (3) can be proved easily using the eigenvalue decomposition ofA. Note that as Hessians are symmetric (if all the second derivatives are continuous), we will use Equation (3) for their condition numbers. As the |λ|min of Hessian is usually very unstable, we calculated |λ|max|λ|(0.1) to represent condition numbers instead, where |λ|(0.1) is the 10th percentile of the absolute values of eigenvalues. Note that the Hessian at zero initial point for 2-shortcut ResNet also has a nice structure of spectrum: see Theorem 1 for details.
For shortcuts with depth deeper than two, the Hessian at the zero initial point is a zero matrix, making it a higher-order stationary point. Intuitively, the higher order the stationary point is, the harder it is to escape from it. Indeed, it is supported both in theory (Anandkumar & Ge, 2016) and by our experiments.
One may still ask: why are we interested in the Hessian at the zero initial point? It is because in order for the outputs of deep neural networks not explode, the singular values of the mapping of each layer are not supposed to be deviating too much from one. Indeed, it is because it is extremely challenging to keep ∏num of layers i=1 λi from exploding or vanishing without keeping all of the λi having unit norm. However, by design ResNet with shortcuts already have an identity mapping every few layers, which forces the mappings inside the shortcuts to have small operator norms. Hence, analyzing the network at zero initial point gives a decent characterization of the searching environment of the optimization algorithm.
FIXOur results also shows that the form of Hessian is more important than the existance of nonlinearitieswhen training the networks. The behaviors of the networks we studied are consistent across both linear and nonlinear structures, where networks with clearer Hessians are much easier to achieve lower training errors.
On the other hand, our experiments reveal that orthogonal initialization (Saxe et al., 2013) is suboptimal. Although better than Xavier initialization (Glorot & Bengio, 2010), the initial condition numbers of the networks still explode as the networks become deeper, which means the networks are still initialized on “bad” submanifolds that are hard to optimize using gradient descent.
3 MODEL
3.1 DEEP LINEAR NETWORKS
Deep linear networks are feed-forward neural networks that only contain linear units, which means their input-output mappings are simply linear transformations. Apparently, increasing their depths will not affect the representational power of the networks. However, linear networks with depth deeper than one show nonlinear dynamics of training (Saxe et al., 2013). As a result, analyzing the training of deep linear networks gives us a better understanding of the training of non-linear networks.
Much theoretical work has been done on deep linear networks. Kawaguchi (2016) extended the work of Choromanska et al. (2015a;b) and proved that with few assumptions, every local minimum point in deep linear networks is a global minimum point. This means that the difficulties in the training of deep linear networks mostly come from saddle points on the loss surfaces, which are also the main causes of slow learning in nonlinear networks (Pascanu et al., 2014).
Saxe et al. (2013) studied the dynamics of training using gradient descent. They found that for a special class of initial conditions, which could be obtained from greedy layerwise pre-training, the training time for a deep linear network with an infinity depth can still be finite. Furthermore, they found that by setting the initial weights to random orthogonal matrices (produced by performing QR or SVD decompositions on random Gaussian matrices), the network will still have a depth independent learning time. They argue that it is caused by the eigenvalue and singular value spectra of the end-to-end linear transformation. When using orthogonal initialization, the overall transformation is an orthogonal matrix, which has all the singular values equal to 1. In the meantime, when using scaled Gaussian initialization, most of the singular values are close to zero, making the network unsuitable for backpropagating errors. However, this explanation is not sufficient to prove that the training difficulty of orthogonal initialized networks is depth-invariant. It only gives us an intuition on why orthogonal initialization performs better than scaled Gaussian initialization.
Thus, we use deep linear networks to study the effect of shortcut connections. After adding the shortcuts, the overall model is still linear and the global minimum does not change.
3.2 NETWORK STRUCTURE
We first generalize a linear network by adding shortcuts to it to make it a linear residual network. We organize the network into R residual units. The r-th residual unit consists of Lr layers whose
weights are W r,1, . . . ,W r,Lr−1, denoted as the transformation path, as well as a shortcut Sr connecting from the first layer to the last one, denoted as the shortcut path. The input-output mapping can be written as
y = R∏ r=1 ( Lr−1∏ l=1 W r,l + Sr)x = Wx, (4)
where x ∈ Rdx , y ∈ Rdy ,W ∈ Rdy×dx . Here if b ≥ a, ∏b i=aW
i denotes W bW (b−1) · · ·W (a+1)W a, otherwise it denotes an identity mapping. The matrix W represents the combination of all the linear transformations in the network. Note that by setting all the shortcuts to zeros, the network will go back to a ( ∑ r(Lr − 1) + 1)-layer plain linear network.
Instead of analyzing the general form, we concentrate on a special kind of linear residual networks, where all the residual units are the same.
Definition 2. A linear residual network is called an n-shortcut linear network if
1. its layers have the same dimension (so that dx = dy);
2. its shortcuts are identity matrices;
3. its shortcuts have the same depth n.
The input-output mapping for such a network becomes
y = R∏ r=1 ( n∏ l=1 W r,l + Idx)x = Wx, (5)
where W r,l ∈ Rdx×dx .
Then we add some activation functions to the networks. We concentrate on the case where activation functions are on the transformation paths, which is also the case in the latest ResNet (He et al., 2016).
Definition 3. An n-shortcut linear network becomes an n-shortcut network if element-wise activation functions σpre(x), σmid(x), σpost(x) are added at the transformation paths, where on a transformation path, σpre(x) is added before the first weight matrix, σmid(x) is added between two weight matrixes and σpost(x) is added after the last weight matrix.
Note that n-shortcut linear networks are special cases of n-shortcut networks, where all the activation functions are identity mappings.
3.3 OPTIMIZATION
We denote the collection of all the variable weight parameters in an n-shortcut linear network as w. Consider m training samples {xµ, yµ}, µ = 1, . . . ,m. Using Frobenius loss, for an n-shortcut linear network, we define the loss function as follows:
L(w) = 1
2m m∑ µ=1 ‖yµ −Wxµ‖22 = 1 2m ‖Y −WX‖2F , (6)
where xµ, yµ are the µ-th columns of X,Y , and ‖·‖F denotes the Frobenius norm. Using gradient descent with learning rate α, we have the weights updating rules as
∆W r,l = α(W rafterW r,l after) T (ΣY X −WΣXX)(W r,lbeforeW r before) T , (7)
where ΣXX and ΣY X denote the input-input and the output-input correlation matrices, defined as
ΣXX = 1
m m∑ µ=1 xµ(xµ)T (8)
ΣY X = 1
m m∑ µ=1 yµ(xµ)T . (9)
Here W rbefore,W r after denote the linear mappings before and after the r-th residual unit, W r,lbefore,W r,l after denote the linear mappings before and after W
r,l within the transformation path of the r-th residual unit. In other words, the overall transformation can be represented as
y = W rafter(W r,l afterW r,lW r,lbefore + Idx)W r beforex. (10)
4 THEORETICAL STUDY
4.1 INITIAL POINT PROPERTIES
Before we analyze the initial point properties of n-shortcut networks, we have to choose the way to initialize them.
NEW ResNet uses MSRA initialization (He et al., 2015b). It is a kind of scaled Gaussian
initialization that tries to keep the variances of signals along a transformation path, which is also the idea behind Xavier initialization (Glorot & Bengio, 2010). However, because of the shortcut paths, the output variance of the entire network will actually explode as the network becomes deeper. Batch normalization units partly solved this problem in ResNet, but still they cannot prevent the large output variance in a deep network.
NEWA simple idea is to zero initialize all the weights, so that the output variances of residual unitsstay the same along the network. It is worth noting that as found in He et al. (2015a), the deeper ResNet has smaller magnitudes of layer responses. This phenomenon has been confirmed in our experiments. As illustrated in Figure 2 and Figure 3, the deeper a residual network is, the small its average Frobenius norm of weight matrixes is, both during the traning process and when the training ends. Also, Hardt & Ma (2016) proves that if all the weight matrixes have small norms, a linear residual network will have no critical points other than the global optimum.
NEWAll these evidences indicate that zero is spacial in a residual network: as the network becomesdeeper, the training tends to end up around it. Thus, we are looking into the Hessian at zero. As the zero is a saddle point, in our experiments we use zero initialization with small random perturbations to escape from it. We first Xavier initialize the weight matrixes, and then multiply a small constant (0,01) to them.
We begin with the definition of k-th order stationary point.
Definition 4. Suppose function f(x) admits k-th order Taylor expansion at point x0. We say that the point x0 is a k-th order stationary point of f(x) if the corresponding k-th order Taylor expansion of f(x) at x = x0 is a constant: f(x) = f(x0) + o(‖x− x0‖k2).
Now we state our main theorem, whose proof can be found in Appendix A.
Theorem 1. Assume that σmid(0) = σpost(0) = 0 and all of σ (k) pre(0), σ (k) mid(0), σ (k) post(0), 1 ≤ k ≤ max(n− 1, 2) exist. For the loss function of an n-shortcut network, at point zero,
1. if n ≥ 2, it is an (n− 1)th-order stationary point. In particular, if n ≥ 3, the Hessian is a zero matrix;
2. if n = 2, the Hessian can be written as
H =  0 AT A 0 0 AT
A 0 . . .
 , (11)
whose condition number is cond(H) = √ cond((ΣXσpre(X) − ΣY σpre(X))T (ΣXσpre(X) − ΣY σpre(X))), (12)
where A only depends on the training set and the activation functions. Except for degenerate cases, it is a strict saddle point (Ge et al., 2015).
3. if n = 1, the Hessian can be written as
H =  B AT AT · · · AT A B AT · · · AT A A B ...
... ... . . . AT A A · · · A B  (13) where A,B only depend on the training set and the activation functions.
Theorem 1 shows that the condition numbers of 2-shortcut networks are depth-invariant with a nice structure of eigenvalues. Indeed, the eigenvalues of the Hessian H at the zero initial point are multiple copies of ± √ eigs(ATA), and the number of copies is equal to the number of shortcut connections.
The Hessian at zero initial point for the 1-shortcut linear network follows block Toeplitz structure, which has been well studied in the literature. In particular, its condition number tends to explode as the number of layers increase (Gray, 2006).
The assumptions hold for most activation functions including tanh, symmetric sigmoid and ReLU (Nair & Hinton, 2010). Note that although ReLU does not have derivatives at zero, one may do a local polynomial approximation to yield σ(k), 1 ≤ k ≤ max(n− 1, 2). To get intuitive explanations of the theorems, imagine changing parameters in an n-shortcut network. One has to change at least n parameters to make any difference in the loss. So zero is an (n− 1)thorder stationary point. Notice that the higher the order of a stationary point, the more difficult for a first order method to escape from it.
On the other hand, if n = 2, one will have to change two parameters in the same residual unit but different weight matrices to affect the loss, leading to a clear block diagonal Hessian.
4.2 LEARNING DYNAMICS
To understand Equation (7) better, we can take n-shortcut linear networks to two extremes. First, when n = 1, let V r,1 = W r,1 + Idx , r = 1, . . . , R− 1. As Idx is a constant, we have
∆V r,1 = α( R−1∏ r′=r+1 V r ′,1)T (ΣY X − ( R−1∏ r′=1 V r ′,1)ΣXX)( r−1∏ r′=1 V r ′,1)T , (14)
which can be seen as a linear network with identity initialization, a special case of orthogonal initialization, if the original 1-shortcut network is zero initialized.
On the other side, if the number of shortcut connections R = 1, the shortcut will only change the distribution of the output training set from Y to Y − X . These two extremes are illustrated in Figure 4
4.3 LEARNING RESULTS
The optimal weights of an n-shortcut linear network can be easily computed via least squares, which leads to
W = Y XT (XXT )−1 = ΣY X(ΣXX)−1, (15)
and the minimum of the loss function is
Lmin = 1
2m ‖Y − ΣY X(ΣXX)−1X‖2F , (16)
where ‖·‖F denotes the Frobenius norm and (ΣXX)−1 denotes any kind of generalized inverse of ΣXX . So given a training set, we can pre-compute its Lmin and use it to evaluate any n-shortcut linear network.
5 EXPERIMENTS
We compare networks with Xavier initialization (Glorot & Bengio, 2010), networks with orthogonal initialization (Saxe et al., 2013) and 2-shortcut networks with zero initialization. The training dynamics of 1-shortcut networks are similar to that of linear networks with orthogonal initialization in our experiments. Setup details can be found in Appendix B.
5.1 INITIAL POINT
We first compute the initial condition numbers for different kinds of linear networks with different depths.
As can be seen in Figure 5, 2-shortcut linear networks have constant condition numbers as expected. On the other hand, when using Xavier or orthogonal initialization in linear networks, the initial condition numbers will go to infinity as the depths become infinity, making the networks hard to train. This also explains why orthogonal initialization is helpful for a linear network, as its initial condition number grows slower than the Xavier initialization.
5.2 LEARNING DYNAMICS
Having a good beginning does not guarantee an easy trip on the loss surface. In order to depict the loss surfaces encountered from different initial points, we plot the maxima and 10th percentiles (instead of minima, as they are very unstable) of the absolute values of Hessians eigenvalues at different losses.
As shown in Figure 6 and Figure 7, the condition numbers of 2-shortcut networks at different losses are always smaller, especially when the loss is large. Also, notice that the condition numbers roughly evolved to the same value for both orthogonal and 2-shortcut linear networks. This may be explained by the fact that the minimizers, as well as any point near them, have similar condition numbers.
Another observation is the changes of negative eigenvalues ratios. Index (ratio of negative eigenvalues) is an important characteristic of a critical point. Usually for the critical points of a neural network, the larger the loss the larger the index (Dauphin et al., 2014). In our experiments, the index of a 2-shortcut network is always smaller, and drops dramatically at the beginning, as shown in Figure 8, left. This might make the networks tend to stop at low critical points.
This is because the initial point is near a saddle point, thus it tends to go towards negative curvature directions, eliminating some negative eigenvalues at the beginning. This phenomenon matches the observation that the gradient reaches its maximum when the index drops dramatically, as shown in Figure 8, right.
5.3 LEARNING RESULTS
We run different networks for 1000 epochs using different learning rates at log scale, and compare the average final losses of the optimal learning rates.
Figure 9 shows the results for linear networks. Just like their depth-invariant initial condition numbers, the final losses of 2-shortcut linear networks stay close to optimal as the networks become deeper. Higher learning rates can also be applied, resulting in fast learning in deep networks.
Then we add ReLUs to the mid positions of the networks. To make a fair comparison, the numbers of ReLU units in different networks are the same when the depths are the same, so 1-shortcut and 3-shortcut networks are omitted. The result is shown in Figure 10.
Note that because of the nonlinearities, the optimal losses vary for different networks with different depths. It is usually thought that deeper networks can represent more complex models, leading to smaller optimal losses. However, our experiments show that linear networks with Xavier or orthogonal initialization have difficulties finding these optimal points, while 2-shortcut networks find these optimal points easily as they did without nonlinear units.
6 FUTURE DIRECTIONS
Further studies should concentrate on the behavior of shortcut connections on convolution networks, as well as the influences of batch normalization units (Ioffe & Szegedy, 2015) in ResNet. Meanwhile, it would be very interesting to extend the insights obtained in this paper to recurrent neural networks such as LSTM (Sainath et al., 2013).
A PROOFS OF THEOREMS
Definition 5. The elements in Hessian of an n-shortcut network is defined as
Hind(w1),ind(w2) = ∂2L
∂w1∂w2 , (17)
where L is the loss function, and the indices ind(·) is ordered lexicographically following the four indices (r, l, j, i) of the weight variable wr,li,j . In other words, the priority decreases along the index of shortcuts, index of weight matrix inside shortcuts, index of column, and index of row.
Note that the collection of all the weight variables in the n-shortcut network is denoted as w. We study the behavior of the loss function in the vicinity of w = 0.
Lemma 1. Assume that w1 = wr1,l1i1,j1 , · · · , wN = w rN ,lN iN ,jN are N parameters of an n-shortcut network. If ∂ 2L
∂w1···∂wN ∣∣∣ w=0
is nonzero, there exists r and k1, · · · , kn such that rkm = r and lkm = m for m = 1, · · · , n.
Proof. Assume there does not exist such r and k1, · · · , kn, then for all the shortcut units r = 1, · · · , R, there exists a weight matrix l such that none of w1, · · · , wN is in W r,l, so all the transformation paths are zero, which means W = Idx . Then ∂2L ∂w1···∂wN ∣∣∣ w=0
= 0, leading to a contradiction.
Lemma 2. Assume that w1 = wr1,l1i1,j1 , w2 = w r2,l2 i2,j2 , r1 ≤ r2. Let L0(w1, w2) denotes the loss function with all the parameters except w1 and w2 set to 0, w′1 = w 1,l1 i1,j1 , w′2 = w 1+1(r1 6=r2),l2 i2,j2 . Then ∂ 2L0(w1,w2) ∂w1∂w2 |(w1,w2)=0 = ∂2L0(w ′ 1,w ′ 2) ∂w′1∂w ′ 2 |(w′1,w′2)=0.
Proof. As all the residual units expect unit r1 and r2 are identity transformations, reordering residual units while preserving the order of units r1 and r2 will not affect the overall transformation, i.e. L0(w1, w2)|w1=a,w2=b = L′0(w′1, w′2)|w′1=a,w′2=b. So ∂2L0(w1,w2) ∂w1∂w2
|(w1,w2)=0 = ∂2L0(w ′ 1,w ′ 2)
∂w′1∂w ′ 2 |(w′1,w′2)=0.
Proof of Theorem 1. Now we can prove Theorem 1 with the help of the previously established lemmas.
1. Using Lemma 1, for an n-shortcut network, at zero, all the k-th order partial derivatives of the loss function are zero, where k ranges from 1 to n − 1. Hence, the initial point zero is a (n− 1)th-order stationary point of the loss function.
2. Consider the Hessian in n = 2 case. Using Lemma 1 and Lemma 2, the form of Hessian can be directly written as Equation (11), as illustrated in Figure 11.
So we have
eigs(H) = eigs(
[ 0 AT
A 0
] ) = ± √ eigs(ATA). (18)
Thus cond(H) = √
cond(ATA), which is depth-invariant. Note that the dimension of A is d2x × d2x. To get the expression of A, consider two parameters that are in the same residual unit but different weight matrices, i.e. w1 = w r,2 i1,j1 , w2 = w r,1 i2,j2 .
Residual Unit 1
Re sid
ua l U
ni t 1
Residual Unit 2
Re sid
ua l U
ni t 2
…
…
… …
So the eigenvalues of H becomes eigs(H) = ±σ′mid(0)σ′post(0) √
eigs((ΣXσpre(X) − ΣY σpre(X))T (ΣXσpre(X) − ΣY σpre(X))), (22)
which leads to Equation (12).
3. Now consider the Hessian in the n = 1 case. Using Lemma 2, the form of Hessian can be directly written as Equation (13).
To get the expressions of A and B in σpre(x) = σpost(x) = x case, consider two parameters that are in the same residual units, i.e. w1 = w r,1 i1,j1 , w2 = w r,1 i2,j2 .
We have
B(j1−1)dx+i1,(j2−1)dx+i2 = ∂2L
∂w1∂w2
∣∣∣ w=0
(23)
=
{ 1 m ∑m µ=1 x µ j1 xµj2 i1 = i2
0 i1 6= i2 (24)
Rearrange the order of variables using P , we have
B = P Σ XX
. . . ΣXX PT . (25) Then consider two parameters that are in different residual units, i.e. w1 = w r1,1 i1,j1 , w2 = wr2,1i2,j2 , r1 > r2.
We have
A(j1−1)dx+i1,(j2−1)dx+i2 = ∂2L
∂w1∂w2
∣∣∣ w=0
(26)
=  1 m ∑m µ=1(x µ i1 − yµi1)x µ j2 + xµj1x µ j2 j1 = i2, i1 = i2 1 m ∑m µ=1(x µ i1 − yµi1)x µ j2 j1 = i2, i1 6= i2 1 m ∑m µ=1 x µ j1 xµj2 j1 6= i2, i1 = i2
0 j1 6= i2, i1 6= i2
(27)
In the same way, we can rewrite A as
A = Σ XX − ΣY X
. . . ΣXX − ΣY X
PT +B. (28)
B EXPERIMENT SETUP
We took the experiments on whitened versions of MNIST. 10 greatest principal components are kept for the dataset inputs. The dataset outputs are represented using one-hot encoding. The network was trained using gradient descent. For every epoch, the Hessians of the networks were calculated using the method proposed in (Bishop, 1992). As the |λ|min of Hessian is usually very unstable, we calculated |λ|max|λ|(0.1) to represent condition number instead, where |λ|(0.1) is the 10
th percentile of the absolute values of eigenvalues.
As pre, mid or post positions are not defined in linear networks without shortcuts, when comparing Xavier or orthogonal initialized linear networks to 2-shortcut networks, we added ReLUs at the same positions in linear networks as in 2-shortcuts networks.

This paper studies the optimization issue of linear ResNet, and shows mathematically that for 2-shortcuts and zero initialization, the Hessian has condition number independent of depth. I skimmed through the proof but have not checked them carefully.
This result is a nice observation for training deep linear networks.  But I do not think the paper has fully resolved the linear vs nonlinear issue. Some question:
1. Though the revision has added some results using ReLU units, it seems it is only added to the mid positions of the network (sec 5.3), is this how it is typically done in ResNet? Moreover, ReLU is not differentiable at zero point, which does not satisfy the condition you had in Theorem 1. Why not use differentiable activations like sigmoid or tanh?
2. From equation (22) in the appendix, it seems for nonlinear activations, the condition number depends on the derivative \sigma^\prime at 0. Therefore, if we use tanh which has derivative 1 at zero, the condition number is the same for linear and tanh activations. But this probably is not enough to explain the bit difference in performance or optimization for linear and nonlinear networks, or how the situations evolve after learning the 0 point.
3. As for the success of ResNet (or convnets in general) in computer vision, I believe there are more types of nonlinearity such as pooling? Can the result here generalizes to pooling as well?
Minor:
- sec 1 last paragraph, low approximation error typically means more powerful model class and better training error, but not necessarily better test error
- sec 4.1 what do you mean by "zero initialization with small random perturbations"? why not exactly zero initialization, how large is the random perturbation?