ON ORTHOGONALITY AND LEARNING RECURRENT NETWORKS WITH LONG TERM DEPENDENCIES
It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue associated with these challenges. One approach to addressing vanishing and exploding gradients is to use either soft or hard constraints on weight matrices so as to encourage or enforce orthogonality. Orthogonal matrices preserve gradient norm during backpropagation and can therefore be a desirable property; however, we find that hard constraints on orthogonality can negatively affect the speed of convergence and model performance. This paper explores the issues of optimization convergence, speed and gradient stability using a variety of different methods for encouraging or enforcing orthogonality. In particular we propose a weight matrix factorization and parameterization strategy through which we can bound matrix norms and therein control the degree of expansivity induced during backpropagation.
1 INTRODUCTION
The depth of deep neural networks confers representational power, but also makes model optimization more challenging. Training deep networks with gradient descent based methods is known to be difficult as a consequence of the vanishing and exploding gradient problem (Hochreiter & Schmidhuber, 1997). Typically, exploding gradients are avoided by clipping large gradients (Pascanu et al., 2013) or introducing an L2 or L1 weight norm penalty. The latter has the effect of bounding the spectral radius of the linear transformations, thus limiting the maximal gain across the transformation. Krueger & Memisevic (2015) attempt to stabilize the norm of propagating signals directly by penalizing differences in successive norm pairs in the forward pass and Pascanu et al. (2013) propose to penalize successive gradient norm pairs in the backward pass. These regularizers affect the network parameterization with respect to the data instead of penalizing weights directly.
Both expansivity and contractivity of linear transformations can also be limited by more tightly bounding their spectra. By limiting the transformations to be orthogonal, their singular spectra are limited to unitary gain causing the transformations to be norm-preserving. Le et al. (2015) and Henaff et al. (2016) have respectively shown that identity initialization and orthogonal initialization can be beneficial. Arjovsky et al. (2015) have gone beyond initialization, building unitary recurrent neural network (RNN) models with transformations that are unitary by construction which they achieved by composing multiple basic unitary transformations. The resulting transformations, for some n-dimensional input, cover only some subset of possible n × n unitary matrices but appear to perform well on simple tasks and have the benefit of having low complexity in memory and computation.
The entire set of possible unitary or orthogonal parameterizations forms the Stiefel manifold. At a much higher computational cost, gradient descent optimization directly along this manifold can be done via geodesic steps (Nishimori, 2005; Tagare, 2011). Recent work (Wisdom et al., 2016) has proposed the optimization of unitary matrices along the Stiefel manifold using geodesic gradient descent. To produce a full-capacity parameterization for unitary matrices they use some insights
from Tagare (2011), combining the use of a canonical inner products and Cayley transformations. Their experimental work indicates that full capacity unitary RNN models can solve the copy memory problem whereas both LSTM networks and restricted capacity unitary RNN models having similar complexity appear unable to solve the task for a longer sequence length (T = 2000).
In contrast, here we explore the optimization of real valued matrices within a configurable margin about the Stiefel manifold. We suspect that a strong constraint of orthogonality limits the model’s representational power, hindering its performance, and may make optimization more difficult. We explore this hypothesis empirically by employing a factorization technique that allows us to limit the degree of deviation from the Stiefel manifold. While we use geodesic gradient descent, we simultaneously update the singular spectra of our matrices along Euclidean steps, allowing optimization to step away from the manifold while still curving about it.
1.1 VANISHING AND EXPLODING GRADIENTS
The issue of vanishing and exploding gradients as it pertains to the parameterization of neural networks can be illuminated by looking at the gradient back-propagation chain through a network.
A neural network with n hidden layers has pre-activations
ai(hi−1) = Wi hi−1 + bi, i ∈ {2, · · · , n} (1)
For notational convenience, we combine parameters Wi and bi to form an affine matrix θ. We can see that for some loss function L at layer n , the derivative with respect to parameters θi is:
∂L ∂θi = ∂an+1 ∂θi ∂L ∂an+1 (2)
The partial derivatives for the pre-activations can be decomposed as follows:
∂ai+1 ∂θi = ∂ai ∂θi ∂hi ∂ai ∂ai+1 ∂hi
= ∂ai ∂θi DiWi+1 → ∂ai+1 ∂ai = DiWi+1,
(3)
where Di is the Jacobian corresponding to the activation function, containing partial derivatives of the hidden units at layer i + 1 with respect to the pre-activation inputs. Typically, D is diagonal. Following the above, the gradient in equation 2 can be fully decomposed into a recursive chain of matrix products:
∂L ∂θi = ∂ai ∂θi n∏ j=i (DjWj+1) ∂L ∂an+1 (4)
In (Pascanu et al., 2013), it is shown that the 2-norm of ∂ai+1 ∂ai is bounded by the product of the norms of the non-linearity’s Jacobian and transition matrix at time t (layer i ), as follows:∣∣∣∣∣∣∣∣∂at+1∂at ∣∣∣∣∣∣∣∣ ≤ ||Dt|| ||Wt|| ≤ λDt λWt = ηt,
λDt , λWt ∈ R. (5)
where λDt and λWt are the largest singular values of the non-linearity’s Jacobian Dt and the transition matrix Wt . In RNNs, Wt is shared across time and can be simply denoted as W.
Equation 5 shows that the gradient can grow or shrink at each layer depending on the gain of each layer’s linear transformation W and the gain of the Jacobian D. The gain caused by each layer is magnified across all time steps or layers. It is easy to have extreme amplification in a recurrent neural network where W is shared across time steps and a non-unitary gain in W is amplified exponentially. The phenomena of extreme growth or contraction of the gradient across time steps or layers are known as the exploding and the vanishing gradient problems, respectively. It is sufficient for RNNs to have ηt ≤ 1 at each time t to enable the possibility of vanishing gradients, typically for some large number of time steps T . The rate at which a gradient (or forward signal) vanishes
depends on both the parameterization of the model and on the input data. The parameterization may be conditioned by placing appropriate constraints on W. It is worth keeping in mind that the Jacobian D is typically contractive, thus tending to be norm-reducing) and is also data-dependent, whereas W can vary from being contractive to norm-preserving, to expansive and applies the same gain on the forward signal as on the back-propagated gradient signal.
2 OUR APPROACH
Vanishing and exploding gradients can be controlled to a large extent by controlling the maximum and minimum gain of W. The maximum gain of a matrix W is given by the spectral norm which is given by
||W||2 = max [ ||Wx|| ||x|| ] . (6)
By keeping our weight matrix W close to orthogonal, one can ensure that it is close to a normpreserving transformation (where the spectral norm is equal to one, but the minimum gain is also one). One way to achieve this is via a simple soft constraint or regularization term of the form:
λ ∑ i ||WTi Wi − I||2. (7)
However, it is possible to formulate a more direct parameterization or factorization for W which permits hard bounds on the amount of expansion and contraction induced by W. This can be achieved by simply parameterizing W according to its singular value decomposition, which consists of the composition of orthogonal basis matrices U and V with a diagonal spectral matrix S containing the singular values which are real and positive by definition. We have
W = USVT . (8)
Since the spectral norm or maximum gain of a matrix is equal to its largest singular value, this decomposition allows us to control the maximum gain or expansivity of the weight matrix by controlling the magnitude of the largest singular value. Similarly, the minimum gain or contractivity of a matrix can be obtained from the minimum singular value.
We can keep the bases U and V orthogonal via geodesic gradient descent along the set of weights that satisfy UTU = I and VTV = I respectively. The submanifolds that satisfy these constraints are called Stiefel manifolds. We discuss how this is achieved in more detail below, then discuss our construction for bounding the singular values.
During optimization, in order to maintain the orthogonality of an orthogonally-initialized matrix M, i.e. where M = U, M = V or M = W if so desired, we employ a Cayley transformation of the update step onto the Stiefel manifold of (semi-)orthogonal matrices, as in Nishimori (2005) and Tagare (2011). Given an orthogonally-initialized parameter matrix M and its Jacobian, G with respect to the objective function, an update is performed as follows:
A = GMT −MGT
Mnew = M+ (I+ η 2 A)−1(I− η 2 A),
(9)
where A is a skew-symmetric matrix (that depends on the Jacobian and on the parameter matrix) which is mapped to an orthogonal matrix via a Cayley transform and η is the learning rate.
While the update rule in (9) allows us to maintain an orthogonal hidden to hidden transition matrix W if desired, we are interested in exploring the effect of stepping away from the Stiefel manifold. As such, we parameterize the transition matrix W in factorized form, as a singular value decomposition with orthogonal bases U and V updated by geodesic gradient descent using the Cayley transform approach above.
If W is an orthogonal matrix, the singular values in the diagonal matrix S are all equal to one. However, in our formulation we allow these singular values to deviate from one and employ a sigmoidal parameterization to apply a hard constraint on the maximum and minimum amount of
deviation. Specifically, we define a margin m around 1 within which the singular values must lie. This is achieved with the parameterization
si = 2m(σ(pi)− 0.5) + 1, si ∈ {diag(S)}, m ∈ [0, 1]. (10)
The singular values are thus restricted to the range [1−m, 1 +m] and the underlying parameters pi are updated freely via stochastic gradient descent. Note that this parameterization strategy also has implications on the step sizes that gradient descent based optimization will take when updating the singular values – they tend to be smaller compared to models with no margin constraining their values. Specifically, a singular value’s progression toward a margin is slowed the closer it is to the margin. The sigmoidal parameterization can also impart another effect on the step size along the spectrum which needs to be accounted for. Considering 10, the gradient backpropagation of some loss L toward parameters pi is found as
dL dpi = dsi dpi dL dsi = 2m dσ(pi) dpi dL dsi . (11)
From (11), it can be seen that the magnitude of the update step for pi is scaled by the margin hyperparameter m . This means for example that for margins less than one, the effective learning rate for the spectrum is reduced in proportion to the margin. Consequently, we adjust the learning rate along the spectrum to be independent of the margin by renormalizing it by 2m .
This margin formulation both guarantees singular values lie within a well defined range and slows deviation from orthogonality. Alternatively, one could enforce the orthogonality of U and V and impose a regularization term corresponding to a mean one Gaussian prior on these singular values. This encourages the weight matrix W to be norm preserving with a controllable strength equivalent to the variance of the Gaussian. We also explore this approach further below.
3 EXPERIMENTS
In this section, we explore hard and soft orthogonality constraints on factorized weight matrices for recurrent neural network hidden to hidden transitions. With hard orthogonality constraints on U and V, we investigate the effect of widening the spectral margin or bounds on convergence and performance. Loosening these bounds allows increasingly larger margins within which the transition matrix W can deviate from orthogonality. We confirm that orthogonal initialization is useful as noted in Henaff et al. (2016), and we show that although strict orthogonality guarantees stable gradient norm, loosening orthogonality constraints can increase the rate of gradient descent convergence. We begin our analyses on tasks that are designed to stress memory: a sequence copying task and a basic addition task (Hochreiter & Schmidhuber, 1997). We then move on to tasks on real data that require models to capture long-range dependencies: digit classification based on sequential and permuted MNIST vectors (Le et al., 2015; LeCun et al., 1998). Finally, we look at a basic language modeling task using the Penn Treebank dataset (Marcus et al., 1993).
The copy and adding tasks, introduced by Hochreiter & Schmidhuber (1997), are synthetic benchmarks with pathologically hard long distance dependencies that require long-term memory in models. The copy task consists of an input sequence that must be remembered by the network, followed by a series of blank inputs terminated by a delimiter that denotes the point at which the network must begin to output a copy of the initial sequence. We use an input sequence of T + 20 elements that begins with a sub-sequence of 10 elements to copy, each containing a symbol ai ∈ {a1 , ..., ap} out of p = 8 possible symbols. This sub-sequence is followed by T − 1 elements of the blank category a0 which is terminated at step T by a delimiter symbol ap+1 and 10 more elements of the blank category. The network must learn to remember the initial 10 element sequence for T time steps and output it after receiving the delimiter symbol.
The goal of the adding task is to add two numbers together after a long delay. Each number is randomly picked at a unique position in a sequence of length T . The sequence is composed of T values sampled from a uniform distribution in the range [0, 1), with each value paired with an indicator value that identifies the value as one of the two numbers to remember (marked 1) or as a value to ignore (marked 0). The two numbers are positioned randomly in the sequence, the first in the range [0, T2 − 1] and the second in the range [ T 2 , T − 1], where 0 marks the first element. The network must learn to identify and remember the two numbers and output their sum.
The sequential MNIST task from Le et al. (2015), MNIST digits are flattened into vectors that can be traversed sequentially by a recurrent neural network. The goal is to classify the digit based on the sequential input of pixels. The simple variant of this task is with a simple flattening of the image matrices; the harder variant of this task includes a random permutation of the pixels in the input vector that is determined once for an experiment. The latter formulation introduces longer distance dependencies between pixels that must be interpreted by the classification model.
The English Penn Treebank (PTB) dataset from Marcus et al. (1993) is an annotated corpus of English sentences, commonly used for benchmarking language models. We employ a sequential character prediction task: given a sentence, a recurrent neural network must predict the next character at each step, from left to right. We use input sequences of variable length, with each sequence containing one sentence. We model 49 characters including lowercase letters (all strings are in lowercase), numbers, common punctuation, and an unknown character placeholder. In our experiments on two subsets of the data: in the first, we first use 23% of the data with strings with up to 75 characters and in the second we include over 99% of the dataset, picking strings with up to 300 characters.
3.1 LOOSENING HARD ORTHOGONALITY CONSTRAINTS
In this section, we experimentally explore the effect of loosening hard orthogonality constraints through loosening the spectral margin defined above for the hidden to hidden transition matrix.
In all experiments, we employed RMSprop (Tieleman & Hinton, 2012) when not using geodesic gradient descent. We used minibatches of size 50 and for generated data (the copy and adding tasks), we assumed an epoch length of 100 minibatches. We cautiously introduced gradient clipping at magnitude 100 (unless stated otherwise) in all of our RNN experiments although it may not be required and we consistently applied a small weight decay of 0.0001. Unless otherwise specified, we trained all simple recurrent neural networks with the hidden to hidden matrix factorization as in (8) using geodesic gradient descent on the bases (learning rate 10−6) and RMSprop on the other parameters (learning rate 0.0001), using a tanh transition nonlinearity, and clipping gradients of 100 magnitude. The neural network code was built on the Theano framework (Theano Development Team, 2016). When parameterizing a matrix in factorized form, we apply the weight decay on the composite matrix rather than on the factors in order to be consistent across experiments. For MNIST and PTB, test set metrics were computed based on the parameterization that gave the best validation set accuracy.
3.1.1 CONVERGENCE ON SYNTHETIC MEMORY TASKS
For different sequence lengths T of the copy and adding tasks, we trained a factorized RNN with 128 hidden units and various spectral margins m . For the copy task, we used Elman networks without a transition non-linearity as in Henaff et al. (2016). We discuss our investigations into the use of a non-linearity on the copy task in the Appendix.
As shown in Figure 1 we see an increase in the rate of convergence as we increase the spectral margin. This observation generally holds across the tested sequence lengths (T = 200, T = 500, T = 1000, T = 10000); however, large spectral margins hinder convergence on extremely long sequence lengths. At sequence length T = 10000, parameterizations with spectral margins larger than 0.001 converge slower than when using a margin of 0.001. In addition, the experiment without a margin failed to converge on the longest sequence length. This follows the expected pattern where stepping away from the Stiefel manifold may help with gradient descent optimization but loosening orthogonality constraints can reduce the stability of signal propagation through the network.
For the adding task, we trained a factorized RNN on T = 1000 length sequences, using a ReLU activation function on the hidden to hidden transition matrix. The mean squared error (MSE) is shown for different spectral margins in Figure 5 in the Appendix. Testing spectral margins m = 0, m = 1, m = 10, m = 100, and no margin, we find that the models with the purely orthogonal (m = 0) and the unconstrained (no margin) transition matrices failed to begin converging beyond baseline MSE within 2000 epochs.
3.1.2 PERFORMANCE ON REAL DATA
Having confirmed that an orthogonality constraint can negatively impact convergence rate, we seek to investigate the effect on model performance for tasks on real data. We show the results of experiments on permuted sequential MNIST in Table 2 and ordered sequential MNIST in Table 1. The loss curves are shown in Figure 6 in the Appendix and reveal an increased convergence rate for larger spectral margins. We trained the factorized RNN models with 128 hidden units for 120 epochs. We also trained an LSTM with 128 hidden units on both tasks for 150 epochs, configured with peephole connections, orthogonally initialized (and forget gate bias initialized to one), and trained with RMSprop (learning rate 0.0001, clipping gradients of magnitude 1).
We show the results of experiments on PTB character prediction, in terms of bits per character (bpc) and prediction accuracy, for a subset of short sequences (up to 75 characters; 23% of data) in Table 3 and for a subset of long sequences (up to 300 characters; 99% of data) in Table 4. We trained factorized RNN models with 512 hidden units for 200 epochs with geodesic gradient descent on the bases (learning rate 10−6) and RMSprop on the other parameters (learning rate 0.001), using a tanh transition nonlinearity, and clipping gradients of 30 magnitude.
Interestingly, for both the ordered and permuted sequential MNIST tasks, models with a non-zero margin significantly outperform those that are constrained to have purely orthogonal transition matri-
ces (margin of zero). The best results on both the ordered and sequential MNIST tasks were yielded by models with a spectral margin of 0.1, at 94.10% accuracy and 91.44% accuracy, respectively. An LSTM outperformed the RNNs in both tasks; nevertheless, RNNs with hidden to hidden transitions initialized as orthogonal matrices performed admirably without a memory component and without all of the additional parameters associated with gates. Indeed, orthogonally initialized RNNs performed almost on par with the LSTM in the permuted sequential MNIST task which presents longer distance dependencies than the ordered task. Although the optimal margin appears to be 0.1, RNNs with large margins perform almost identically to an RNN without a margin, as long as the transition matrix is initialized as orthogonal. On these tasks, orthogonal initialization appears to significantly outperform Glorot normal initialization (Glorot & Bengio, 2010) or initializing the matrix as identity. It is interesting to note that for the MNIST tasks, orthogonal initialization appears useful while orthogonality constraints appear mainly detrimental. This suggests that while orthogonality helps early training by stabilizing gradient flow across many time steps, orthogonality constraints may need to be loosened on some tasks so as not to over-constrain the model’s representational ability.
Curiously, larger margins and even models without sigmoidal constraints on the spectrum (no margin) performed well as long as they were initialized to be orthogonal, suggesting that evolution away from orthogonality is not a serious problem on MNIST. It is not surprising that orthogonality is useful for the MNIST tasks since they depend on long distance signal propagation with a single output at the end of the input sequence. On the other hand, character prediction with PTB produces an output at every time step. Constraining deviation from orthogonality proved detrimental for short sentences (Table 3) and beneficial when long sentences were included (Table 4). Furthermore, Glorot normal initialization did not perform worse than orthogonal initialization for PTB. Since an output is generated for every character in a sentence, short distance signal propagation is possible. Thus it is possible that the RNN is first learning very local dependencies between neighbouring characters and that given enough context, constraining deviation from orthogonality can help force the network to learn longer distance dependencies.
3.1.3 SPECTRAL AND GRADIENT EVOLUTION
It is interesting to note that even long sequence lengths (T=1000) in the copy task can be solved efficiently with rather large margins on the spectrum. In Figure 2 we look at the gradient propagation of the loss from the last time step in the network with respect to the hidden activations. We can see that for a purely orthogonal parameterization of the transition matrix (when the margin is zero), the gradient norm is preserved across time steps, as expected. We further observe that with increasing margin size, the number of update steps over which this norm preservation survives decreases, though surprisingly not as quickly as expected.
Although the deviation of singular values from one should be slowed by the sigmoidal parameterizations, even parameterizations without a sigmoid (no margin) can be effectively trained for all but the longest sequence lengths. This suggests that the spectrum is not deviating far from orthogonality and that inputs to the hidden to hidden transitions are mostly not aligned along the dimensions of great-
est expansion or contraction. We evaluated the spread of the spectrum in all of our experiments and found that indeed, singular values tend to stay well within their prescribed bounds and only reach the margin when using a very large learning rate that does not permit convergence. Furthermore, when transition matrices are initialized as orthogonal, singular values remain near one throughout training even without a sigmoidal margin for tasks that require long term memory (copy, adding, sequential MNIST). On the other hand, singular value distributions tend to drift away from one for PTB character prediction which may help explain why enforcing an orthogonality constraint can be helpful for this task, when modeling long sequences. Interestingly, singular values spread out less for longer sequence lengths (nevertheless, the T=10000 copy task could not be solved with no sigmoid on the spectrum).
We visualize the spread of singular values for different model parameterizations on the permuted sequential MNIST task in Figure 3. Curiously, we find that the distribution of singular values tends to shift upward to a mean of approximately 1.05 on both the ordered and permuted sequential MNIST tasks. We note that in those experiments, a tanh transition nonlinearity was used which is contractive in both the forward signal pass and the gradient backward pass. An upward shift in the distribution of singular values of the transition matrix would help compensate for that contraction. Indeed, (Saxe et al., 2013) describe this as a possibly good regime for learning in deep neural networks. That the model appears to evolve toward this regime suggests that deviating from it may incur a cost. This is interesting because the cost function cannot take into account numerical issues such as vanishing or exploding gradients (or forward signals); we do not know what could make this deviation costly. That the transition matrix may be compensating for the contraction of the tanh is supported by further experiments: applying a 1.05 pre-activation gain appears to allow a model with a margin of 0 to nearly match the top performance reached on both of the MNIST tasks. Furthermore, when using the OPLU norm-preserving activation function (Chernodub & Nowicki, 2016), we found that orthogonally initialized models performed equally well with all margins, achieving over 90% accuracy on the permuted sequential MNIST task. Unlike orthgonally initialized models, the RNN on the bottom right of Figure 3 with Glorot normal initialized transition matrices, begins and ends with a wide singular spectrum. While there is no clear positive shift in the distribution of singular values, the mean value appears to very gradually increase for both the ordered and permuted sequential MNIST tasks. If the model is to be expected to positively shift singular values to compensate for the contractivity of the tanh nonlinearity, it is not doing so well for the Glorot-initialized case; however, this may be due to the inefficiency of training as a result of vanishing gradients, given that initialization.
3.2 EXPLORING SOFT ORTHOGONALITY CONSTRAINTS
Having established that it may indeed be useful to step away from orthogonality, here we explore two forms of soft constraints (rather than hard bounds as above) on hidden to hidden transition matrix orthogonality. The first is a simple penalty that directly encourages a transition matrix W to be orthogonal, of the form λ||WTW − I||22. This is similar to the orthogonality penalty introduced by Henaff et al. (2016). In the first two subfigures on the left of Figure 4, we explore the effect of weakening this form of regularization. We trained both a regular non-factorized RNN on the T = 200 copy task and a factorized RNN with orthogonal bases on the T = 500 copy task. For the regular RNN, we had to reduce the learning rate to 10−5. Here again we see that weakening the strength of the orthogonality-encouraging penalty can increase convergence speed.
The second approach we explore replaces the sigmoidal margin parameterization with a mean one Gaussian prior on the singular values. In the two right subfigures of Figure 4, we visualize the accuracy on the length 200 copy task, using geoSGD (learning rate 10−6) to keep U and V orthogonal and different strengths of a Gaussian prior with mean one on the singular values. We trained these experiments with regular SGD on the spectrum and other non-orthogonal parameter matrices, using a 10−5 learning rate. We see that priors which are too strong lead to slow convergence. Loosening the strength of the prior makes the optimization more efficient. Furthermore, we compare a direct parameterization of the spectrum (no sigmoid) in Figure 4 with a sigmoidal parameterization, using a large margin of 1. Without the sigmoidal parameterization, optimization quickly becomes unstable; on the other hand, the optimization also becomes unstable if the prior is removed completely in the sigmoidal formulation (margin 1). These results further motivate the idea that parameterizations that deviate from orthogonality may perform better than purely orthogonal ones, as long as they are sufficiently constrained to avoid instability during training.
4 CONCLUSIONS
We have explored a number of methods for controlling the expansivity of gradients during backpropagation based learning in RNNs through manipulating orthogonality constraints and regularization on matrices. Our experiments indicate that while orthogonal initialization may be beneficial, maintaining constraints on orthogonality can be detrimental. Indeed, moving away from hard constraints on matrix orthogonality can help improve optimization convergence rate and model performance. However, we also observe with synthetic tasks that relaxing regularization which encourages the spectral norms of weight matrices to be close to one, or allowing bounds on the spectral norms of weight matrices to be too wide, can reverse these gains and may lead to unstable optimization.
ACKNOWLEDGMENTS
We thank the Natural Sciences and Engineeering Research Council (NSERC) of Canada and Samsung for supporting this research.
5 APPENDIX
5.1 ADDITIONAL FIGURES
5.2 COPY TASK NONLINEARITY
We found that nonlinearities such as a rectified linear unit (ReLU) (Nair & Hinton, 2010) or hyperbolic tangent (tanh) made the copy task far more difficult to solve. Using tanh, a short sequence length (T = 100) copy task required both a soft constraint that encourages orthogonality and thousands of epochs for training. It is worth noting that in the unitary evolution recurrent neural network of Arjovsky et al. (2015), the non-linearity (referred to as the ”modReLU”) is actually initialized as an identity operation that is free to deviate from identity during training. Furthermore, Henaff et al. (2016) derive a solution mechanism for the copy task that drops the non-linearity from an RNN. To explore this further, we experimented with a parametric leaky ReLU activation function (PReLU) which introduces a trainable slope α for negative valued inputs x , producing f (x ) = max(x , 0) + αmin(x , 0) (He et al., 2015). Setting the slope α to one would make the PReLU equivalent to an identity function. We experimented with clamping α to 0.5, 0.7 or 1 in a factorized RNN with a spectral margin of 0.3 and found that only the model with α = 1 solved the T = 1000 length copy task. We also experimented with a trainable slope α, initialized to 0.7 and found that it converges to 0.96, further suggesting the optimal solution for the copy task is without a transition nonlinearity. Since the copy task is purely a memory task, one may imagine that a transition nonlinearity such as a tanh or ReLU may be detrimental to the task as it can lose information. Thus, we also tried a recent activation function that preserves information, called an orthogonal permutation linear unit (OPLU) (Chernodub & Nowicki, 2016). The OPLU preserves norm, making a fully norm-preserving RNN possible. Interestingly, this activation function allowed us to recover identical results on the copy task to those without a nonlinearity for different spectral margins.
5.3 METHOD RUNNING TIME
Although the method proposed in section 2 relies on a matrix inversion, an operation with O(n3) complexity for an n × n matrix, the running time of an RNN factorized in such a way actually remains reasonable. This running time is summarized in Table 5 and includes all computations in the graph, together with the matrix inversion. As this method is meant to be used only for the analysis in this work, we find the running times acceptable for that purpose. Models were run on an Nvidia GTX-770 GPU and were run against the T=100 length copy task.

This paper investigates the impact of orthogonal weight matrices on learning dynamics in RNNs. The paper proposes a variety of interesting optimization formulations that enforce orthogonality in the recurrent weight matrix to varying degrees. The experimental results demonstrate several conclusions: enforcing exact orthogonality does not help learning, while enforcing soft orthogonality or initializing to orthogonal weights can substantially improve learning. While some of the optimization methods proposed currently require matrix inversion and are therefore slow in wall clock time, orthogonal initialization and some of the soft orthogonality constraints are relatively inexpensive and may find their way into practical use.
The experiments are generally done to a high standard and yield a variety of useful insights, and the writing is clear.
The experimental results are based on using a fixed learning rate for the different regularization strengths. Learning speed might be highly dependent on this, and different strengths may admit different maximal stable learning rates. It would be instructive to optimize the learning rate for each margin separately (maybe on one of the shorter sequence lengths) to see how soft orthogonality impacts the stability of the learning process. Fig. 5, for instance, shows that a sigmoid improves stability—but perhaps slightly reducing the learning rate for the non-sigmoid Gaussian prior RNN would make the learning well-behaved again for weightings less than 1.
Fig. 4 shows singular values converging around 1.05 rather than 1. Does initializing to orthogonal matrices multiplied by 1.05 confer any noticeable advantage over standard orthogonal matrices? Especially on the T=10K copy task?
“Curiously, larger margins and even models without sigmoidal constraints on the spectrum (no margin) performed well as long as they were initialized to be orthogonal suggesting that evolution away from orthogonality is not a serious problem on this task.” This is consistent with the analysis given in Saxe et al. 2013, where for deep linear nets, if a singular value is initialized to 1 but dies away during training, this is because it must be zero to implement the desired input-output map. More broadly, an open question has been whether orthogonality is useful as an initialization, as proposed by Saxe et al., where its role is mainly as a preconditioner which makes optimization proceed quickly but doesn’t fundamentally change the optimization problem; or whether it is useful as a regularizer, as proposed by Arjovsky et al. 2015 and Henaff et al. 2015, that is, as an additional constraint in the optimization problem (minimize loss subject to weights being orthogonal). These experiments seem to show that mere initialization to orthogonal weights is enough to reap an optimization speed advantage, and that too much regularization begins to hurt performance—i.e., substantially changing the optimization problem is undesirable. This point is also apparent in Fig. 2: In terms of the training loss on MNIST (Fig. 2), no margin does almost indistinguishably from a margin of 1 or .1. However in terms of accuracy, a margin of .1 is best. This shows that large or nonexistent margins (i.e., orthogonal initializations) enable fast optimization of the training loss, but among models that attain similar training loss, the more nearly orthogonal weights perform better. This starts to separate out the optimization speed advantage conferred by orthogonality from the regularization advantage it confers. It may be useful to more explicitly discuss the initialization vs regularization dimension in the text.
Overall, this paper contributes a variety of techniques and intuitions which are likely to be useful in training RNNs.


Neural networks are powerful and flexible models that work well for many difficult learning tasks in image, speech and natural language understanding. Despite their success, neural networks are still hard to design. In this paper, we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. Our CIFAR-10 model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. On the Penn Treebank dataset, our model can compose a novel recurrent cell that outperforms the widely-used LSTM cell, and other state-of-the-art baselines. Our cell achieves a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. The cell can also be transferred to the character language modeling task on PTB and achieves a state-of-the-art perplexity of 1.214.
1 INTRODUCTION
The last few years have seen much success of deep neural networks in many challenging applications, such as speech recognition (Hinton et al., 2012), image recognition (LeCun et al., 1998; Krizhevsky et al., 2012) and machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016). Along with this success is a paradigm shift from feature designing to architecture designing, i.e., from SIFT (Lowe, 1999), and HOG (Dalal & Triggs, 2005), to AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan & Zisserman, 2014), GoogleNet (Szegedy et al., 2015), and ResNet (He et al., 2016a). Although it has become easier, designing architectures still requires a lot of expert knowledge and takes ample time.
This paper presents Neural Architecture Search, a gradient-based method for finding good architectures (see Figure 1) . Our work is based on the observation that the structure and connectivity of a ∗Work done as a member of the Google Brain Residency program (g.co/brainresidency.)
neural network can be typically specified by a variable-length string. It is therefore possible to use a recurrent network – the controller – to generate such string. Training the network specified by the string – the “child network” – on the real data will result in an accuracy on a validation set. Using this accuracy as the reward signal, we can compute the policy gradient to update the controller. As a result, in the next iteration, the controller will give higher probabilities to architectures that receive high accuracies. In other words, the controller will learn to improve its search over time.
Our experiments show that Neural Architecture Search can design good models from scratch, an achievement considered not possible with other methods. On image recognition with CIFAR-10, Neural Architecture Search can find a novel ConvNet model that is better than most human-invented architectures. Our CIFAR-10 model achieves a 3.65 test set error, while being 1.05x faster than the current best model. On language modeling with Penn Treebank, Neural Architecture Search can design a novel recurrent cell that is also better than previous RNN and LSTM architectures. The cell that our model found achieves a test set perplexity of 62.4 on the Penn Treebank dataset, which is 3.6 perplexity better than the previous state-of-the-art.
2 RELATED WORK
Hyperparameter optimization is an important research topic in machine learning, and is widely used in practice (Bergstra et al., 2011; Bergstra & Bengio, 2012; Snoek et al., 2012; 2015; Saxena & Verbeek, 2016). Despite their success, these methods are still limited in that they only search models from a fixed-length space. In other words, it is difficult to ask them to generate a variable-length configuration that specifies the structure and connectivity of a network. In practice, these methods often work better if they are supplied with a good initial model (Bergstra & Bengio, 2012; Snoek et al., 2012; 2015). There are Bayesian optimization methods that allow to search non fixed length architectures (Bergstra et al., 2013; Mendoza et al., 2016), but they are less general and less flexible than the method proposed in this paper.
Modern neuro-evolution algorithms, e.g., Wierstra et al. (2005); Floreano et al. (2008); Stanley et al. (2009), on the other hand, are much more flexible for composing novel models, yet they are usually less practical at a large scale. Their limitations lie in the fact that they are search-based methods, thus they are slow or require many heuristics to work well.
Neural Architecture Search has some parallels to program synthesis and inductive programming, the idea of searching a program from examples (Summers, 1977; Biermann, 1978). In machine learning, probabilistic program induction has been used successfully in many settings, such as learning to solve simple Q&A (Liang et al., 2010; Neelakantan et al., 2015; Andreas et al., 2016), sort a list of numbers (Reed & de Freitas, 2015), and learning with very few examples (Lake et al., 2015).
The controller in Neural Architecture Search is auto-regressive, which means it predicts hyperparameters one a time, conditioned on previous predictions. This idea is borrowed from the decoder in end-to-end sequence to sequence learning (Sutskever et al., 2014). Unlike sequence to sequence learning, our method optimizes a non-differentiable metric, which is the accuracy of the child network. It is therefore similar to the work on BLEU optimization in Neural Machine Translation (Ranzato et al., 2015; Shen et al., 2016). Unlike these approaches, our method learns directly from the reward signal without any supervised bootstrapping.
Also related to our work is the idea of learning to learn or meta-learning (Thrun & Pratt, 2012), a general framework of using information learned in one task to improve a future task. More closely related is the idea of using a neural network to learn the gradient descent updates for another network (Andrychowicz et al., 2016) and the idea of using reinforcement learning to find update policies for another network (Li & Malik, 2016).
3 METHODS
In the following section, we will first describe a simple method of using a recurrent network to generate convolutional architectures. We will show how the recurrent network can be trained with a policy gradient method to maximize the expected accuracy of the sampled architectures. We will present several improvements of our core approach such as forming skip connections to increase model complexity and using a parameter server approach to speed up training. In the last part of
the section, we will focus on generating recurrent architectures, which is another key contribution of our paper.
3.1 GENERATE MODEL DESCRIPTIONS WITH A CONTROLLER RECURRENT NEURAL NETWORK
In Neural Architecture Search, we use a controller to generate architectural hyperparameters of neural networks. To be flexible, the controller is implemented as a recurrent neural network. Let’s suppose we would like to predict feedforward neural networks with only convolutional layers, we can use the controller to generate their hyperparameters as a sequence of tokens:
In our experiments, the process of generating an architecture stops if the number of layers exceeds a certain value. This value follows a schedule where we increase it as training progresses. Once the controller RNN finishes generating an architecture, a neural network with this architecture is built and trained. At convergence, the accuracy of the network on a held-out validation set is recorded. The parameters of the controller RNN, θc, are then optimized in order to maximize the expected validation accuracy of the proposed architectures. In the next section, we will describe a policy gradient method which we use to update parameters θc so that the controller RNN generates better architectures over time.
3.2 TRAINING WITH REINFORCE
The list of tokens that the controller predicts can be viewed as a list of actions a1:T to design an architecture for a child network. At convergence, this child network will achieve an accuracy R on a held-out dataset. We can use this accuracy R as the reward signal and use reinforcement learning to train the controller. More concretely, to find the optimal architecture, we ask our controller to maximize its expected reward, represented by J(θc):
J(θc) = EP (a1:T ;θc)[R]
Since the reward signalR is non-differentiable, we need to use a policy gradient method to iteratively update θc. In this work, we use the REINFORCE rule from Williams (1992):
5θcJ(θc) = T∑ t=1 EP (a1:T ;θc) [ 5θc logP (at|a(t−1):1; θc)R ] An empirical approximation of the above quantity is:
1
m m∑ k=1 T∑ t=1 5θc logP (at|a(t−1):1; θc)Rk
Where m is the number of different architectures that the controller samples in one batch and T is the number of hyperparameters our controller has to predict to design a neural network architecture.
The validation accuracy that the k-th neural network architecture achieves after being trained on a training dataset is Rk.
The above update is an unbiased estimate for our gradient, but has a very high variance. In order to reduce the variance of this estimate we employ a baseline function:
1
m m∑ k=1 T∑ t=1 5θc logP (at|a(t−1):1; θc)(Rk − b)
As long as the baseline function b does not depend on the on the current action, then this is still an unbiased gradient estimate. In this work, our baseline b is an exponential moving average of the previous architecture accuracies.
Accelerate Training with Parallelism and Asynchronous Updates: In Neural Architecture Search, each gradient update to the controller parameters θc corresponds to training one child network to convergence. As training a child network can take hours, we use distributed training and asynchronous parameter updates in order to speed up the learning process of the controller (Dean et al., 2012). We use a parameter-server scheme where we have a parameter server of S shards, that store the shared parameters for K controller replicas. Each controller replica samples m different child architectures that are trained in parallel. The controller then collects gradients according to the results of that minibatch of m architectures at convergence and sends them to the parameter server in order to update the weights across all controller replicas. In our implementation, convergence of each child network is reached when its training exceeds a certain number of epochs. This scheme of parallelism is summarized in Figure 3.
3.3 INCREASE ARCHITECTURE COMPLEXITY WITH SKIP CONNECTIONS AND OTHER LAYER TYPES
In Section 3.1, the search space does not have skip connections, or branching layers used in modern architectures such as GoogleNet (Szegedy et al., 2015), and Residual Net (He et al., 2016a). In this section we introduce a method that allows our controller to propose skip connections or branching layers, thereby widening the search space.
To enable the controller to predict such connections, we use a set-selection type attention (Neelakantan et al., 2015) which was built upon the attention mechanism (Bahdanau et al., 2015; Vinyals et al., 2015). At layer N , we add an anchor point which has N − 1 content-based sigmoids to indicate the previous layers that need to be connected. Each sigmoid is a function of the current hiddenstate of the controller and the previous hiddenstates of the previous N − 1 anchor points:
P(Layer j is an input to layer i) = sigmoid(vTtanh(Wprev ∗ hj +Wcurr ∗ hi)), where hj represents the hiddenstate of the controller at anchor point for the j-th layer, where j ranges from 0 to N − 1. We then sample from these sigmoids to decide what previous layers to be used as inputs to the current layer. The matrices Wprev , Wcurr and v are trainable parameters. As
these connections are also defined by probability distributions, the REINFORCE method still applies without any significant modifications. Figure 4 shows how the controller uses skip connections to decide what layers it wants as inputs to the current layer.
In our framework, if one layer has many input layers then all input layers are concatenated in the depth dimension. Skip connections can cause “compilation failures” where one layer is not compatible with another layer, or one layer may not have any input or output. To circumvent these issues, we employ three simple techniques. First, if a layer is not connected to any input layer then the image is used as the input layer. Second, at the final layer we take all layer outputs that have not been connected and concatenate them before sending this final hiddenstate to the classifier. Lastly, if input layers to be concatenated have different sizes, we pad the small layers with zeros so that the concatenated layers have the same sizes.
Finally, in Section 3.1, we do not predict the learning rate and we also assume that the architectures consist of only convolutional layers, which is also quite restrictive. It is possible to add the learning rate as one of the predictions. Additionally, it is also possible to predict pooling, local contrast normalization (Jarrett et al., 2009; Krizhevsky et al., 2012), and batchnorm (Ioffe & Szegedy, 2015) in the architectures. To be able to add more types of layers, we need to add an additional step in the controller RNN to predict the layer type, then other hyperparameters associated with it.
3.4 GENERATE RECURRENT CELL ARCHITECTURES
In this section, we will modify the above method to generate recurrent cells. At every time step t, the controller needs to find a functional form for ht that takes xt and ht−1 as inputs. The simplest way is to have ht = tanh(W1∗xt+W2∗ht−1), which is the formulation of a basic recurrent cell. A more complicated formulation is the widely-used LSTM recurrent cell (Hochreiter & Schmidhuber, 1997).
The computations for basic RNN and LSTM cells can be generalized as a tree of steps that take xt and ht−1 as inputs and produce ht as final output. The controller RNN needs to label each node in the tree with a combination method (addition, elementwise multiplication, etc.) and an activation function (tanh, sigmoid, etc.) to merge two inputs and produce one output. Two outputs are then fed as inputs to the next node in the tree. To allow the controller RNN to select these methods and functions, we index the nodes in the tree in an order so that the controller RNN can visit each node one by one and label the needed hyperparameters.
Inspired by the construction of the LSTM cell (Hochreiter & Schmidhuber, 1997), we also need cell variables ct−1 and ct to represent the memory states. To incorporate these variables, we need the controller RNN to predict what nodes in the tree to connect these two variables to. These predictions can be done in the last two blocks of the controller RNN.
To make this process more clear, we show an example in Figure 5, for a tree structure that has two leaf nodes and one internal node. The leaf nodes are indexed by 0 and 1, and the internal node is indexed by 2. The controller RNN needs to first predict 3 blocks, each block specifying a combination method and an activation function for each tree index. After that it needs to predict the last 2 blocks that specify how to connect ct and ct−1 to temporary variables inside the tree. Specifically,
according to the predictions of the controller RNN in this example, the following computation steps will occur:
• The controller predicts Add and Tanh for tree index 0, this means we need to compute a0 = tanh(W1 ∗ xt +W2 ∗ ht−1).
• The controller predicts ElemMult and ReLU for tree index 1, this means we need to compute a1 = ReLU ( (W3 ∗ xt) (W4 ∗ ht−1) ) .
• The controller predicts 0 for the second element of the “Cell Index”, Add and ReLU for elements in “Cell Inject”, which means we need to compute anew0 = ReLU(a0 + ct−1). Notice that we don’t have any learnable parameters for the internal nodes of the tree.
• The controller predicts ElemMult and Sigmoid for tree index 2, this means we need to compute a2 = sigmoid(anew0 a1). Since the maximum index in the tree is 2, ht is set to a2.
• The controller RNN predicts 1 for the first element of the “Cell Index”, this means that we should set ct to the output of the tree at index 1 before the activation, i.e., ct = (W3 ∗xt) (W4 ∗ ht−1).
In the above example, the tree has two leaf nodes, thus it is called a “base 2” architecture. In our experiments, we use a base number of 8 to make sure that the cell is expressive.
4 EXPERIMENTS AND RESULTS
We apply our method to an image classification task with CIFAR-10 and a language modeling task with Penn Treebank, two of the most benchmarked datasets in deep learning. On CIFAR-10, our goal is to find a good convolutional architecture whereas on Penn Treebank our goal is to find a good recurrent cell. On each dataset, we have a separate held-out validation dataset to compute the reward signal. The reported performance on the test set is computed only once for the network that achieves the best result on the held-out validation dataset. More details about our experimental procedures and results are as follows.
4.1 LEARNING CONVOLUTIONAL ARCHITECTURES FOR CIFAR-10
Dataset: In these experiments we use the CIFAR-10 dataset with data preprocessing and augmentation procedures that are in line with other previous results. We first preprocess the data by whitening all the images. Additionally, we upsample each image then choose a random 32x32 crop of this upsampled image. Finally, we use random horizontal flips on this 32x32 cropped image.
Search space: Our search space consists of convolutional architectures, with rectified linear units as non-linearities (Nair & Hinton, 2010), batch normalization (Ioffe & Szegedy, 2015) and skip connections between layers (Section 3.3). For every convolutional layer, the controller RNN has to select a filter height in [1, 3, 5, 7], a filter width in [1, 3, 5, 7], and a number of filters in [24, 36, 48,
64]. For strides, we perform two sets of experiments, one where we fix the strides to be 1, and one where we allow the controller to predict the strides in [1, 2, 3].
Training details: The controller RNN is a two-layer LSTM with 35 hidden units on each layer. It is trained with the ADAM optimizer (Kingma & Ba, 2015) with a learning rate of 0.0006. The weights of the controller are initialized uniformly between -0.08 and 0.08. For the distributed training, we set the number of parameter server shards S to 20, the number of controller replicas K to 100 and the number of child replicas m to 8, which means there are 800 networks being trained on 800 GPUs concurrently at any time.
Once the controller RNN samples an architecture, a child model is constructed and trained for 50 epochs. The reward used for updating the controller is the maximum validation accuracy of the last 5 epochs cubed. The validation set has 5,000 examples randomly sampled from the training set, the remaining 45,000 examples are used for training. The settings for training the CIFAR-10 child models are the same with those used in Huang et al. (2016a). We use the Momentum Optimizer with a learning rate of 0.1, weight decay of 1e-4, momentum of 0.9 and used Nesterov Momentum (Sutskever et al., 2013).
During the training of the controller, we use a schedule of increasing number of layers in the child networks as training progresses. On CIFAR-10, we ask the controller to increase the depth by 2 for the child models every 1,600 samples, starting at 6 layers.
Results: After the controller trains 12,800 architectures, we find the architecture that achieves the best validation accuracy. We then run a small grid search over learning rate, weight decay, batchnorm epsilon and what epoch to decay the learning rate. The best model from this grid search is then run until convergence and we then compute the test accuracy of such model and summarize the results in Table 1. As can be seen from the table, Neural Architecture Search can design several promising architectures that perform as well as some of the best models on this dataset.
First, if we ask the controller to not predict stride or pooling, it can design a 15-layer architecture that achieves 5.50% error rate on the test set. This architecture has a good balance between accuracy and depth. In fact, it is the shallowest and perhaps the most inexpensive architecture among the top performing networks in this table. This architecture is shown in Appendix A, Figure 7. A notable feature of this architecture is that it has many rectangular filters and it prefers larger filters at the top layers. Like residual networks (He et al., 2016a), the architecture also has many one-step skip connections. This architecture is a local optimum in the sense that if we perturb it, its performance becomes worse. For example, if we densely connect all layers with skip connections, its performance becomes slightly worse: 5.56%. If we remove all skip connections, its performance drops to 7.97%.
In the second set of experiments, we ask the controller to predict strides in addition to other hyperparameters. As stated earlier, this is more challenging because the search space is larger. In this case, it finds a 20-layer architecture that achieves 6.01% error rate on the test set, which is not much worse than the first set of experiments.
Finally, if we allow the controller to include 2 pooling layers at layer 13 and layer 24 of the architectures, the controller can design a 39-layer network that achieves 4.47% which is very close to the best human-invented architecture that achieves 3.74%. To limit the search space complexity we have our model predict 13 layers where each layer prediction is a fully connected block of 3 layers. Additionally, we change the number of filters our model can predict from [24, 36, 48, 64] to [6, 12, 24, 36]. Our result can be improved to 3.65% by adding 40 more filters to each layer of our architecture. Additionally this model with 40 filters added is 1.05x as fast as the DenseNet model that achieves 3.74%, while having better performance. The DenseNet model that achieves 3.46% error rate (Huang et al., 2016b) uses 1x1 convolutions to reduce its total number of parameters, which we did not do, so it is not an exact comparison.
4.2 LEARNING RECURRENT CELLS FOR PENN TREEBANK
Dataset: We apply Neural Architecture Search to the Penn Treebank dataset, a well-known benchmark for language modeling. On this task, LSTM architectures tend to excel (Zaremba et al., 2014; Gal, 2015), and improving them is difficult (Jozefowicz et al., 2015). As PTB is a small dataset, regularization methods are needed to avoid overfitting. First, we make use of the embedding dropout and recurrent dropout techniques proposed in Zaremba et al. (2014) and (Gal, 2015). We also try to combine them with the method of sharing Input and Output embeddings, e.g., Bengio et al. (2003); Mnih & Hinton (2007), especially Inan et al. (2016) and Press & Wolf (2016). Results with this method are marked with “shared embeddings.”
Search space: Following Section 3.4, our controller sequentially predicts a combination method then an activation function for each node in the tree. For each node in the tree, the controller RNN needs to select a combination method in [add, elem mult] and an activation method in [identity, tanh, sigmoid, relu]. The number of input pairs to the RNN cell is called the “base number” and set to 8 in our experiments. When the base number is 8, the search space is has approximately 6 × 1016 architectures, which is much larger than 15,000, the number of architectures that we allow our controller to evaluate.
Training details: The controller and its training are almost identical to the CIFAR-10 experiments except for a few modifications: 1) the learning rate for the controller RNN is 0.0005, slightly smaller than that of the controller RNN in CIFAR-10, 2) in the distributed training, we set S to 20, K to 400 and m to 1, which means there are 400 networks being trained on 400 CPUs concurrently at any time, 3) during asynchronous training we only do parameter updates to the parameter-server once 10 gradients from replicas have been accumulated.
In our experiments, every child model is constructed and trained for 35 epochs. Every child model has two layers, with the number of hidden units adjusted so that total number of learnable parameters approximately match the “medium” baselines (Zaremba et al., 2014; Gal, 2015). In these experiments we only have the controller predict the RNN cell structure and fix all other hyperparameters. The reward function is c(validation perplexity)2 where c is a constant, usually set at 80.
After the controller RNN is done training, we take the best RNN cell according to the lowest validation perplexity and then run a grid search over learning rate, weight initialization, dropout rates
and decay epoch. The best cell found was then run with three different configurations and sizes to increase its capacity.
Results: In Table 2, we provide a comprehensive list of architectures and their performance on the PTB dataset. As can be seen from the table, the models found by Neural Architecture Search outperform other state-of-the-art models on this dataset, and one of our best models achieves a gain of almost 3.6 perplexity. Not only is our cell is better, the model that achieves 64 perplexity is also more than two times faster because the previous best network requires running a cell 10 times per time step (Zilly et al., 2016).
The newly discovered cell is visualized in Figure 8 in Appendix A. The visualization reveals that the new cell has many similarities to the LSTM cell in the first few steps, such as it likes to compute W1 ∗ ht−1 +W2 ∗ xt several times and send them to different components in the cell.
Transfer Learning Results: To understand whether the cell can generalize to a different task, we apply it to the character language modeling task on the same dataset. We use an experimental setup that is similar to Ha et al. (2016), but use variational dropout by Gal (2015). We also train our own LSTM with our setup to get a fair LSTM baseline. Models are trained for 80K steps and the best test set perplexity is taken according to the step where validation set perplexity is the best. The results on the test set of our method and state-of-art methods are reported in Table 3. The results on small settings with 5-6M parameters confirm that the new cell does indeed generalize, and is better than the LSTM cell.
Additionally, we carry out a larger experiment where the model has 16.28M parameters. This model has a weight decay rate of 1e − 4, was trained for 600K steps (longer than the above models) and the test perplexity is taken where the validation set perplexity is highest. We use dropout rates of 0.2 and 0.5 as described in Gal (2015), but do not use embedding dropout. We use the ADAM optimizer with a learning rate of 0.001 and an input embedding size of 128. Our model had two layers with 800 hidden units. We used a minibatch size of 32 and BPTT length of 100. With this setting, our model achieves 1.214 perplexity, which is the new state-of-the-art result on this task.
Finally, we also drop our cell into the GNMT framework (Wu et al., 2016), which was previously tuned for LSTM cells, and train an WMT14 English → German translation model. The GNMT
network has 8 layers in the encoder, 8 layers in the decoder. The first layer of the encoder has bidirectional connections. The attention module is a neural network with 1 hidden layer. When a LSTM cell is used, the number of hidden units in each layer is 1024. The model is trained in a distributed setting with a parameter sever and 12 workers. Additionally, each worker uses 8 GPUs and a minibatch of 128. We use Adam with a learning rate of 0.0002 in the first 60K training steps, and SGD with a learning rate of 0.5 until 400K steps. After that the learning rate is annealed by dividing by 2 after every 100K steps until it reaches 0.1. Training is stopped at 800K steps. More details can be found in Wu et al. (2016).
In our experiment with the new cell, we make no change to the above settings except for dropping in the new cell and adjusting the hyperparameters so that the new model should have the same computational complexity with the base model. The result shows that our cell, with the same computational complexity, achieves an improvement of 0.5 test set BLEU than the default LSTM cell. Though this improvement is not huge, the fact that the new cell can be used without any tuning on the existing GNMT framework is encouraging. We expect further tuning can help our cell perform better.
Control Experiment 1 – Adding more functions in the search space: To test the robustness of Neural Architecture Search, we add max to the list of combination functions and sin to the list of activation functions and rerun our experiments. The results show that even with a bigger search space, the model can achieve somewhat comparable performance. The best architecture with max and sin is shown in Figure 8 in Appendix A.
Control Experiment 2 – Comparison against Random Search: Instead of policy gradient, one can use random search to find the best network. Although this baseline seems simple, it is often very hard to surpass (Bergstra & Bengio, 2012). We report the perplexity improvements using policy gradient against random search as training progresses in Figure 6. The results show that not only the best model using policy gradient is better than the best model using random search, but also the average of top models is also much better.
5 CONCLUSION
In this paper we introduce Neural Architecture Search, an idea of using a recurrent neural network to compose neural network architectures. By using recurrent network as the controller, our method is flexible so that it can search variable-length architecture space. Our method has strong empirical performance on very challenging benchmarks and presents a new research direction for automatically finding good neural network architectures. The code for running the models found by the controller on CIFAR-10 and PTB will be released at https://github.com/tensorflow/models . Additionally, we have added the RNN cell found using our method under the name NASCell into TensorFlow, so others can easily use it.
ACKNOWLEDGMENTS
We thank Greg Corrado, Jeff Dean, David Ha, Lukasz Kaiser and the Google Brain team for their help with the project.
A APPENDIX

This paper presents search for optimal neural-net architectures based on actor-critic framework. The method treats DNN as a variable length sequence, and uses RL to find the target architecture, which acts as an actor. The node selection is an action in the RL context, and evaluation error of the outcome architecture corresponds to reward. A auto-regressive two-layer LSTM is used as a controller and critic. The method is evaluated on two different problems, and each compared with number of other human-created architectures.
This is very exciting paper! Hand selecting architectures is difficult, and it is hard to know how far from optimal results the hand-designed networks are. The presented method is  novel. The authors do an excellent job of describing it in detail, with all the improvements that needed to be done. The tested data represents well the capability of the method. It is very interesting to see the differences between the generated architectures and human generated ones. The paper is written very clearly, and is very accessible. The coverage and contrast with the related literature is done well.
It would be interesting to see the data about the time needed for training, and correlation between time/resources needed to train and the quality of the model. It would also be interesting to see how human bootstrapped models perform and involve.
Overall, an excellent and interesting paper.