DYNAMIC STEERABLE FRAME NETWORKS
Filters in a convolutional network are typically parametrized in a pixel basis. As an orthonormal basis, pixels may represent any arbitrary vector in R. In this paper, we relax this orthonormality requirement and extend the set of viable bases to the generalized notion of frames. When applying suitable frame bases to ResNets on Cifar-10+ we demonstrate improved error rates by substitution only. By exploiting the transformation properties of such generalized bases, we arrive at steerable frames, that allow to continuously transform CNN filters under arbitrary Lie-groups. Further allowing us to locally separate pose from canonical appearance. We implement this in the Dynamic Steerable Frame Network, that dynamically estimates the transformations of filters, conditioned on its input. The derived method presents a hybrid of Dynamic Filter Networks and Spatial Transformer Networks that can be implemented in any convolutional architecture, as we illustrate in two examples. First, we illustrate estimation properties of steerable frames with a Dynamic Steerable Frame Network, compared to a Dynamic Filter Network on the task of edge detection, where we show clear advantages of the derived steerable frames. Lastly, we insert the Dynamic Steerable Frame Network as a module in a convolutional LSTM on the task of limited-data hand-gesture recognition from video and illustrate effective dynamic regularization and show clear advantages over Spatial Transformer Networks. In this paper, we have laid out the foundations of Frame-based convolutional networks and Dynamic Steerable Frame Networks while illustrating their advantages for continuously transforming features and data-efficient learning.
1 INTRODUCTION
For images, as well as any other sensory data, convolutional networks are typically learned from individual pixel values. Using them as a basis of the learned parameters is the standard approach for almost all CNNs. In this paper, we argue, that the pixel basis is not necessarily the best choice for representing signals. We show, that suitable alternatives yield increased classification performance by replacement only, while such a replacement adds additional properties to the learned filters that allow us to transform them under arbitrary pre-defined Lie groups.
From our perspective, the pixel values span an orthogonal basis for the filters in the network (in every layer). Such a pixel basis is complete as it may represent an arbitrary vector in Rn by linear combination, where n is the dimensionality of the filter. In this paper we consider alternatives to this basis, both orthogonal bases, and non-orthogonal frames, arriving at superior expressiveness through steerable function spaces that allow us to transform filters locally and continuously, conditioned on their input.
Utilizing the steerability properties of frames in practice, we propose Dynamic Steerable Frame Networks (DSFNs) that fill the gap between Spatial Transformer Networks (STNs) (Jaderberg et al., 2015) and Dynamic Filter Networks (DFNs) (De Brabandere et al., 2016). STNs are not locally adaptive, thus they fail in many cases where it is not beneficial to transform the image globally as it would destroy discriminative information (multiple deformable objects, discriminative dynamic movements) or where global registration is performed as a preprocessing step (medical images).
DFNs are overcoming this restriction by locally transforming filters instead of globally transforming the whole feature stack as STNs do. However, DFNs are black boxes and not data-efficient, as they introduce many unconstrained parameters. Such a behavior is undesirable when data is limited and interpretability is key. DSFNs are locally adaptive, interpretable and data-efficient. They overcome the weaknesses of both approaches by combining their strengths, as illustrated in multiple experiments.
Our contributions:
• We argue that suitable frame bases are beneficial when representing sensory data compared to the commonly used pixel basis.
• Exploiting the transformation properties of frames further, we derive Dynamic Steerable Frame Networks that are able to continuously transform features locally and fill the gap between Spatial Transformer Networks and Dynamic Filter Networks.
• Dynamic Steerable Frame Networks learn to separate pose and feature. This enables the network to be locally equivariant or invariant with respect to certain feature poses, or even to perform in network quasi data-augmentation, while only the inputs and the backpropagated error signals determine which and to what extent these are applied.
We introduce the generalized notion of frames to CNNs that extend possible bases to learn from to non-orthogonal and overcomplete sets without loss in generalization. We show that many choices are possible, while overcomplete, non-orthogonal bases consistently outperform the pixel basis when applied to a ResNet (He et al., 2016) for image classification, as illustrated on Cifar-10+. We derive the Dynamic Steerable Frame Networks, based on the notion of steerable frames, that can locally adapt the filters in every feature map, conditioned on the input. We illustrate the strength of the approach in an edge detection task, where it outperforms a Dynamic Filter Network. We further show in a limited data video classification task, that Dynamic Steerable Frame Networks improve classification performance over Spatial Transformer Networks when global invariance is not desirable.
2 DYNAMICALLY STEERABLE FRAME NETWORKS
2.1 FRAMES
Frames are a natural generalization of orthogonal bases (Christensen, 2003). In frame terminology, an orthonormal basis is a Parseval-tight frame with unit norm. Every tight frame preserves the signal norm and exhibits perfect reconstruction. Frames can be seen as a superset of orthogonal bases in the sense that every basis is a frame, but not the reverse, see figure 1. The advantage of considering frames over orthogonal bases is that intrinsic signal properties can be spelled out explicitly in the new representation with the advantage, that these properties are directly accessible during learning. From an overcomplete representation, it will be more easily visible which part of the features is robust and which part is sensitive to accidental noise variations.
In a standard convolutional network, a filter kernel is a linear combination over the standard basis for l2(N). The standard basis is composed from a delta function for every dimension and Wi is the ith filter of the network with parameters win:
e1 = {1, 0, 0, ..., 0} e2 = {0, 1, 0, ..., 0} ...
en = {0, 0, 0, ..., 1}
Wi = N∑ n=1 winen
Without loss of generalization the orthonormal standard basis can be replaced by a frame to include non-orthogonality, overcompleteness, increased symmetries or steerability into the representation. Changing from the pixel to an arbitrary frame is as simple as replacing the pixel basis en with a frame of choice with elements vn as follows:
Wi = N∑ n=1 winvn (1)
where wi1, ..., w i n are again the filter coefficients being learned.
In practice for CNNs working on images we investigate derived bases from steerability requirements, orthogonal polynomials, Framelets and members of the Gaussian derivative family. See figure 2 for a selection of frames.
2.2 STEERING FRAMES UNDER ARBITRARY LIE-GROUPS
A pleasant property of many frames is steerability (Unser & Chenouard, 2013; Hel-Or & Teo, 1998; Michaelis & Sommer, 1995), the power of a function to represent transformed versions of itself by linear combination. The advantage of steerability in CNNs working on images is the ability to produce infinitely many transformed variants of a visual feature fτ (x, y) ∈ R2 → R from its canonical appearance.
To achieve this goal we cast these variations as the result of the action of a family of transformations g(τ) on the canonical features f(x, y), where τ ∈ Rk parametrizes these k-parameter transformations. If the problem at hand requires the distinction between multiple unknown poses of the same feature in a typical CNN they all have to be computed exhaustively to determine if a particular pose is present or not. Things go out of hand when the search space is a continuous transformation group, such as the Lie group of affine transformations, requiring k →∞ number of feature maps which is computationally intractable or requires expensive searches over all possible transformations (Gens & Domingos, 2014). One way out is to coarsely sample a few equally spaced points on the equivariant transformation manifold or to restrict the space to a smaller group (Cohen & Welling, 2016; Dieleman et al., 2016). What remains, however, is that the number of resulting feature maps for more general groups quickly becomes infeasible. An elegant way to overcome these limitations is the concept of steerability by (Freeman & Adelson, 1991; Perona, 1992; Unser & Chenouard, 2013) which is taken as inspiration here.
In this work, we focus on Lie groups. Transformations g(τ) over a range constitute a Lie group if they are closed under composition, they are associative, they are invertible, there exists an identity element, and their maps for inverse and composition are infinitely differentiable (Hel-Or & Teo, 1998). Teo and colleagues (Teo & Hel-Or, 1998) have given the following definition.
Definition 1 (Steerability): A function f(x, y): R2 → R is steerable under a k-parameter Lie transformation group G if any transformation g(τ) ∈ G of f can be written as a linear combination of a fixed, finite set of frame functions φm(x, y):
g(τ)f(x, y) = M∑ m=1 βm(τ)φm(x, y) = B T (τ)Φ(x, y) (2)
Where BT (τ) denote the collected steering functions describing the transformation and Φ(x, y) the collected steerable frame functions.
A function steerable under a k-parameter Lie group is capable of representing infinitely many states of a particular set of transformations. In many cases, only a finite set of frame functions is needed to represent these. In CNN terms, this means that a limited number of feature maps are sufficient to represent complete continuous transformation groups when the frame functions and the steering functions are chosen appropriately. Finding appropriate frame functions is the biggest challenge in steering arbitrary functions over arbitrary Lie groups.
To study the action of a Lie group G on a function we use the close relation between the Lie group and its tangent space. The Algebra’s tangent space spanned by the group’s infinitesimal generators. The differential operators of the group action are obtained by computing the derivative of the group action with respect to its parameters at the identity element. A Lie Algebra can be considered as an ”infinitesimal” Lie group. If the group is simply connected, the group action on a visual feature f(x, y) can be obtained via the exponential map (Teo & Hel-Or, 1998):
g(τ1, ..., τk)f(x, y) = e ( ∑k i=1 τiLi)f(x, y) (3)
where
eτiLi = I + τiLi + 1
2! τ2i L 2 i + ... (4)
where Li are the group’s infinitesimal generators and I is the identity element. This implies that one can compute the Taylor expansion with respect to the desired transformation group parameters to obtain elements of the group. If a finite frame set is equivariant towards the desired transformation group (it contains the orbit of the function to be steered), the series expansion yields linearly dependent elements after a finite number of steps. Then the frame is globally steerable under the desired transformation group. If this is not the case, as for example when scaling a Gaussian function, a finite frame set is only sufficient to accurately steer the function over a bounded interval, the function is locally steerable, but not globally.
2.3 SEPARATING POSE AND CANONICAL APPEARANCE
When training a CNN the functions represented by each feature naturally change from update to update. It is desirable to separate the frame functions from the effective features as learned by the network. In such a Structured Receptive Fields Network (RFNN) (Jacobsen et al., 2016), each filters parameters are not its mere pixel values, but the coefficients weighting the sum over a fixed frame set. Thus, analogous to equation 1, every effective filter Wi(x, y) has the following form:
Wi(x, y) = w i 1v1 + w i 2v2 + ...+ w i nvn, (5)
where vn denotes the nth element of the frame.
To be able to separate a features pose from its canonical appearance, we are interested in a steerable version of an arbitrary filter Wi(x, y) under a k-parameter Lie group. From 5 follows:
g(τ)Wi(x, y) = N∑ n=1 wing(τ)v i n. (6)
And by substituting according to equation 2 it follows:
g(τ)Wi(x, y) = N∑ n=1 win M∑ m=1 βm(τ)φm(x, y). (7)
Thus it is sufficient to determine the group action on the fixed frame by steering it to separate the canonical feature itself from its k-parameter variants, i.e. vin govern the weight of each frame coefficient to form a featureWi(x, y) and βm are the steering functions governing the transformation of g(τ) acting on Wi(x, y) as a whole. From now on learning and transforming features amounts to a point-wise multiplication of frame coefficients with cos, sin and exp activation functions, which is suitable for learning in a CNN.
2.4 DERIVING THE FRAME AND STEERING FUNCTIONS
Now the problem is reduced to finding a suitable frame as a function space underlying the learned filters. There are many approaches to derive a function space that is closed under the desired transformation group and as we show, many options give rise to bases that work considerably well when inserted into state-of-the-art CNNs. The most straightforward way is to derive it from the group’s infinitesimal generators, for brevity we refer the interested reader to (Hel-Or & Teo, 1998) and directly cite some derived equivariant function spaces from the paper.
Once a frame is chosen, we can simply check if it is closed under the given transformation group by verifying for each generator Li that:
LiΦ(x, y) = BiΦ(x, y), (8)
where Bi is some finite dimensional n×nmatrix. If this is the case, the function space is equivariant under the transformation group and we can compute the steering equations of the group composed of multiple generators as: A(τ) = eτkBk · ... · eτ1B1 . (9) To arrive at a practical solution, we have to consider the problem that locally bounded functions can not be steered globally with a finite steerable frame. To achieve a suitable approximation for our case, we separate scaling into two parts, an inner {σx, σy} and an outer scale σa, where a stands for aperture. The inner scale can directly be steered via the above derivation and represents the slope of the local measurement taken by a filter, while the outer scale represents the size and shape of the filters receptive field. To achieve anisotropic receptive fields, we propose to first steer the scale at every pixel and steer the derived function space on this non-uniformly scaled grid, resulting in locally deformable receptive fields. Due to associativity of convolution, we can combine steering the derived function space and the receptive field scale into one operation. In this work, we use a second order approximation of the Gaussian that is capable of giving a good approximation to common CNN receptive field sizes 3x3, 5x5 and 7x7. For scaling over larger ranges, we recommend the spectral decomposition approach (Koutaki & Uchimura, 2014).
2.5 DYNAMIC STEERABLE FRAME NETWORKS
Estimating the local pose of a feature from a steerable function space is analytically intractable in the case of most multi-parameter groups. In this paper, we introduce the Dynamic Steerable Frame Network that combines the advantages of steerable function spaces with the power of neural network function estimators, by estimating pose parameters from a function space equivariant under the transformation group at hand. Specifically, our architecture is inspired by the recently introduced Dynamic Filter Networks (De Brabandere et al., 2016). The Dynamic Filter Network (DFN) generates one feature per location in a feature map, which boils down to a locally connected convolution layer, for which the parameters are generated by a different network that estimates them from the input, yielding a different filter kernel for every location in the input.
The DFN takes the form: O(x, y) = F x,yτ (I(x, y)), (10)
where F x,yτ are generated by another network from the input. We propose the Dynamic Steerable Frame Network, where the parameters θ that condition the filter are pose transformation parameters of the steerable function space, estimated from the input, similar to how it is done in the Spatial Transformer Networks, just that in our case we aim for locally adaptive filters. The filters F x,yτ share the same set of weights in the whole feature map, so they represent the same canonical appearance everywhere. While their local pose is dynamically estimated by a Pose-Generating Network Ψ that takes the form:
τ(x, y) = Ψ(I(x, y)). (11)
Thus, the canonical appearance is translation invariant, but its geometrical pose is not. In terms of equation 5, this means the set of win is fixed, but the frame v τ(x,y) n is locally transformed under a pre-defined k-parameter group with parameters τ . See figure 5 for an illustration.
The method consists of two parts: i) A Pose-Generating network estimating local pose parameters of a feature conditioned on the input from a steerable input space. ii) A Dynamic Filtering mechanism, convolving transformed versions of a feature with every location in the input feature map, based on the estimates of the pose generating network. Due to linearity of convolution, we can first perform a transformation of the input into the steerable frame space and in this space we perform i) and ii) as point-wise multiplications.
3 RELATED WORK
Steerable Filters is a concept established early for signal processing. Initially introduced by (Freeman & Adelson, 1991), the concept was extended to the Steerable Pyramid by (Simoncelli & Freeman, 1995) and further extended to a Lie-group formulation by (Hel-Or & Teo, 1998; Michaelis & Sommer, 1995). Further, steerability has recently been extended to tight frames, presenting Simoncelli’s Steerable Pyramid and multiple other Wavelets arising as a special case of the non-orthogonal Riesz transform (Unser & Chenouard, 2013). Steerable pyramids have been applied to CNNs as a pre-processing step (Xue et al., 2016), but have not yet been learnable. We incorporate steerable frames in CNNs to increase their de facto expressiveness and to allow them to learn their configurations, rather than picking them a priori.
Convolutional Networks with alternative bases have been proposed with various degrees of flexibility. A number of works utilizes change of basis to stabilize training and increase convergence behavior (Rippel et al., 2015; Arjovsky et al., 2015). Another line of research is concerned with complex-valued CNNs, either learned (Tygert et al., 2016), or fully designed like the Scattering networks (Bruna & Mallat, 2013; Oyallon & Mallat, 2015).
Scattering, as well as the complex-valued networks, rest upon a direct connection between the signal processing literature and CNNs. Inspired by the former, Structured Receptive Field Networks are learned from an overcomplete multi-scale frame, effectively improving performance for small datasets due to restricted feature spaces (Jacobsen et al., 2016). Also related is the work on Groupequivariant CNNs (Cohen & Welling, 2016) and Cyclic Pooling (Dieleman et al., 2016), where equivariance towards the dihedral group is theoretically guaranteed, yielding increased accuracy. Inspired by CNNs learned from alternative bases, we introduce the general principle of Frame-based convolutional networks that allow for non-orthogonal, overcomplete and steerable feature spaces.
Another way to impose structure onto CNN representations and subsequently increase their dataefficiency is to incorporate explicit geometrical transformations into them. Either by learning transformation operators and group representations (Cohen et al., 2014; Wang et al., 2009). Or by predefining the possible transformations, as done in Transforming Autoencoders (Hinton et al., 2011), which map their inputs from the image to pose space through a neural network. The Spatial Transformer Networks (Jaderberg et al., 2015) learn global transformation parameters in a similar way while applying them to a nonlinear co-registration of the feature stack to some learned pose. This yields especially high performance on tasks where centering the objects is beneficial. Dynamic Filter Networks move one step further and estimate filters for each location, conditioned on their input. These approaches are all dynamic in a sense that they condition their parameters on the input appearance. We combine the idea of Dynamic Filter Networks with explicit pose prediction into Dynamic Steerable Frame Networks that can estimate poses from continuous input space, conditioned on the input. As such, we overcome the difficulty of estimating local pose, while being able to separate pose and feature learning globally.
4 EXPERIMENTS
4.1 GENERALIZING PIXELS TO FRAMES ON CIFAR-10+
To show the validity of general frame representations, we compare different bases in a state-of-theart pre-activation deep residual network architecture (He et al., 2016) on the Cifar-10+ (Krizhevsky & Hinton, 2009) dataset with moderate data augmentation of crops and flips.
We evaluated our approach on multiple networks and network sizes. The setup used for the ResNet is as described in (He et al., 2016). The batch size is chosen to be 64 and we train for 164 epochs with the described learning rate decrease. The ResNet architectures used are without bottlenecks having 20, 56 and 110 layers. For the Densenets we follow (Huang et al., 2016) and evaluate on the K=12 and L=40, and the K=12 and L=100 models. We run our experiments in Keras (Chollet, 2015) and Tensorflow (Abadi et al., 2016). In the first experiment, we run the models on the standard pixel basis to get a viable baseline. Secondly, we replace the pixel-basis with widely-used frames that take natural image statistics into account, namely non-orthogonal, overcomplete Gaussian derivatives (Florack et al., 1992) and non-orthogonal framelets (Daubechies et al., 2003) in an alternating fashion, yielding superior performance compared to the pixel-basis by replacement only.
We also show that the naive xpyq frame (see table 1) performs consistently worse than the other two choices, as it does not take natural image properties into account, while it is important to mention that this 1% performance decrease also comes with additional properties that might be highly beneficial in particular tasks. We have also found orthogonal polynomials to not work very well (around 3% performance decrease), which is in line with our expectation that suitable frames should take natural image statistics into account. 2D frames are generated from 1D functions via the following generating process:
Frame = {v0, v1, v2, v3} ⊗ {v0, v1, v2, v3}T . (12)
The results are reported in table 2. The fact that the pixel-basis can be replaced by steerable frames and performance even improves when the frame is chosen well, is remarkable, as this means every filter in the CNN enjoys additional properties, while performance improves in the standard setting already and finding suitable frames is not more expensive than running the same smallest CNN as many times as one has frames to choose from, as the performance we observed was consistent across multiple model sizes. Frame-based CNNs run at the same runtime as vanilla CNNs.
4.2 DYNAMIC STEERABLE FRAME NETWORKS
In this section we report two experiments. The first experiment is an edge detection task, highlighting the difference between our approach and multiple baselines in a fine-grained pixel-wise labeling task. In the second experiment, we apply a 2D convolutional LSTM on a small hand gesture recognition video dataset to illustrate how the Dynamic Steerable Frame Network regularizes the model effectively and to illustrate its benefits over Spatial Transform Networks.
The model used in both experiments is learned from a steerable Gauss-Hermite frame. The Dynamic Steerable Frame Network consists of three processing steps. 1) Change to frame space on the input, 2) the Pose-Generating network estimates the pose from this transformed input, outputting a set of pose variables for each location in the image. 3) the steering functions derived in section 2.4 are applied to these pose variable maps and effectively act as nonlinear pose-parametrized activation functions that regularize the Pose-Generating network to output an explicitly interpretable pose space. Finally, a 1x1 convolution layer is applied to the already transformed output maps, representing the weights win, governing the canonical appearance of the ith feature map, see also figure 5. Dynamic Steerable Frame Networks run at the same computational cost as vanilla Dynamic Filter Networks.
4.2.1 EDGE DETECTION
In this experiment, we compare a Dynamic Filter Network (De Brabandere et al., 2016) baseline with an autoencoder and a Dynamic Steerable Frame Network on the task of edge detection. The problem is formulated as a pixel-wise classification task and reported is the root mean-squared error on an unseen test set. The labels are the edges. The dataset is infinite, as we produce random blobs and create the edge labels with a standard scikit image function. The standard DFN can freely learn an input layer with 2 filters and 3 subsequent 1x1 layers that can non-linearly recombine the inputs, whereas the Frame DFN receives a steerable frame as an input, allowing it to leverage the finegrained orientation information without the need to learn it. The Dynamic Steerable Frame Network has the exact same architecture as the DFN but is geometrically regularized on its output as can be seen in figure 5, as an input it receives a first order Gauss-Hermite frame that can be steered globally towards rotation and locally towards scale.
Location varying methods are clearly superior in this task, compared to the location invariant autoencoder. The DFN increases its performance substantially when getting the steerable frame as an input, indicating its inability to learn a continuously transforming frame by itself. Finally, the Dynamic Steerable Frame Network clearly outperforms all baselines due to its ability to continuously transform its filters in a well-regularized manner. As an extra, we get the local feature pose for free from the output of the DSFN, the baseline has no notion of an explicit pose parameter, see figure 4.
4.2.2 SMALL SCALE VIDEO CLASSIFICATION
To show the ability of the Dynamic Steerable Frame Network to effectively regularize in dynamic settings where poses play an important role and where Spatial Transformer Networks do not work well, we apply it on the task of Hand-Gesture Recognition. Namely, on the Cambridge Hand-Gesture dataset (Kim & Cipolla, 2009), consisting of 9 classes of hand movement and poses in 900 videos, we use 750 for training, 50 for validation and 100 for testing. The dataset is very small and contains classes where global movement plays an important role and thus provides a good test bed to show effectiveness of the DSFN regularization ability compared to Spatial Transformers.
Our baseline model is a convolutional LSTM with 10 output feature maps, batch normalization and a dense layer for classification. As a second baseline, we increase the capacity of the model by adding a second LSTM layer and a second batch normalization step. We combine two instances of a Spatial Transformer Network with a convolutional LSTM, one that can perform full affine transformations and one that is restricted to rotation and scaling. The DSFN module is applied to the input layer of the smaller model with 4 output feature maps. The setup of the steerable frame used in this model is a Gauss-Hermite frame.
The steerability is uniquely parametrized as: {BsBθ}. Allowing for scaling and rotation. Both Spatial Transformer Networks do not manage to learn useful warps of the input image and therefore decrease performance of the baseline. The affine model only manages to correctly classify multiple instances of a static class that has no movement information related to its label, while the rot/scale model increases performance, but still does not manage to learn useful scalings or rotations. The DSFN manages to learn locally rotation and scale invariant filters, that follow the boundaries and other features across the video as desired. Results are reported in Table 3.
For visualizations of the learned transformations, see Appendix A. The results illustrate the effectiveness of the DSFN to regularize the LSTM on a small-scale task where mostly local invariance is desired, but global invariance destroys most of the class-specific information. The DSFN improves performance over the baseline by about 22%, while the Spatial Transformer Network decreases performance by about 15%, or even to random in the full-affine case.
5 DISCUSSION
We have introduced the notion of Frame-based convolutional networks. Our experiments illustrate that a simple replacement of the standard basis by a frame suitable for natural images leads to increased performance.
The insight that multiple frames can be considered as viable spanning sets for CNN representations leads us to steerable frames whose properties we exploit explicitly in our derived Dynamic Steerable Frame Networks, such that they can readily be accessed during training. The proposed method is a hybrid of Dynamic Filter Networks and Spatial Transformer Networks, enabling locally adaptive filtering with geometrical constraints.
We illustrate the effectiveness of the approach on an edge detection task, that requires fine-grained pixel-wise labeling, where Dynamic Steerable Frame Networks outperform a standard Dynamic Filter Network and an autoencoder baseline. Further, we illustrate the ability of the Dynamic Steerable Frame Network to regularize recurrent networks in a small-data video classification scenario where Spatial Transformer Networks fail to learn meaningful transformations.
Future work is to apply the model to other problem domains like egocentric video, robotics applications, as well as volumetric medical imaging videos of moving organs. We expect our Dynamic Steerable Frame Network approach to be beneficial in any problem where spatiotemporal continuity, data-efficiency, or interpretable pose spaces are key.
ACKNOWLEDGEMENTS
We would like to thank Edouard Oyallon and Taco Cohen for insightful comments and discussions.
APPENDIX A VISUALIZING DSFN AND STN TRANSFORMATIONS
APPENDIX B EQUIVARIANCE PROOF & STEERING EQUATION DERIVATION
To prove that a frame is equivariant with respect to the action of a group transformation, determined by its generator Li, we simply have to show that:
LiΦ(x, y) = BiΦ(x, y), (13)
Where Bi is some n× n matrix. In case of the Hermite polynomials (here considered up to second order), we have:
Φ(x, y) = {1, x, y, x2 − 1, xy, y2 − 1}. (14)
To verify that these functions span an equivariant function space with respect to rotation, we apply the generator of rotations to the frame and verify that equation 13 holds. The generator of rotations
in the plane is given by Lr = −x ddy + y d dx , applied to each frame element, we get:
LrΦ(x, y) =  0 y −x 2xy
−x2 + y2 −2xy
 = Br  1 x y x2 − 1 xy
y2 − 1  . (15) It is straightforward to solve this linear system and obtain the 6× 6 matrix Br:
Br =  0 0 0 0 0 0 0 0 1 0 0 0 0 −1 0 0 0 0 0 0 0 0 2 0 0 0 0 −1 0 1 0 0 0 0 −2 0  . Thus, we have proven that the function space is closed under the action of the group and the Hermite polynomials constitute an equivariant function space with respect to rotation.
Subsequently, the exponential map directly yields the steering equations collected in the interpolation matrix Aθ, that can rotate the whole frame by θ:
Φθ(x, y) =eθBrΦ(x, y) = AθΦ(x, y),
Φθ(x, y) =  1 0 0 0 0 0 0 cos θ sin θ 0 0 0 0 − sin θ cos θ 0 0 0 0 0 0 12 + 1 2 cos 2θ sin 2θ 1 2 − 1 2 cos 2θ
0 0 0 − 12 sin 2θ cos 2θ 1 2 sin 2θ 0 0 0 12 − 1 2 cos 2θ − sin 2θ 1 2 + 1 2 cos 2θ
  1 x y x2 − 1 xy
y2 − 1
 .
Φθ(x, y) is the frame rotated by some angle θ. Combining this result with equation 7, gives us the possibility to rotate any learned feature by arbitrary and continuous angles θ. The whole procedure is completely analogous for any other Lie group transformation. Further, k-parameter transformation groups can be composed according to equation 9 from smaller groups. Here an example of the general linear group of rotation, anisotropic scalings and skew:
Φ{θ1,sx,sy,θ2}(x, y) = A{θ1,sx,sy,θ2}Φ(x, y) = eθ2Br · esxBsx · esyBsy · eθ1BrΦ(x, y). (16)
APPENDIX C BACKPROPAGATION THROUGH STEERABLE FILTERS
Will be added to final manuscript.

I sincerely apologize for the late review!
The first part has a strong emphasis on the technical part. It could benefit from some high level arguments on what the method aims to achieve, what limitation is there to overcome. I may have misunderstood the contribution (in which case please correct me) that the main novel part of the paper is the suggestion to learn the group parameterizations instead of pre-fixing them. So instead of applying it to common spatial filters as in De Brabandere et al., it is applied to Steerable Frames?
The first contribution suggests that "general frame bases are better suited to represent sensory input data than the commonly used pixel basis.". The experiments on Cifar10+ indicate that this is not true in general. Considering the basis as a hyper-parameter, expensive search has to be conducted to find that the Gauss-Frame gives better results. I assume this does not suggest that the Gauss-Frame is always better, at least there is weak evidence on a single network presented. Maybe the first contribution has to be re-stated. Further is the "Pixel" network representation corrected for the larger number of parameters. As someone who is interested in using this, what are the runtime considerations?
I would strongly suggest to improve Fig.3. The Figure uses "w" several times in different notations and depictions. It mixes boxes, single symbols and illustrative figures. It took some time to decipher the Figure and its flow.
Summary: The paper is sufficiently clear, technical at many places and readability can be improved. E.g., the introduction of frames in the beginning lacks motivation and is rather unclear to someone new to this concept. The work falls in the general category of methods that impose knowledge about filter transformations into the network architecture. For me that has always two sides, the algorithmic and technical part (there are several ways to do this) and the practical side (should I do it)? This is a possible approach to this problem but after the paper I was a bit wondering what I have learned, I am certainly not inspired based on the content of the paper to integrate or build on this work. I am lacking insights into transformational parameters that are relevant for a problem. While the spatial transformer network paper was weaker on the technical elegance side, it provided exactly this: an insight into the feature transformation learned by the algorithm. I am missing this here, e.g., from Table 2  I learn that among four choices one works empirically better. What is destroyed by the x^py^p and Hermite frames that the ResNet is *not* able to recover from? You can construct network architectures that are the superset of both, so that inferior performance could be avoided.
The algorithm is clear but it is similar to the Dynamic Filter Networks paper. And I am unfortunately not convinced about the usefulness of this particular formulation. I'd expect a stronger paper with more insights into transformations and comparisons to standard techniques, a clear delineation of when this is advised.


ADDING GRADIENT NOISE IMPROVES LEARNING FOR VERY DEEP NETWORKS
Deep feedforward and recurrent networks have achieved impressive results in many perception and language processing applications. Recently, more complex architectures such as Neural Turing Machines and Memory Networks have been proposed for tasks including question answering and general computation, creating a new set of optimization challenges. In this paper, we explore the lowoverhead and easy-to-implement optimization technique of adding annealed Gaussian noise to the gradient, which we find surprisingly effective when training these very deep architectures. Unlike classical weight noise, gradient noise injection is complementary to advanced stochastic optimization algorithms such as Adam and AdaGrad. The technique not only helps to avoid overfitting, but also can result in lower training loss. We see consistent improvements in performance across an array of complex models, including state-of-the-art deep networks for question answering and algorithm learning. We observe that this optimization strategy allows a fully-connected 20-layer deep network to escape a bad initialization with standard stochastic gradient descent. We encourage further application of this technique to additional modern neural architectures.
1 INTRODUCTION
Deep neural networks have shown remarkable success in diverse domains including image recognition (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012) and language processing applications (Sutskever et al., 2014; Bahdanau et al., 2014). This broad success comes from a confluence of several factors. First, the creation of massive labeled datasets has allowed deep networks to demonstrate their advantages in expressiveness and scalability. The increase in computing power has also enabled training of far larger networks with more forgiving optimization dynamics (Choromanska et al., 2015). Additionally, architectures such as convolutional networks (LeCun et al., 1998) and long short-term memory networks (Hochreiter & Schmidhuber, 1997) have proven to be easier to optimize than classical feedforward and recurrent models. Finally, the success of deep networks ∗First two authors contributed equally †Work was done when author was at Google, Inc.
is also a result of the development of simple and broadly applicable learning techniques such as dropout (Srivastava et al., 2014), ReLUs (Nair & Hinton, 2010), gradient clipping (Pascanu et al., 2013; Graves, 2013), optimization algorithms and weight initialization strategies (Glorot & Bengio, 2010; Sutskever et al., 2013; He et al., 2015).
Recent work has aimed to push neural network learning into more challenging domains, such as question answering or program induction. These more complicated problems demand more complicated architectures (e.g. Graves et al. (2014); Sukhbaatar et al. (2015)), thereby posing new optimization challenges. While there is very active research in improving learning in deep feedforward and recurrent networks, such as layer-wise deep supervision (Lee et al., 2015), novel activation functions (Maas et al., 2013), initialization schemes (He et al., 2015), and cell architectures (Cho et al., 2014a; Yao et al., 2015), these are not always sufficient or applicable in networks with complex structure over the latent variables. In order to achieve good performance, researchers have reported the necessity of additional techniques such as explicit labeling of latent variables (Weston et al., 2014), relaxing weight-tying constraints (Kaiser & Sutskever, 2016), warmstarts (Peng et al., 2015), random restarts, and the removal of certain activation functions in early stages of training (Sukhbaatar et al., 2015).
The recurring theme is that commonly-used optimization techniques are not always sufficient to robustly optimize the models of interest. In this work, we explore a simple technique of adding annealed Gaussian noise to the gradient, which we find to be surprisingly effective in training deep neural networks with stochastic gradient descent. While there is a long tradition of adding random weight noise in neural networks, it has been under-explored in the optimization of modern deep architectures. Furthermore, although weight and gradient noise are equivalent when using standard SGD updates, the use of adaptive and momentum based stochastic optimizers such as Adam and AdaGrad (Duchi et al., 2011; Kingma & Ba, 2014) breaks this equivalence, allowing the noise to effectively adapt to the curvature of the optimization landscape. We find this property to be important when optimizing the most complex models.
While there exist theoretical and empirical results on the regularizing effects of conventional stochastic gradient descent, especially for the minimization of convex losses (Bousquet & Bottou, 2008), we find that in practice the added noise can actually help us achieve lower training loss by encouraging active exploration of parameter space. This exploration proves especially necessary and fruitful when optimizing neural network models containing many layers or complex latent structures. For neural network learning, it has long been known that the noise in the stochastic gradient can help to escape saddle points and local optima (Bottou, 1992). For this reason, neural network practitioners sometimes avoid overly-large mini-batch sizes to achieve the best results. We find that the Gaussian noise added in our technique is complementary to the noisy stochastic gradient, and a combination of Gaussian noise and tuned mini-batch sizes is necessary for the most complex models.
The main contribution of this work is to demonstrate the broad applicability of this simple method to the training of many complex modern neural architectures. To our knowledge, neither the exponentially decayed noise schedule nor the black box combination of injected gradient noise with adaptive optimizers have been used before in the training of deep networks. We consistently see improvements from Gaussian gradient noise when optimizing a wide variety of models, including very deep fully-connected networks, and special-purpose architectures for question answering and algorithm learning. For example, this method allows us to escape a poor initialization and successfully train a 20-layer rectifier network on MNIST with standard gradient descent. It also enables a 72% relative reduction in error in question answering, and doubles the number of accurate binary multiplication models learned across 7,000 random restarts. Gradient noise also possesses attractive robustness properties. We examine only two distinct settings of the noise variance hyperparameter in total across all experiments. We additionally observe that in cases where gradient noise fails to improve over other learning techniques, it rarely significantly hurts a models ability to generalize.
We hope that practitioners will see similar improvements in their own research by adding this simple technique, implementable in a single line of code, to their repertoire.
2 RELATED WORK
Adding random noise to the weights, inputs, or hidden units has been a known technique amongst neural network practitioners for many years (e.g. Murray & Edwards; An (1996)). However, the benefits of gradient noise have not been fully explored with modern deep networks nor combined with advanced stochastic optimization techniques, which allow the noise to take into account the geometry of the optimization problem and the statistical manifold.
Weight noise (Steijvers, 1996) and adaptive weight noise (Graves, 2011; Blundell et al., 2015), which usually maintains a Gaussian variational posterior over network weights, similarly aim to improve learning by added noise during training. In adaptive weight noise, an extra set of parameters for the variance must be maintained. This adaptation is different than our use of an adaptive optimizer, as it aims to capture an accurate estimate of uncertainty in the weights and not guide the exploration of parameter space. They differ from our proposed method in that the noise is not annealed and at convergence will be non-zero.
Similarly, the technique of dropout (Srivastava et al., 2014) randomly sets groups of hidden units to zero at train time to improve generalization in a manner similar to ensembling.
An annealed Gaussian gradient noise schedule was used to train the highly non-convex Stochastic Neighbor Embedding model in Hinton & Roweis (2002). The gradient noise schedule that we found to be most effective is very similar to the Stochastic Gradient Langevin Dynamics (SGLD) algorithm of Welling & Teh (2011), who use gradients with added noise to accelerate MCMC inference for logistic regression and independent component analysis models. This use of gradient information in MCMC sampling for machine learning to allow faster exploration of state space was previously proposed by Neal (2011). However, standard SGLD analysis does not allow for the use of adaptive optimizers or momentum, limiting the efficiency for very pathological optimization landscapes. Stochastic Gradient Riemannian Langevin Dynamics (Patterson & Teh, 2013) adapts the gradient and noise using the Fisher information matrix, effectively following trajectories along the same manifold as the natural gradient (Amari, 1998), but is applied only to models for which that matrix is tractable to estimate in closed form.
Various optimization techniques have been proposed to improve the training of neural networks. Most notable is the use of momentum (Polyak, 1964; Sutskever et al., 2013; Kingma & Ba, 2014) or adaptive learning rates (Duchi et al., 2011; Dean et al., 2012; Zeiler, 2012). These methods are normally developed to provide good convergence rates for the convex setting, and then heuristically applied to nonconvex problems. Similarly, batch normalization and related methods (Ioffe & Szegedy, 2015; Arpit et al., 2016; Salimans & Kingma, 2016), natural gradient descent (Amari, 1998; Desjardins et al., 2015), and K-FAC (Martens & Grosse, 2015) can all be seen as various preconditioning methods using approximations to the inverse Fisher information of the neural network. While there has been some difficulty in combining batch normalization-type algorithms with recurrent networks (Laurent et al., 2015), recent work has had success in this area (Cooijmans et al., 2016; Ba et al., 2016).
Injecting noise in the gradient can be combined with any of the above methods, and can be seen as a complementary technique especially suitable for nonconvex problems. By adding additional artificial stochasticity to the gradient, this technique allows the model more chances to escape local minima or saddle-points (see a similar argument in Bottou (1992)), or to traverse quickly through the “transient” plateau phase of early learning (see a similar analysis for momentum in Sutskever et al. (2013)). This is born out empirically in our observation that adding gradient noise can actually result in lower training loss. In this sense, we suspect adding gradient noise is similar to simulated annealing (Kirkpatrick et al., 1983) which exploits random noise to explore complex optimization landscapes. This can be contrasted with well-known benefits of stochastic gradient descent as a learning algorithm (Robbins & Monro, 1951; Bousquet & Bottou, 2008), where both theory and practice have shown that the noise induced by the stochastic process aids generalization by reducing overfitting.
Recently, there has been a surge in research examining the use of gradient and weight noise when training deep neural networks. Mobahi (2016) present an optimization technique for recurrent networks that applies an annealed Gaussian kernel smoothing method to the loss function, of which annealed weight noise is a Monte Carlo estimator. Li et al. (2016) present a version of SGLD that
incorporates both Gaussian noise and adaptively estimated learning rates (but no momentum term). Though significantly more complex than our proposed method, the most similar work is the Santa algorithm of Chen et al. (2016). Santa combines SGLD with adaptive learning rates and adaptive per-coordinate momentum parameters, and shows that the scheme can approach global optima of the objective function under certain assumptions.
3 METHOD
We consider a simple technique of adding time-dependent Gaussian noise to the gradient g at every training step t:
gt ← gt +N(0, σ2t )
The gradient gt is then used to update the weights θt as if it were the original gradient of the loss function, and can be used with any stochastic optimization algorithm. Our experiments indicate that adding annealed Gaussian noise by decaying the variance often works better and more robustly than using fixed Gaussian noise (see Section 4.6). We use a schedule inspired from Welling & Teh (2011) in our experiments and take:
σ2t = η
(1 + t)γ (1)
We examine only 2 distinct noise hyperparameter configurations in our experiments, selecting η from {0.01, 1.0} and setting γ = 0.55 in all experiments. We believe this shows that annealed gradient noise is robust to minimal tuning. For example, in the experiments on Neural Programmer and Neural GPUs, we tried only a single configuration of noise parameters, simply setting η = 1.0 and tuning only the model hyperparameters as normal.
4 EXPERIMENTS
In the following experiments, we examine the effect of gradient noise on deep networks for MNIST digit classification, and consider a variety of complex neural network architectures: EndTo-End Memory Networks (Sukhbaatar et al., 2015) and Neural Programmer (Neelakantan et al., 2016) for question answering, Neural Random Access Machines (Kurach et al., 2016) and Neural GPUs (Kaiser & Sutskever, 2016) for algorithm learning. The models and results are described as follows.
4.1 DEEP FULLY-CONNECTED NETWORKS
For our first set of experiments, we examine the impact of adding gradient noise when training a very deep fully-connected network on the MNIST handwritten digit classification dataset (LeCun et al., 1998). Our network is deep: it has 20 hidden layers, with each layer containing 50 hidden units, posing a significant optimization and generalization problem. We use the ReLU activation function (Nair & Hinton, 2010).
In this experiment, we train with SGD without momentum, using the fixed learning rates of 0.1 and 0.01. Unless otherwise specified, the weights of the network are initialized from a Gaussian with mean zero, and standard deviation of 0.1, which we call Simple Init. When adding gradient noise, we tried both settings of the variance detailed in Section 3, and found that decaying variance according to the schedule in Equation (1) with η = 0.01 worked best.
The results of our experiment are in Table 1. When trained from Simple Init we can see that adding noise to the gradient helps in achieving higher average and best accuracy over 20 runs using each learning rate for a total of 40 runs (Table 1, Experiment 1). We note that the average is closer to 50% because the small learning rate of 0.01 usually gives very slow convergence. We also try our approach on a more shallow network of 5 layers, but adding noise does not improve the training in that case.
Next, we experiment with clipping the gradients with two threshold values: 100 and 10 (Table 1, Experiment 2, and 3). Here, we find training with gradient noise is insensitive to the gradient
clipping values. By tuning the clipping threshold, it is possible to get comparable accuracy without noise for this problem.
In our fourth and fifth experiments (Table 1, Experiment 4), we use two analytically-derived ReLU initialization techniques (which we term Good Init 1 and 2) recently-proposed by Sussillo (2014) and He et al. (2015), and find that adding gradient noise does not help. Previous work has found that stochastic gradient descent with carefully tuned initialization, momentum, learning rate, and learning rate decay can optimize such extremely deep fully-connected ReLU networks (Srivastava et al., 2015). It would be harder to find such a robust initialization technique for the more complex heterogeneous architectures considered in later sections. Accordingly, we find in later experiments (e.g., Section 4.3) that random restarts and the use of a momentum-based optimizer like Adam are not sufficient to achieve the best results in the absence of added gradient noise.
To test how sensitive the methods are to poor initialization, in addition to the sub-optimal Simple Init, we run an experiment where all the weights in the neural network are initialized at zero. The results (Table 1, Experiment 5) show that if we do not add noise to the gradient, the networks fail to learn. If we add some noise, the networks can learn and reach 94.5% accuracy. While the pessimal performance of the noiseless model is unsurprising (initializing weights at 0 introduces symmetries that make gradient-descent impossible), it is interesting to note that gradient noise can overcome what is perhaps the canonical “bad initialization.”
In summary, these experiments show that if we are careful with initialization and gradient clipping values, it is possible to train a very deep fully-connected network without adding gradient noise. However, if the initialization is poor, optimization can be difficult, and adding noise to the gradient is a good mechanism to overcome the optimization difficulty. Additionally, the noise need not be heavily tuned and rarely decreases performance.
This set of results suggests that added gradient noise can be an effective mechanism for training complex networks. This is because it is more difficult to initialize the weights properly for these
architectures. In the following, we explore the training of more complex models such as End-ToEnd Memory Networks and Neural Programmer, whose initialization is less well studied.
4.2 END-TO-END MEMORY NETWORKS
We test added gradient noise for training End-To-End Memory Networks (Sukhbaatar et al., 2015), an approach for question answering using deep networks. Memory Networks have been demonstrated to perform well on a relatively challenging toy question answering problem (Weston et al., 2015).
In Memory Networks, the model has access to a context, a question, and is asked to predict an answer. Internally, the model has an attention mechanism which focuses on the right clue to answer the question. In the original formulation (Weston et al., 2015), Memory Networks were provided with additional supervision as to what pieces of context were necessary to answer the question. This was replaced in the End-To-End formulation by a latent attention mechanism implemented by a softmax over contexts. As this greatly complicates the learning problem, the authors implement a two-stage training procedure: First train the networks with a linear attention, then use those weights to warmstart the model with softmax attention.
In our experiments with Memory Networks, we use the same model hyperparameter settings as Sukhbaatar et al. (2015), and we try both settings of the variance detailed in Section 3, finding η = 0.01 worked best for this task. This noise is added to the gradient after clipping.
We set the number of training epochs to 200 because we would like to understand the behaviors of Memory Networks near convergence. We test the effect of gradient noise with the published two-stage training approach, and additionally with a one-stage approach where we train the networks with softmax attention and without warmstarting. Following the experimental protocol of Sukhbaatar et al. (2015), we take the model with lowest training error out of 10 random restarts. Results are reported in Table 2. We find some fluctuations during each run of the training, but the reported results reflect the typical gains obtained by adding random noise.
We find that warmstarting does indeed help the networks. In all cases, adding random noise to the gradient also helps the network both in terms of training errors and validation errors, and never hurts. Added noise, however, is especially helpful for the training of End-To-End Memory Networks without the warmstarting stage.
4.3 NEURAL PROGRAMMER
Neural Programmer is a neural network architecture augmented with a small set of built-in arithmetic and logic operations that learns to induce latent programs. It is proposed for the task of question answering from tables (Neelakantan et al., 2016). Examples of operations on a table include the sum of a set of numbers, or the list of numbers greater than a particular value. Key to Neural Programmer is the use of “soft selection” to assign a probability distribution over the list of operations. This probability distribution weighs the result of each operation, and the cost function compares this weighted result to the ground truth. This soft selection, inspired by the soft attention mechanism of Bahdanau et al. (2014), allows for full differentiability of the model. Running the model for
several steps of selection allows the model to induce a complex program by chaining the operations, one after the other. At convergence, the soft selection tends to become peaky (hard selection). Figure 1 shows the architecture of Neural Programmer at a high level.
In a synthetic table comprehension task, Neural Programmer takes a question and a table (or database) as input and the goal is to predict the correct answer. To solve this task, the model has to induce a program and execute it on the table. A major challenge is that the supervision signal is in the form of the correct answer and not the program itself. The model runs for a fixed number of steps, and at each step selects a data segment and an operation to apply to the selected data segment. Soft selection is performed at training time so that the model is differentiable, while at test time hard selection is employed.
We examine only the noise configuration with η = 1.0, and add noise to the gradient after clipping, optimizing all other hyperparameters of the model. The model is optimized with Adam (Kingma & Ba, 2014), which combines momentum and adaptive learning rates.
For our first experiment, we train Neural Programmer to answer questions involving a single column of numbers. We use 72 different hyper-parameter configurations with and without adding annealed random noise to the gradients. We also run each of these experiments for 3 different random initializations of the model parameters and we find that only 1/216 runs achieve 100% test accuracy without adding noise while 9/216 runs achieve 100% accuracy when random noise is added. The 9 successful runs consisted of models initialized with all the three different random seeds, demonstrating robustness to initialization. We find that when using dropout (Srivastava et al., 2014) none of the 216 runs give 100% accuracy.
We consider a more difficult question answering task where tables have up to five columns containing numbers. We also experiment on a task containing one column of numbers and another column of text entries. Table 3 shows the performance of adding noise vs. no noise on Neural Programmer.
Figure 2 shows an example of the effect of adding random noise to the gradients in our experiment with 5 columns. The differences between the two models are much more pronounced than Table 3 indicates because that table reflects the results from the best hyperparameters. Figure 2 indicates a more typical training run.
In all cases, we see that added gradient noise improves performance of Neural Programmer. Its performance when combined with or used instead of dropout is mixed depending on the problem, but the positive results indicate that it is worth attempting on a case-by-case basis.
4.4 NEURAL RANDOM ACCESS MACHINES
We now conduct experiments with Neural Random-Access Machines (NRAM) (Kurach et al., 2016). NRAM is a model for algorithm learning that can store data, and explicitly manipulate and dereference pointers. NRAM consists of a neural network controller, memory, registers and a set of built-in operations. This is similar to the Neural Programmer in that it uses a controller network to compose built-in operations, but both reads and writes to an external memory. An operation can either read (a subset of) contents from the memory, write content to the memory or perform an arithmetic operation on either input registers or outputs from other operations. The controller runs for a fixed number of time steps. At every step, the model selects a “circuit” to be executed: both the operations and its inputs.
These selections are made using soft attention (Bahdanau et al., 2014) making the model end-to-end differentiable. NRAM uses an LSTM (Hochreiter & Schmidhuber, 1997) controller. Figure 3 gives an overview of the model.
For our experiment, we consider a problem of finding the k-th element’s value in a linked list. The network is given a pointer to the head of the linked list, and has to find the value of the k-th element. Note that this is highly nontrivial because pointers and their values are stored at random locations in memory, so the model must learn to traverse a complex graph for k steps.
Because of this complexity, training the NRAM architecture can be unstable, especially when the number of steps and operations is large. We once again experiment with the decaying noise schedule from Equation (1), setting η = 0.01. We run a large grid search over the model hyperparameters (detailed in Kurach et al. (2016)), and find the top 3 parameter settings separately for both noised
and un-noised models. For each model, for each of these 3 settings, we try 100 different random initializations and look at the percentage of runs that give 100% accuracy across each one for training both with and without noise.
As in our experiments with Neural Programmer, we find that adding the noise after gradient clipping is crucial. This is likely because the effect of random noise is washed away when gradients become too large. For models trained with noise we observed much better reproduce rates, which are presented in Table 4. Although it is possible to train the model to achieve 100% accuracy without noise, it is less robust across multiple random restarts, with over 10x as many initializations leading to a correct answer when using noise.
4.5 CONVOLUTIONAL GATED RECURRENT NETWORKS (NEURAL GPUS)
Convolutional Gated Recurrent Networks (CGRN) or Neural GPUs (Kaiser & Sutskever, 2016) are a recently proposed model that is capable of learning arbitrary algorithms. CGRNs use a stack of convolution layers, unfolded with tied parameters like a recurrent network. The input data (usually a list of symbols) is first converted to a three dimensional tensor representation containing a sequence of embedded symbols in the first two dimensions, and zeros padding the next dimension. Then, multiple layers of modified convolution kernels are applied at each step. The modified kernel is a combination of convolution and Gated Recurrent Units (GRU) (Cho et al., 2014b). The use of convolution kernels allows computation to be applied in parallel across the input data, while the gating mechanism helps the gradient flow. The additional dimension of the tensor serves as a working memory while the repeated operations are applied at each layer. The output at the final layer is the predicted answer.
The key difference between Neural GPUs and other architectures for algorithmic tasks (e.g., Neural Turing Machines (Graves et al., 2014)) is that instead of using sequential data access, convolution kernels are applied in parallel across the input, enabling the use of very deep and wide models. The model is referred to as Neural GPU because the input data is accessed in parallel. Neural GPUs were shown to outperform previous sequential architectures for algorithm learning on tasks such as binary addition and multiplication, by being able to generalize from much shorter to longer data cases.
In our experiments, we use Neural GPUs for the task of binary multiplication. The input consists two concatenated sequences of binary digits separated by an operator token, and the goal is to multiply the given numbers. During training, the model is trained on 20-digit binary numbers while at test time, the task is to multiply 200-digit numbers. We add Gaussian noise with decaying variance according to the schedule in Equation (1) with η = 1.0, to the gradient after clipping. The model is optimized using Adam (Kingma & Ba, 2014).
Table 5 gives the results of a large-scale experiment using Neural GPUs with a 7290 grid search. The experiment shows that models trained with added gradient noise are more robust across many random initializations and parameter settings. As you can see, adding gradient noise both allows us to achieve the best performance, with the number of models with < 1% error over twice as large as without noise. But it also helps throughout, improving the robustness of training, with more models training to higher error rates as well. This experiment shows that the simple technique of added gradient noise is effective even in regimes where we can afford a very large numbers of random restarts.
4.6 DISCUSSION
In this work we propose an annealed Gaussian gradient noise scheme for the optimization of complex neural networks. Our experiments show improvement from gradient noise on a variety of models. We conduct a small set of additional experiments below to examine the factors that make this technique successful, and report a failure mode.
Annealed vs. fixed noise We use a single fixed decay value γ = 0.55 when applying Equation (1) in our experiments, inspired by Stochastic Gradient Langevin Dynamics, and recommend it as a default. We conduct several experiments to determine the importance of annealed vs. fixed noise added to the gradient. We find that for the End2End model, similar results can be achieved with fixed noise values, however requiring significantly more tuning (compared to trying only two different values of η in our experiments with annealed noise). We achieve nearly identical results on the End2End experiment using a fixed noise value of η = 0.001. We also experiment with fixed noise on the Neural Programmer and NRAM models, and find that they make a larger difference. For both models, we select fixed noise values log-uniformly from between 1e−4 and 0.1 and optimize the other hyperparameters. Using 216 runs per variance setting, the best Neural Programmer models without annealing can achieve equivalent errors to the annealed models. However, only 5/216 achieve the best error compared to 9/216 for the model using annealing. For NRAM, using 180 runs per setting, fixed noise never achieves the perfect error of 0 that is achieved by the annealed model. While annealing shows the most benefit with the most complex models, we generally recommend it as a robust default that requires less hyperparameter tuning than fixed noise.
Gaussian noise vs. gradient stochasticity We assert that gradient noise helps the model explore the optimization landscape, escaping saddle points and local minima. Analysis of SGD for neural networks suggests that the stochasticity of the gradient serves much the same purpose (Bottou, 1992). This suggests a strategy: add noise to the gradient by simply reducing the minibatch size, increasing the variance of the gradient estimator. While arguments based on SGLD and kernel smoothing provide evidence that the specific form of the Gaussian noise is important, we run a pair of small experiments. For both Neural Programmer and NRAM, we tried batch sizes of 10, 25, and 50 (50 being the value used in the best results). For NRAM, after 100 tasks at each batch size and no gradient noise, 2 tasks at batch size 50 converged to 0 error, 1 task at batch size 10, and none at batch size 25. For Neural Programmer, over 216 experiments at each batch size we see none of the models without gradient noise converge to the best error. These results are far worse than our results using added noise, indicating that merely lowering the batch size does not introduce the same sort of helpful stochasticity.
Gradient noise vs. weight noise While weight noise is relatively well-known, it is not equivalent to gradient noise in the case of adaptive or momentum-based optimizers, which effectively adapt the noise to the curvature of the optimization landscape. Both Neural Programmer and NRAM are greatly helped in training by the use of the Adam algorithm for optimization. We find here, using the same experimental setup as when examining annealed vs. fixed noise, that the models fail to learn when adding noise directly to the weights. Even when using starting noise rates as low as 1e−6, with the usual annealing schedule, the models fail to train significantly, achieving 57% error for NRAM and 68% for Neural Programmer at the lowest. Importantly, these noise rates are on the same order as the adaptive learning rates. This indicates that the issue is not just the noise scale, but that the very poor conditioning of the loss functions makes it necessary to adapt the noise. Similar concerns motivated the development of very recent algorithms for preconditioned SGLD in the Bayesian setting (Li et al., 2016).
Negative results While we see improvements on a large number of neural network architectures, we note a case where gradient noise does not improve over standard SGD. We conduct language modeling experiments on the Penn Treebank (Marcus et al., 1993), using the experimental setup and architecture from Zaremba et al. (2014). We report results using a 200-unit LSTM with dropout, but observe a similar lack of improvement from gradient noise when using models without dropout. We try the two proposed noise rates from Section (method) and find the best results using η = 0.01 are slightly worse than the noiseless model, achieving a perplexity of 98 rather than 95. By further lowering the noise parameter to η = 0.001 we are able to achieve the same perplexity as the baseline, but do not see improvement. While adding gradient noise does not help in this case, it is simple to try and does not significantly hurt the results.
5 CONCLUSION
In this paper, we demonstrate the effectiveness of adding noise to the gradient when training deep neural networks. We find that adding noise to the gradient helps optimization and generalization of complicated neural networks and is compatible with and complementary to other stochastic optimization methods. We suspect that the effects are pronounced for complex models because they have many saddle points.
We believe that this surprisingly simple yet effective idea, essentially a single line of code, should be in the toolset of neural network practitioners when facing issues with training neural networks.

The authors propose to add noise to the gradients computed while optimizing deep neural networks with stochastic gradient based methods. They show results multiple data sets which indicate that the method can counteract bad parameter initialization and that it can be especially beneficial for training more complicated architectures.
The method is tested on a multitude of different tasks and architectures. The results would be more convincing if they would be accompanied by confidence intervals but I understand that some of the experiments must have taken very long to run. I like that the results include both situations in which the gradient noise helps a lot and situations in which it doesn’t seem to add much to the other optimization or initialization tools employed. The quantity of the experiments and the variety of the models provide quite convincing evidence that the effect of the gradient noise generalizes to many settings. The results were not always that convincing. In Section 4.2, the method only helped significantly when a sub-optimal training scheme was used, for example. The results on MNIST are not very good compared to the state-of-the-art. Since the method is so simple, I was hoping to see more theoretical arguments for its usefulness. That said, the experimental investigations into the importance of the annealing procedure, the comparison with the effect of gradient stochasticity and the comparison with weight noise, provide some additional insight.
The paper is well written and cites relevant prior work. The proposed method is described clearly and concisely, which is to be expected given its simplicity.
The proposed idea is not very original. As the authors acknowledge, very similar algorithms have been used for training and it is pretty much identical to simulating Langevin dynamics but with the goal of finding a single optimum in mind rather than approximating an expected value. The work is the evaluation of an old tool in a new era where models have become bigger and more complex.
Despite the lack of novelty of the method, I do think that the results are valuable. The method is so easy to implement and seems to be so useful for complicated model which are hard to initialize, that it is important for others in the field to know about it. I suspect many people will at least try the method. The variety of the architectures and tasks for which the method was useful suggests that many people may also add it to their repertoire of optimization tricks.
Pros:
* The idea is easy to implement.
* The method is evaluated on a variety of tasks and for very different models.
* Some interesting experiments which compare the method with similar approaches and investigate the importance of the annealing scheme.
* The paper is well-written.
Cons:
* The idea is not very original.
* There is no clear theoretical motivation of analysis.
* Not all the results are convincing.