Convolutional Neural Networks have proved to be very efficient in image and audio processing. Their success is mostly attributed to the convolutions which utilize the geometric properties of a low dimensional grid structure. This paper suggests a generalization of CNNs to graph-structured data with varying graph structure, that can be applied to standard regression or classification problems by learning the graph structure of the data. We propose a novel convolution framework approach on graphs which utilizes a random walk to select relevant nodes. The convolution shares weights on all features, providing the desired parameter efficiency. Furthermore, the additional computations in the training process are only executed once in the pre-processing step. We empirically demonstrate the performance of the proposed CNN on MNIST data set, and challenge the state-of-the-art on Merck molecular activity data set.
1 INTRODUCTION
Convolutional Neural Networks (CNNs) (LeCun et al., 1998) are variants of multi-layer perceptrons that have been inspired by biological cells in the visual cortex. The cells act as local filters over the input space and are well-suited to exploit the strong local spatial correlation present in natural images (Hubel & Wiesel, 1968). In recent years, following a breakthrough by Krizhevsky et al. (2012) at the 2012 ImageNet challenge, CNN has repeatedly demonstrated significant improvements in a large number of computer vision problems.
The major success of CNN for visual data is justly credited to the convolution. But its strength is dependent on three crucial underlying attributes found in visual data.
1. Local connectivity assumption: The signal in visual data tends to be highly correlated in local regions, and mostly uncorrelated in global regions.
2. Shared weights assumption: The same convolution is globally valid across the image, resulting in a significant parameter reduction.
3. Grid structure of the image: Enabling a straight forward re-scaling of the feature layers through the process of max pooling.
These assumptions make it challenging to duplicate the success of CNN on a different data structure. Nevertheless, CNNs have also proved effective for non-image data, usually relying on the grid structure of the inputs. Results on acoustic data (Hinton et al., 2012), videos (Le et al., 2011) and even Go board (Silver et al., 2016) indicate that it might be sensible to generalize CNN on other data structures that lack the under-lying grid structure.
The main contribution of this work is a generalization of CNNs to general graph-structured data, directed or undirected, offering a single method that incorporates the structural information present in the graph of the features into supervised learning algorithms. Due to the active research on learning the graph structure of features, this proves to be quite a general framework. As demonstrated by the examples, large number of standard continuous regression and classification problems fall within the
scope of this paper, by first estimating the graph structure of the data and then applying the proposed CNN on it.
The main hurdle for generalizing CNNs to graph-structured data is to find a corresponding generalized convolution operator. We first consider a random walk on the graph in order to select the top k neighbors for every node during the pre-processing step, as Figure 1 shows. Then during the training process, the convolution is performed as an inner product of the weights and the selected top neighbors of the corresponding node in the preference order. Thus the weights are shared by each node and reflect the dependency between each node and its closest neighbors. When an image is considered as an undirected graph, this convolution operation is the same as the standard convolution. The proposed convolution is also applicable when the graph structure varies between observations.
In order to demonstrate the potential of the suggested method, we perform a set of experiments on the Merck molecular activity challenge and the MNIST data sets. The Merck molecular activity challenge data can be seen as a standard regression problem with significant correlation between the features. Essentially, for any regression or classification problem, the data can be visualized as a graph and its correlation matrix can be used to learn the corresponding graph structure. By treating the data as a graph, we show that a simple application of the graph convolutional neural network gives results that are comparable to state-of-the-art models.
2 LITERATURE REVIEW
Graph theory has been heavily studied in the last few decades, both from mathematical and statistical/computational perspectives, with a large body of algorithms developed for a variety of problems. Despite that, research on algorithms that incorporate CNNs with graph structured-data is still emerging. The idea of extending CNN to graph-structured data was recently explored by Bruna et al. (2013) and Henaff et al. (2015). They suggested two solutions. The first uses multi-scale clustering to define the network architecture, with the convolutions being defined per cluster without any weight sharing. The second defines the convolution through the eigen-values of the graph Laplacian, weighting out the distance induced by the graph’s similarity matrix. The drawback of the methods is that there is no easy way to induce weight sharing among the different nodes of the graph. Also, these methods only handle inputs of a fixed size as the graph structure is fixed.
Standard CNN architectures use a fixed-dimensional input which makes it difficult to apply them on data with changing graph-structure. Recently, Kalchbrenner et al. (2014) developed a CNN for modeling sentences of varying lengths. Another interesting example of a convolution over a changing graph structure has recently been suggested by Duvenaud et al. (2015).
Several deep neural networks have been suggested in the past for predicting the properties of molecules (for example, Glen et al. (2006) and Lusci et al. (2013)). One of the proposed ideas is to extract features from the molecular structure into a fixed-dimensional feature vector and then use it
as an input in a machine learning method. Specifically, Duvenaud and Maclaurin Duvenaud et al. (2015), propose a neural network to extract features or molecular fingerprints from molecules that can be of arbitrary size and shape. Their neural network consists of layers which are local filters being applied to all the nodes and its neighbors. After using several such convolutional layers to create representations of the original data, they apply a global pooling step to features and feed that into a standard classifier. However, this method is limited in its ability to propagate information across the graph, limited by the depth of the network in its pooling stage.
The problem of selecting nodes for a convolution on a graph is a particular instance of the problem of selecting local receptive fields in a general neural network. The work of Coates & Ng (2011) suggest to select the local receptive fields in a general neural network according to closest neighbors induced by the similarity matrix.
In contrast to previous research, we suggest a novel efficient convolution that captures the local connectivity reflected in the graph structure. The convolution weights are shared among the different nodes and can even be applied to changing graph structures. We do so by considering the closest neighbors obtained in a random walk, using information contained in the similarity matrix.
3 GRAPH CONVOLUTIONAL NEURAL NETWORK
The key step which differentiates CNNs on images from regular neural networks, is the selection of neighbors on the grid in a k× k window combined with the shared weight assumption. We propose a convolution operator analogous to the convolution performed on images in standard CNNs. In order to select the local neighbors of a given node, we use the graph transition matrix and calculate the expected number of visits of a random walk starting from the given node. The convolution would then be applied on the nodes being visited the most. In this section we discuss the application of the convolution in a single layer on a single graph. It is immediate to extend the definition to more complex structures, and it will be explicitly explained in 3.4. We introduce some notation in order to proceed into further discussion.
Notation: Let G = (V, E) be a graph over a set of N features, V = (X1, . . . , XN ), and a set of edges E . Let P denote the transition matrix of a random walk on the graph, such that Pij is the probability to move from node Xi to Xj . Let the similarity matrix and the correlation matrix of the graph be given by S and R respectively. Define D as a diagonal matrix where Dii = ∑ j Sij .
3.1 TRANSITION MATRIX AND EXPECTED NUMBER OF VISITS
This work assumes the existence of the graph transition matrix P . This is not a restriction. If graph structure of the data is already known, i.e. if the similarity matrix S is already known, then the transition matrix can be obtained, as explained in Lovász et al. (1996), by
P = D−1S. (1)
If the graph structure is unknown, it can be learned using several unsupervised or supervised graph learning algorithms. Learning the data graph structure is an active research topic and is not in the scope of this paper. The interested reader can start with Belkin & Niyogi (2001), and Henaff et al. (2015) discussing similarity matrix estimation. We use the absolute value of the correlation matrix as the similarity matrix, following Roux et al. (2008) who showed that correlation between the features is usually enough to capture the geometrical structure of images. That is, we assume
Si,j = |Ri,j | ∀ i, j. (2)
Once we derive the transition matrix P , we define Q(k) := ∑k i=0 P
k, where [P k]ij is the probability of transitioning from Xi to Xj in k steps. That is,
Q(0) = I, Q(1) = I + P, · · · , Q(k) = k∑ i=0 P k. (3)
Note that Q(k)i,j is also the expected number of visits to node Xj starting from Xi in k steps. The ith row, Q(k)i,· provides a measure of similarity between node Xi and its neighbors by considering
a random walk on the graph. As k increases we incorporate neighbors further away from the node, while the act of summation still gives proper weights to the node itself and its closest neighbors. Figure 2 provides a visualization of the matrix Q over the 2-D grid.
To the best of our knowledge, this is the first use of the expected number of visits on a graph to select neural nets architecture. Coates & Ng (2011) and others suggest using the similarity matrix. This definition extends the notion of the similarity matrix, since Q(1) agrees with the variable order induced by the similarity matrix. Furthermore, higher powers of k emphasize more on the graph structure of the data, giving major hubs more weight. This might be valuable, for example, in social network data.
3.2 CONVOLUTIONS ON GRAPHS
As discussed earlier, each row of Q(k) can be used to obtain the closest neighbors of a node. Hence it seems natural to define the convolution over the graph node Xi using the i’th row of Q(k). In order to do so, we denote π(k)i as the permutation order of the i
th row of Q(k) in descending order. That is, for every i = 1, 2, ..., N and every k,
π (k) i : {1, 2, ..., N} −→ {1, 2, ..., N},
such that Q i,π
(k) i (1)
> Q i,π
(k) i (2)
> ... > Q i,π
(k) i (N)
.
The notion of ordered distance between the nodes is a global feature of all graphs and nodes. Therefore, we can take advantage of it to satisfy the desired shared weights assumption. We define Conv1, as the size p convolution over the graph G with nodes x ∈ RN and weights w ∈ Rp, for the p nearest neighbors of each node, as the inner product:
Conv1(x) =  x π (k) 1 (1) · · · x π (k) 1 (p) x π (k) 2 (1) · · · x π (k) 2 (p) ... . . .
... x π (k) N (1) · · · x π (k) N (p)  ·  w1 w2 ... wp  , where x =  x1 x2 ... xN  (4) The order of the weights follows from the distance induced by the transition matrix. That is, w1 will be convoluted with the variable which has the largest value in each row according to the matrix Q(k). For example, when Q(1) = I + P , w1 will always correspond to the node itself, and w2 will correspond to the node’s closest neighbor. For higher values of k, the order will be defined by the unique graph structure. An interesting attribute of this convolution, as compared to other convolutions on graphs is that, it preserves locality while still being applicable over different graphs with different structures.
It should be noted that Conv1 is susceptible to the effects of negative correlation between the features, and does not take into account the actual distance between the nodes (it only uses that for the selection of the closest neighbors of a node). Since the weights are being learned globally, in order to account for that, we have also defined Conv2 as:
Conv2(x) =  y 1,π (k) 1 (1) · · · y 1,π (k) 1 (p) y 2,π (k) 2 (1) · · · y 2,π (k) 2 (p) ... . . .
... y N,π
(k) N (1)
· · · y N,π
(k) N (p)
 ·  w1 w2 ... wp  , (5)
where x =  x1 x2 ... xN  and yij = sign(Rij) Qij xj , ∀ i = 1, ..., N, j = 1, ..., N. In practice Conv2 performs slightly better than Conv1, although the major differences between them are mostly smoothed out during the training process.
An important feature of the suggested convolution is the operation complexity. For a graph with N nodes, a single p level convolution only requires O(N · p) operations, where p is a very small
natural number (the number of neighbors considered). The major computational effort goes in the computation of Q which is being done once per graph structure in the pre-processing step.
3.3 SELECTION OF THE POWER OF Q (K)
The selection of the value of k is data dependent, as with every hyper-parameter. But there are two main components affecting its value. Firstly, it is necessary for k to be large enough to detect the top p neighbors of every node. If the transition matrix P is sparse, it might require higher values of k. Secondly, from properties of stochastic processes, we know that if we denote π as the Markov chain stationary distribution, then
lim k→∞
Q (k) ij
k = πj ∀ i, j. (6)
This implies that for large values of k, local information will be smoothed out and the convolution will repeatedly be applied on the features with maximum connections. For this reason, we suggest the value of k to be kept relatively low (but large enough to capture sufficient amount of features, when needed).
3.4 IMPLEMENTATION
Similar to standard convolution implementation (Chellapilla et al., 2006), it is possible to represent the graph convolution as a tensor dot product, transferring most of the computational burden to the GPU while using highly optimized matrix product libraries.
For every graph convolution layer, we have as an input a 3D tensor of observations, their features and depth. We first extend the input with an additional dimension that includes the top p neighbors of each feature selected by Q(k), transforming the input dimension from 3D to 4D tensor as
(Observations, Features, Depth)→ (Observations, Features, Neighbors, Depth) .
Now for every graph convolution layer, the weights are a 3D tensor with the dimension of (Neighbors, Depth, Filters). Therefore application of a graph convolution which is a tensor dot product between the input and the weights, along the (Neighbors, Depth) axes, results with an output dimension:(
(Observations, Features) , (Neighbors, Depth) ) • ( (Neighbors, Depth) , (Filters) ) = (Observations, Features, Filters) .
Implementation of the algorithm has been done using Keras and Theano libraries in Python, inheriting all the tools provided by the libraries to train neural networks, such as dropout regularization, advanced optimizers and efficient initialization methods. Source code will be publicly available prior to the ICLR conference on the authors’ website.
4 EXPERIMENTS
In order to test the feasibility of the proposed CNN on graphs, we have conducted experiments on well known data sets functioning as benchmarks - Merck molecular activity challenge and MNIST. Both the data sets are popular and well-studied challenges in computational biology and computer vision respectively.
In all the implementations we kept the architecture shallow and simple, instead of deep and complex. This was done to enable better comparisons between the models, and reduce the chance of over-fitting the test set by the model selection process. The hyper-parameters were chosen arbitrarily when possible rather than being tuned and optimized. Nevertheless, we still report state-of-the-art, or competitive results on the experimented data sets.
In this section, we denote a graph convolution layer with k feature maps by Ck and a fully connected layer with k hidden units by FCk.
4.1 MERCK MOLECULAR ACTIVITY CHALLENGE
The Merck molecular activity is a Kaggle 1 challenge which is based on 15 molecular activity data sets. The target is to predict activity levels for different molecules based on the structure between the different atoms in the molecule. This helps to identify molecules in medicines which hit the intended target and do not cause side effects.
Following Henaff et al. (2015), we apply our algorithm on the DPP4 dataset. DPP4 contains 6148 training and 2045 test molecules. Some of the features of the molecules are very sparse and are only active in a handful number of molecules. For these features, the correlation estimation is not very accurate. Therefore we use features that are active in at least 20 molecules (observations). This results in 2153 features. As can be seen in Figure 3, there is significant correlation structure between different features. This implies strong connectivity among the features which is important for the application of the proposed method.
The training was performed using Adam optimization procedure (Kingma & Ba, 2014) where the gradients are derived from back-propagation algorithm. We used learning rate, α = 0.001, fixed the number of epochs to 40 and implemented dropout regularization on every layer during the optimization procedure. The correlation matrix absolute values were used to learn the graph structure. We found that a small number of nearest neighbors (p) between 5 to 10 works the best, and used p = 5 in all models.
Since this is a regression problem, we used the root mean-squared error loss (RMSE). Following the standard set by the Kaggle challenge, results are reported in terms of the squared correlation (R2),
1Challenge website is https://www.kaggle.com/c/MerckActivity
that is, R2 = Corr(Y, Ŷ )2,
where Y is the actual activity level and Ŷ is the predicted one.
The convergence plot given in Figure 3 demonstrates convergence of the selected architectures. The contribution of the suggested convolution is explained in view of the alternatives:
• Fully connected Neural Network: Models first applying convolution, followed by fully connected hidden layer converge better than more complex fully connected models. Furthermore, convergence is more stable in comparison to the fully connected models, due to the parameter reduction.
• Linear Regression: Optimizing over the set of convolutions is often considered as automation of the feature extraction process. From that perspective, a simple application of one layer of convolution, followed by linear regression, significantly outperforms the results of a standalone linear regression.
Table 1 provides more thorough R2 results for the different architectures explored, and compares it to two of the winners of the Kaggle challenge, namely the Deep Neural Network and the Random forest in Ma et al. (2015). We perform better than both the winners of the Kaggle contest. Since the challenge is already over, and we had full access to the test set, the results should mostly be considered as a proof of concept.
The models in Henaff et al. (2015) and Bruna et al. (2013) use a spectral approach, and currently hold the state-of-the-art. In comparison to them, we perform better than the Spectral Networks CNN on unsupervised graph structure, which is equivalent to what was done by using the correlation matrix as similarity matrix. The one using Spectral Networks on supervised graph structure holds the state-of-the-art by learning the graph structure. This is a direction we have not yet explored, as graph learning is beyond the scope of this paper, although it will be straightforward to apply the proposed graph CNN in a similar way to any learned graph.
4.2 MNIST DATA
The MNIST data often functions as a benchmark data set to test new machine learning methods. We have experimented with two different graph structures for the images. First, we considered the images as observations from an undirected graph on the 2-D grid, where each pixel is connected to its 8 adjunct neighbors pixels. We used the convolutions over the grid structure as presented in Figure 2, and Q(3) with p = 25 as the number of nearest neighbors. Due to the symmetry of the graph in most regions of the image, many pixels has equal distance from the pixel being convoluted. If ties were broken in a consistent manner, this example would be reduced to the regular convolution on a 5× 5 window for exactly the entire space but pixels 3 steps away from the boundary. In order to make the example more compelling, we have broken ties arbitrarily, making the training process harder compared to regular CNN. Imitating LeNet, with C20, Pooling(2×2), C50, Pooling(2×2) , FC100
followed by a linear classifier, resulted with 1.1% error rate. This is a worse than a regular CNN which achieves with similar architecture around 0.75% error rate, and better than a fully connected neural network which achieves around 1.4%, as expected from the complexity differences of the models.
Second, we used the correlation matrix to estimate the graph structure directly from the pixels. Since some of the MNIST pixels are constant (e.g the corners are always black), we restricted the data only to the active 717 pixels not constant. We used Q(1) with p = 6 as the number of neighbors. This was done in order to ensure that the spatial structure of the image no longer effect the results. With only 6 neighbors, and a partial subset of the pixels, the relative location of the top correlated pixels necessary varies per pixel. As a result, regular CNN are no longer applicable on the data, and we have compared the performance to fully connected Neural Networks.
Table 2 present the experiment results. During training a dropout rate of 0.2 has been applied on all layers to prevent overfitting. In all the experiments the final layer is the standard softmax logistic regression classifier. The Graph CNN perform on par with fully connected neural networks, with lower number of parameters. Also a single layer of graph convolution, followed by logistic regression greatly improve the performance of logistic regression, demonstrating the potential of the graph convolution for feature extraction purposes. As with regular convolutions, C20 − FC512 had over 7M parameters, due to the fact that the convolution uses small amount of parameters to generate different maps of the input. This implies that the graph convolution might be even more effective with the development of an efficient pooling methods on graphs, a problem that will be covered in future research.
5 CONCLUSIONS
We suggest a method to address the problem of supervised learning over graph-structured data, by extending convolutional neural networks to graph input. Our main contribution is a new way to define a convolution over graph that can handle different graph structures as its input. The convolution can be applied on standard regression or classification problems by learning the graph structure in the data, using the correlation matrix, or other methods. Compared to a fully connected layer, the suggested convolution has significantly lower number of parameters, while providing stable convergence and comparable performance. We validated and demonstrated the predictive performance of our proposed method on benchmark machine learning data sets such as: the Merck Molecular Activity data set and MNIST data.
Convolutional Neural Networks have already revolutionized the field of computer vision, speech recognition and language processing. We think an important step forward is to extend it to all other problems which have an inherent graph structure within them.
ACKNOWLEDGMENTS
We would like to thank Alessandro Rinaldo, Ruslan Salakhutdinov and Matthew Gormley for suggestions, insights and remarks that has greatly improved the quality of this paper.

Update: I thank the authors for their comments! After reading them, I decided to increase the rating.
This paper proposes a variant of the convolution operation suitable for a broad class of graph structures. For each node in the graph, a set of neighbours is devised by means of random walk (the neighbours are ordered by the expected number of visits). As a result, the graph is transformed into a feature matrix resembling MATLAB’s/Caffe’s im2col output. The convolution itself becomes a matrix multiplication.
Although the proposed convolution variant seems reasonable, I’m not convinced by the empirical evaluation. The MNIST experiment looks especially suspicious. I don’t think that this dataset is appropriate for the demonstration purposes in this case. In order to make their method applicable to the data, the authors remove important structural information (relative locations of pixels) thus artificially increasing the difficulty of the task. At the same time, they are comparing their approach with regular CNNs and conclude that the former performs poorly (and does not even reach an acceptable accuracy for the particular dataset).
I guess, to justify the presence of MNIST (or similar datasets) in the experimental section, the authors should modify their method to incorporate additional graph structure (e.g. relative locations of nodes) in cases when the relation between nodes cannot be fully described by a similarity matrix.
I believe, in its current form, the paper is not yet ready for publication but may be later resubmitted to a workshop or another conference after the concern above is addressed.


SOFTTARGET REGULARIZATION AN EFFECTIVE TECHNIQUE TO REDUCE OVER-FITTING IN NEURAL NETWORKS
Deep neural networks are learning models with a very high capacity and therefore prone to over-fitting. Many regularization techniques such as Dropout, DropConnect, and weight decay all attempt to solve the problem of over-fitting by reducing the capacity of their respective models (Srivastava et al., 2014), (Wan et al., 2013), (Krogh & Hertz, 1992). In this paper we introduce a new form of regularization that guides the learning problem in a way that reduces over-fitting without sacrificing the capacity of the model. The mistakes that models make in early stages of training carry information about the learning problem. By adjusting the labels of the current epoch of training through a weighted average of the real labels, and an exponential average of the past soft-targets we achieved a regularization scheme as powerful as Dropout without necessarily reducing the capacity of the model, and simplified the complexity of the learning problem. SoftTarget regularization proved to be an effective tool in various neural network architectures.
Deep neural networks are learning models with a very high capacity and therefore prone to over-fitting. Many regularization techniques such as Dropout, DropConnect, and weight decay all attempt to solve the problem of over-fitting by reducing the capacity of their respective models (Srivastava et al., 2014), (Wan et al., 2013), (Krogh & Hertz, 1992). In this paper we introduce a new form of regularization that guides the learning problem in a way that reduces over-fitting without sacrificing the capacity of the model. The mistakes that models make in early stages of training carry information about the learning problem. By adjusting the labels of the current epoch of training through a weighted average of the real labels, and an exponential average of the past soft-targets we achieved a regularization scheme as powerful as Dropout without necessarily reducing the capacity of the model, and simplified the complexity of the learning problem. SoftTarget regularization proved to be an effective tool in various neural network architectures.
1 INTRODUCTION
Many regularization techniques have been created to rectify the problem of over-fitting in deep neural networks, but the majority of these methods reduce models capacities to force them to learn general enough features. For example, Dropout reduces the amount of learn-able parameters by randomly dropping activations, and DropConnect extends this idea by randomly dropping weights (Srivastava et al., 2014), (Wan et al., 2013). Weight decay regularization reduces the capacity of the model, not by dropping learn-able parameters, but by reducing the space of viable solutions (Krogh & Hertz, 1992).
1.1 MOTIVATION
Hinton has shown that soft-labels, or labels predicted from a model contain more information that binary hard labels due to the fact that they encode similarity measures between the classes (Hinton et al., 2015). Incorrect labels tagged by the model describe co-label similarities, and these similarities should be evident in future stages of learning, even if the effect is diminished. For example, imagine training a deep neural net on a classification dataset of various dog breeds. In the initial few stages of learning the model will not accurately distinguish between similar dog-breeds such as a Belgian Shepherd versus a German Shepherd. This same effect, although not so exaggerated, should appear in later stages of training. If, given an image of a German Shepherd, the model predicts the class German Shepherd with a high-accuracy, the next highest predicted dog should still be a Belgian Shepherd, or a similar looking dog. Over-fitting starts to occur when the majority of these co-label effects begin to disappear. By forcing the model to contain these effects in the later stages of training, we reduced the amount of over-fitting.
1.2 METHOD
Consider the standard supervised learning problem. Given a dataset containing inputs and outputs, X and Y , a regularization function R and a model prediction function F we attempted to minimize the loss function L given by:
L(X,Y ) = 1 N N∑ i=0 Li(F(Xi,W), Yi) + λR(W) (1)
where W are the weights in F that are adjusted to minimize the loss function, and λ controls the effect of the regularization function. For our method to fit into the supervised learning scheme we altered the optimization problem by adding a time dimension (t) to the loss function:
Lt(X,Y ) = 1 N N∑ i=0 Lti(F(Xi,W), Yi) + λR(W) (2)
SoftTarget regularization requires into two steps: first, we kept an exponential moving average of past labels Ŷ t, and second, we updated the current epochs label Y tc through a weighted average of the exponential moving average of past labels and of the true hard labels:
Ŷ t = βŶ t−1 + (1− β)F(Xi,W) (3) Y tc = γŶ t + (1− γ)Y (4)
Here, γ and β are hyper-parameters that can be tuned to specific applications. The loss function then becomes:
Lt(X,Y ) = 1 N N∑ i=0 Lti(F(Xi,W), Y tc ) + λR(W) (5)
The algorithm also contains a ‘burn-in’ period, where no SoftTarget regularization is done and the model is trained freely in order to learn the basic co-label similarities. We will denote the number of epochs trained freely as nb, and the total number of epochs as n. Experimentally we also discovered that it is sometimes best to run the network for more than one epoch on a single Yc, so we will denote nt as the number of epochs per every time-step. We have provided the pseudo-code in Algorithm 1.
Algorithm 1 SoftTarget Regularization input: X,Y,F ,G, β, γ, nb, nt, n
F ← G(F , {X,Y }, nb)
Ŷ 0 ← F(Xi,W), t← 1 for i← 0 to n−nbnt do
Ŷ t = βŶ t−1 + (1− β)F(Xi,W) Y tc = γŶ t + (1− γ)Y
F ← G(F , {X,Y tc }, nt), t← t+ 1 end
Here G represents the training of the neural network, taking in a model F , dataset {X,Y } and an integer representing number of epochs.
A large nt allows the network to learn a better mapping to the intermediate soft-labels and therefore allows the regularization to be more effective. But increasing nt has a diminishing effect, because
as nt becomes large the network begins to over-fit to those soft-labels, and reduces the effect of the regularization, as well as increasing the training time of the network significantly. nt should be optimized experimentally through standard hyper-parameter optimization practices. We found nt = {1, 2} to work best through standard grid hyper-parameter optimization (Bergstra & Bengio, 2012). The small nt insures that the model does not overfit to the intermediate representation introduced by SoftTarget.
Through hyper-parameter optimization the same range of {1, 2} was found to be optimal in the experiments we ran for nb. A small nb insures that the co-label similarities captured by SoftTarget would not have been affected by any type of overfitting. This insures that as the experiments are further ran the true co-label similarties are propagated correctly. More complicated learning scenarias where the amount of labels and data is greater, the chances of corruption in co-label similarties is reduced and therefore larger nb can be choosen.
1.3 SIMILARITIES TO OTHER METHODS
Other methods similar to this are specific to the case where the β hyper-parameter is set to zero, with no burn-in period.
• Reed et al. study the specific case of the SoftTarget method described above with the β parameter set to zero (Reed et al., 2014). They focus on the capability of the network to be robust to noise, rather than the regularization abilities of the method.
• Grandvalet and Bengio have proposed minimum entropy regularization in the setting of semi-supervised learning (Grandvalet & Bengio, 2005). This algorithm changes the categorical cross-entropy loss to force the network to make predictions with high degrees of confidence on the unlabeled portion of the dataset. Assuming cross-entropy loss with SoftTarget normalization with a zero burn-in period, and zero β, our algorithm becomes equivalent to a softmax regression with minimum entropy regularization.
• Another similar approach to minimum entropy regularization is an approach called pseudolabeling. Pseudo-labeling tags unlabeled data with the class predicted highest by a learning model (Lee, 2013). No soft-targets are kept, instead the predicted label is binarized, i.e. the highest class is labeled with a value of one, and every other class is labeled with a value of zero. These hard pseudo-labels are then fed as input to the model.
• Hinton et al described the power of soft targets in the use of transferring knowledge from one model to another, usually to a model that contains less parameters (Hinton et al., 2015). Soft-Target regularization can be interpreted as weighted distillation where the donor model is the state of the model at some previous time in training, and the weighting target are the hard-targets.
2 EXPERIMENTS
We conducted experiments in python using the Theano and Keras libraries (The Theano Development Team, 2016), (Chollet, 2015). All of our code ran on a single Nvidia Titan X GPU, while using the CnMEM and cuDNN (5.103) extensions, and we visualized our results using matplotlib (Hunter, 2007). We used the same seed in all our calculations to insure the starting weights were equivalent in every set of experiments. The only source of randomness stemmed from the non-deterministic behavior of the cuDNN libraries.
2.1 MNIST
We first considered the famous MNIST dataset (LeCun et al., 1998). For each of the experiments discussed below, we performed a random grid-search over the hyper-parameters of the optimization algorithm, and a very small brute force grid search was done for the hyper-parameters of SoftTarget regularization. We compared our results to the cases where the hyper-parameters resulted in the best performance of the vanilla neural network without SoftTarget regularization. All of our reported values were computed on the standardized test portion of the MNIST dataset, as provided by the Keras library. The networks were trained strictly on the training portion of the dataset. We tested on
eight different architectures, with four combinations of every architecture. The four combinations stem from testing each architecture via a combination of: no regularization, Dropout, SoftTarget, and Dropout+SoftTarget regularization.
We used a fully connected network, with a varying amount of hidden layers, and a set constant of neurons throughout each layer. Dropout was not introduced at the input layer, but was introduced at every layer after that. All of the layers activations we’re rectified linear units (ReLu), except for the final layer which was a SoftMax. The net was trained using a categorical cross-entropy loss, and the ADADELTA optimization method. (Zeiler, 2012).
The frozen hyper-parameters for the SoftTarget regularization were: nb = 2, nt = 2, n = 100, β = 0.70, γ = 0.5. Our results are described in Table 1. We described the nets using the notation: 4← 256 denoting a 4 hidden layer neural network, with each of the hidden layers having 256 units. We reported the minimum loss during training, the loss at the 100th epoch, and the maximum accuracy reached respectively.
In all our experiments, the best performing regularization for all of the architectures described above included SoftTarget regularization. Two representative results are plotted in Figure 1 for a shallow (three layer) and deep (seven layer) neural network. We saw that for deep neural networks (greater than three layers) SoftTarget regularization outperformed all the other regularization schemes. For shallow (three layer) neural networks SoftTarget+Dropout was the optimal scheme.
2.2 CIFAR-10
We then considered the CIFAR-10 dataset (Krizhevsky & Hinton, 2009), comparing various combinations of SoftTarget, Dropout and BatchNormalization (BN) (Ioffe & Szegedy, 2015). BatchNormalization has been shown to have a regularization effect on neural networks due to the noise inherent to the mini-batch statistics. We ran each configuration of the network through sixty iterations through the whole training set. The complete architecture used was:
Input→ Convolution (64,3,3)→ BN→ ReLU→ Convolution (64,3,3)→ BN→ ReLU→ MaxPooling ((3,3), (2,2)) → Dropout (p) → Convolution (128,3,3) → BN → ReLU → Convolution (128,3,3)→ BN→ ReLU→MaxPooling ((3,3), (2,2))→ Dropout (p)→ Convolution (256,3,3)→ BN→ ReLU→ Convolution (256,1,1)→ BN→ ReLU→ Convolution (256,1,1)→ BN→ ReLU → Dropout (p)→ AveragePooling ((6,6))→ Flatten ()→ Dense (256)→ BN→ ReLU→ Dense (256)→ BN→ ReLU→ Dense (256)→ SoftMax. where: Convolution (64,3,3) signifies the convolution operator with 64 filters, and a kernel size of 3 by 3, MaxPooling ((3,3), (2,2)) represents the max-pooling operation with a kernel size of 3 by 3, and a stride of 2 by 2, AveragePooling ((6,6)) represents the average pooling operator with a kernel size of 6 by 6, Flatten represents a flattening of the tensor into a matrix, and Dense (256) a fully-connected layer (Krizhevsky et al., 2012), (Scherer et al., 2010). In our results, when we note that BN or Dropout weren’t used, we simply omitted those layers from the architecture. We trained the networks using ADADELTA on the cross-entropy loss, using the same SoftTarget hyperparameters we reported for the MNIST dataset. Our results are summarized in Table 2. The first column specifies the amount of Dropout used on the combinations listed in the next columns. As with the MNIST experiments, we reported the minimum loss during training, and the loss at the 100th epoch.
The use of SoftTarget regularization resulted in the lowest loss in four out of the five experiments on this architecture, and resulted in the lowest last epoch loss value and highest accuracy in all five of the experiments. As the dropout rate is increased the need for any other type of regularization is decreased. However, by increasing the rate of dropout, the resulting loss is increased because of the
reduced capacity of the network. SoftTarget regularization allowed a lower dropout rate to be used, and this lowered the test error.
2.3 SVHN
Finally, we considered the Street View House Numbers (SVHN) dataset, consisting of various images mapping to one of ten digits (Netzer et al., 2011). This is similar to the MNIST dataset, but is much more organic in nature, as these images contain much more natural noise, such as lighting conditions and camera orientation. We tested residual networks in four configurations: No regularization, Batch Normalization (BN), SoftTarget, and BN+SoftTarget (?). Our architecture consisted of the same building blocks as the residual network outlined by He et al., consisting of identity and convolution blocks (He et al., 2015). Identity blocks are blocks that do not contain a convolution layer at the shortcut, while convolution blocks do. In our notation I (3,[16,16,32], BN) will mean an identity block with an intermediate square convolution kernel size of 3, with three convolution blocks of size 16, 16 and 32. The outer convolutions contain kernel sizes of 1. C (3,[16,16,32], BN) contains the same initial architecture as I (3,[16,16,32]) but an additional convolution layer of size 32 at the shortcut connection. All of these blocks contained the rectified linear function as their activation, and BN prior to activation. Our final architecture was:
Input → ZeroPadding (3,3) → Convolution (64,7,7,subsample = (2,2)) → BN → ReLU → MaxPooling ((3,3), (2,2)) → C (3,[16,16,32], BN) → I (3,[16,16,32], BN) → I (3,[16,16,32], BN) C (3,[32,32,64], BN) → I (3,[32,32,64], BN) → I (3,[32,32,64], BN) → AveragePooling ((7,7)) → Dense (10)→ SoftMax We used the ADADELTA optimization method with a random grid search for hyper-parameter optimization. All configurations of the networks were run for 60 iterations apart from the overfit configuration which was run for 100 iterations.
We reported our results in Table 3 and Figure 3, as before giving the minimum test loss and the test loss at the last epoch. SoftTarget regularized configurations (with and without BN) again scored the lowest test loss and highest accuracy, compared to Batch Normalization alone.
2.4 CO-LABEL SIMILARITIES
We claimed that over-fitting begins to occur when co-label similarities that appeared in the initial stages of training, are not longer present. To test this hypothesis we compared the covariate matrices of a over-fitted network, early training stopped networks, and regularized networks. We tested again on the CIFAR10 dataset, with the same architecture as the previous CIFAR10 experiment,
except that the number of filters and dense units were reduced exactly by two. We compared four configurations: Early (10 epochs), Overfit (100 epochs), Dropout (p=0.2, 100 epochs) and SoftTarget (nb = 2, nt = 2, β = 0.7, γ = 0.5, 100 epochs). After training each configuration for its respected amount we predicted the labels of the training set. We then calculated a covariance matrix scaled to a range of [0, 1] since we are only interested in the relative co-label similarities. We set the diagonal to all zeros, as to make it easier to see other relations. The covariance function used is defined below.
ci,i = 0 (6)
cx,y =
∑N i=1(xi − x̄)(yi − ȳ)
N − 1 (7)
covs(x, y) = cx,y −min(cx,y)
max(cx,y)−min(cx,y) (8)
We plotted the covariance matrices in Figure 4. For the early stop case, there we observed the highest covariance between labels 3 and 5, which correspond to cats and dogs respectively. This intuitively makes sense, during earlier steps of training, the network learns to first detect differences between varying entities, such as frog and airplane, and then later learns to detect subtle difference. It is interesting to note, that this is the core principle behind prototype theory in human psychology (Osherson & Smith, 1981), (Duch, 1996), (Rosch, 1978). Some concepts are in nature closer to each other than others. Dog and cat are closer in relation than frog and airplane, and our regularization method mimics this phenomena. Another interesting thing to note is that the dropout method of regularization produces a covariance matrix that is very similar to that produced by SoftTarget regularization. The phenomena of co-label similarities being propagated throughout learning is not specific to just SoftTarget regularization, but regularization in general. Therefore co-label similarities can be seen as a measure of over-fitting.
3 CONCLUSION AND FUTURE WORK
In conclusion, we presented a new regularization method based on the observation that co-label similarities apparent in the beginning of training, disappear once a network begins to over-fit. SoftTarget
regularization reduced over-fitting as well as Dropout without adding complexity to the network, therefore reducing computational time, and we provided novel insights into the problem of overfitting.
Future work will focus on methods to reduce the number of hyper-parameters introduced by SoftTarget regularization, as well as providing a formal mathematical framework to understand the phenomenon of co-label similarities.

This manuscript tries to tackle neural network regularization by blending the target distribution with predictions of the model itself. In this sense it is similar in spirit to scheduled sampling (Bengio et al) and SEARN (Daume et al) DAgger (Ross et al) which consider a "roll-in" mixture of the target and model distributions during training. It was clarified in the pre-review questions that these targets are generated on-line rather than from a lagged distribution, which I think makes the algorithm pseudocode somewhat misleading if I understand it correctly.
This is an incremental improvement on the idea of label softening/smoothing that has recently been revived, and so the novelty is not that high. The author points out that co-label similarity is better preserved by this method but it doesn't follow that this is causal re: regularization; a natural baseline would be a fixed, soft label distribution, as well as one where the softening/temperature of the label distribution is gradually reduced (as one would expect for this method to do as the model gets closer and closer to reproducing the target distribution).
It's an interesting and somewhat appealing idea but the case is not clearly made that this is all that useful. The dropout baselines for MNIST seem quite far from results already in the literature (Srivastava et al 2014 achieves 1.06% with a 3x1024 MLP with dropout and a simple max norm constraint; the dropout baselines here fail to break 1.3% which is rather high by contemporary standards on the permutation-invariant task), and results for CIFAR10 are quite far from the current state of the art, making it difficult to judge the contribution in light of other innovations. The largest benchmark considered is SVHN where the reported accuracies are quite bad indeed; SOTA for single net performance has been less than half the reported error rates for 3-4 years now. It's unclear what conclusions can be drawn about how this would help (or even hurt) in a better-tuned setting.
I have remaining reservations about data hygiene, namely reporting minimum test loss/maximum test accuracy rather than an unbiased method for model selection (minimum validation set error, for example). Relatedly, the regularization potential of early stopping on a validation set is not considered. See, e.g. the protocol in Goodfellow et al (2013).


Recurrent neural nets are widely used for predicting temporal data. Their inherent deep feedforward structure allows learning complex sequential patterns. It is believed that top-down feedback might be an important missing ingredient which in theory could help disambiguate similar patterns depending on broader context. In this paper, we introduce surprisal-driven recurrent networks, which take into account past error information when making new predictions. This is achieved by continuously monitoring the discrepancy between most recent predictions and the actual observations. Furthermore, we show that it outperforms other stochastic and fully deterministic approaches on enwik8 character level prediction task achieving 1.37 BPC.
1 INTRODUCTION
Based on human performance on the same task, it is believed that an important ingredient which is missing in state-of-the-art variants of recurrent networks is top-down feedback. Despite evidence of its existence, it is not entirely clear how mammalian brain might implement such a mechanism. It is important to understand what kind of top-down interaction contributes to improved prediction capability in order to tackle more challenging AI problems requiring interpretation of deeper contextual information. Furthermore, it might provide clues as what makes human cognitive abilities so unique. Existing approaches which consider top-down feedback in neural networks are primarily focused on stacked layers of neurons, where higher-level representations constitute a top-down signal source. In this paper, we propose that the discrepancy between most recent predictions and observations might be effectively used as a feedback signal affecting further predictions. It is very common to use such a discrepancy during learning phase as the error which is subject to minimization, but not during inference. We show that is also possible to use such top-down signal without losing generality of the algorithm and that it improves generalization capabilities when applied to Long-Short Term Memory (Hochreiter & Schmidhuber, 1997) architecture. It is important to point out that the feedback idea presented here applies only to temporal data.
1.1 SUMMARY OF CONTRIBUTIONS
The main contributions of this work are:
• the introduction of a novel way of incorporating most recent misprediction measure as an additional input signal
• extending state-of-the-art performance on character-level text modeling using Hutter Wikipedia dataset.
1.2 RELATED WORK
There exist other approaches which attempted to introduce top-down input for improving predictions. One such architecture is Gated-Feedback RNN (Chung et al., 2015). An important difference between architecture proposed here and theirs is the source of the feedback signal. In GF-RNN it is assumed that there exist higher level representation layers and they constitute the feedback source.
On the other hand, here, feedback depends directly on the discrepancy between past predictions and current observation and operates even within a single layer. Another related concept is Ladder Networks (Rasmus et al., 2015), where top-down connections contribute to improved semi-supervised learning performance.
2 FEEDBACK: MISPREDICTION-DRIVEN PREDICTION
2.1 NOTATION
The following notation is used throughout the section:
x - inputs h - hidden units y - outputs p - output probabilities (normalized y) s - surprisal t - time step W - feedforward x→ h connection matrix U - recurrent h→ h connection matrix V - feedback s→ h connection matrix S - truncated BPTT length M - number of inputs N - number of hidden units
· denotes matrix multiplication denotes elementwise multiplication σ(·), tanh(·) - elementwise nonlinearities δx = ∂E∂x
In case of LSTM, the following concatenated representations are used:
gt =  itftot ut  b = b i bf bo bu U = U i Uf Uo Uu W = W i Wf Wo Wu V = V i Vf Vo Vu  (1)
2.2 SIMPLE RNN WITHOUT FEEDBACK
First, we show a simple recurrent neural network architecture without feedback which serves as a basis for demonstrating our approach. It is illustrated in Fig. 2 and formulated as follows:
ht = tanh(W · xt + U · ht−1 + b) (2)
2.3 FEEDBACK AUGMENTED RECURRENT NETWORKS
Figure 3 presents the main idea of surprisal-driven feedback in recurrent networks. In addition to feedforward and recurrent connections W and U , we added one additional matrix V . One more input signal, namely V · st is being considered when updating hidden states of the network. We propose that the discrepancy st between most recent predictions pt−1 and observations xt might be effectively used as a feedback signal affecting further predictions. Such information is usually used during learning phase as an error signal, but not during inference. Our hypothesis is that it represents an important source of information which can be used during the inference phase, should be used and that it bring benefits in the form of improved generalization capability. Figure 1 presents examples of feedback signal being considered. Intuitively, when surprisal is near zero, the sum of input signals is the same as in a typical RNN. Next subsections provide mathematical description of the feedback architecture in terms of forward and backward passes for the Back Propagation Through Time (BPTT) (Werbos, 1990) algorithm.
2.4 FORWARD PASS
Set h0, c0 to zero and p0 to uniform distribution or carry over the last state to emulate full BPTT.
∀i, pi0 = 1
M , i ∈ {0, 1, ..,M − 1}, t = 0 (3)
for t = 1:1:S-1
I. Surprisal part
st = − i∑ log pit−1 xit (4)
IIa. Computing hidden activities, Simple RNN
ht = tanh(W · xt + U · ht−1 + V · st + b) (5)
IIb. Computing hidden activities, LSTM (to be used instead of IIa)
ft = σ(Wf · xt + Uf · ht−1 + Vf · st + bf ) (6)
it = σ(W i · xt + U i · ht−1 + Vi · st + bi) (7)
ot = σ(Wo · xt + Uo · ht−1 + Vo · st + bo) (8)
ut = tanh(Wu · xt + Uu · ht−1 + Vu · st + bu) (9)
ct = (1− ft) ct−1 + it ut (10)
ĉt = tanh(ct) (11)
ht = ot ĉt (12)
III. Outputs
yit =Wy · ht + by (13)
Softmax normalization
pit = ey i t∑i ey i t
(14)
2.5 BACKWARD PASS
for t = S-1:-1:1
I. Backprop through predictions
Backprop through softmax, cross-entropy error, accumulate
∂Et ∂yt = ∂Et ∂yt + pt−1 − xt (15)
δy → δWy, δby ∂E
∂Wy =
∂E
∂Wy + hTt · ∂Et ∂yt
(16)
∂E ∂by = ∂E ∂by + M∑ i=1 ∂Eit ∂yit
(17)
δy → δh ∂Et ∂ht = ∂Et ∂ht + ∂Et ∂yt ·WTy (18)
IIa. Backprop through hidden nonlinearity (simple RNN version)
∂Et ∂ht = ∂Et ∂ht + ∂Et ∂ht tanh′(ht) (19)
∂Et ∂gt = ∂Et ∂ht
(20)
IIb. Backprop through c, h, g (LSTM version)
Backprop through memory cells, (keep gradients from the previous iteration)
∂Et ∂ct = ∂Et ∂ct + ∂Et ∂ht ot tanh′(ĉt) (21)
Carry error over to ∂Et∂ct−1
∂Et ∂ct−1 = ∂Et ∂ct−1 + ∂Et ∂ct (1− ft) (22)
Propagate error through the gates
∂Et ∂ot = ∂Et ∂ht ĉt σ′(ot) (23)
∂Et ∂it = ∂Et ∂ct ut σ′(it) (24)
∂Et ∂ft = −∂Et ∂ct ct−1 σ′(ft) (25)
∂Et ∂ut = ∂Et ∂ct it tanh′(ut) (26)
Carry error over to ∂Et∂ht−1 ∂Et ∂ht−1 = ∂Et ∂gt · UT (27)
III. Backprop through linearities
∂Et ∂b = ∂Et ∂b + N∑ i=1 ∂Et ∂git
(28)
∂E ∂U = ∂E ∂U + hTt−1 · ∂Et ∂gt
(29)
∂E ∂W = ∂E ∂W + xTt · ∂Et ∂gt
(30)
∂E ∂x = ∂E ∂x + ∂Et ∂gt ·WT (31)
IV. Surprisal part
∂E ∂V = ∂E ∂V + sTt · ∂Et ∂gt
(32)
∂E ∂st = ∂E ∂gt · V T (33)
∂Et ∂pt−1 = ∂Et ∂st xt (34)
Adjust ∂Et∂pt−1 according to the sum of gradients and carry over to ∂Et ∂yt−1
∂Et ∂yt−1 = ∂Et ∂pt−1 − pt−1 M∑ i=1 ∂Et ∂pit−1
(35)
3 EXPERIMENTS
We ran experiments on the enwik8 dataset. It constitutes first 108 bytes of English Wikipedia dump (with all extra symbols present in XML), also known as Hutter Prize challenge dataset2. First 90% of each corpus was used for training, the next 5% for validation and the last 5% for reporting test accuracy. In each iteration sequences of length 10000 were randomly selected. The learning algorithm used was Adagrad1 with a learning rate of 0.001. Weights were initialized using so-called Xavier initialization Glorot & Bengio (2010). Sequence length for BPTT was 100 and batch size 128, states were carried over for the entire sequence of 10000 emulating full BPTT. Forget bias was set initially to 1. Other parameters set to zero. The algorithm was written in C++ and CUDA 8 and ran on GTX Titan GPU for up to 10 days. Table 1 presents results comparing existing state-of-theart approaches to the introduced Feedback LSTM algorithm which outperforms all other methods despite not having any regularizer.
4 SUMMARY
We introduced feedback recurrent network architecture, which takes advantage of temporal nature of the data and monitors the discrepancy between predictions and observations. This prediction error
1with a modification taking into consideration only recent window of gradient updates 2http://mattmahoney.net/dc/text.html 3This method does not belong to the ’dynamic evaluation’ group: 1. It never actually sees test data during
training. 2. It does not adapt weights during testing 4our implementation
information, also known as surprisal, is used when making new guesses. We showed that combining commonly used feedforward, recurrent and such feedback signals improves generalization capabilities of Long-Short Term Memory network. It outperforms other stochastic and fully deterministic approaches on enwik8 character level prediction achieving 1.37 BPC.
5 FURTHER WORK
It is still an open question what the feedback should really constitute as well as how it should interact with lower-level neurons (additive, multiplicative or another type of connection). Further improvements may be possible with the addition of regularization. Another research direction is incorporating sparsity in order improve disentangling sources of variation in temporal data.
ACKNOWLEDGEMENTS
This work has been supported in part by the Defense Advanced Research Projects Agency (DARPA).

This paper proposes to leverage "surprisal" as top-down signal in RNN. More specifically author uses the error corresponding to the previous prediction as an extra input at the current timestep in a LSTM.
The general idea of suprising-driven feedback is interesting for online prediction task. It is a simple enough idea that seems to bring some significant improvements. However, the paper in its current form has some important flaws.
- Overall, the paper writing could be improved. In particular, section 2.4 and 2.5 is composed mostly by the equations of the forward and backward propagation of feedback RNN and feedback LSTM. However, author provides no analysis along with those equations. It is therefore not clear what insight the author tries to express in those sections. In addition, feedback RNN is not evaluated in the experimental section, so it is not clear why feedback RNN is described.
- The experimental evaluation is limited. Only one dataset enwik8 is explored. I think it is necessary to try the idea on different datasets to see if feedback LSTM sees some consistent improvements.
Also, author claims state-of-art on enwik8, but hypernetwork, already cited in the paper, achieves better results (1.34 BPC, table 4 in the hypernetworks paper).
- Author only compares to methods that do not use last prediction error as extra signal. I would argue that a comparison with dynamic evaluation would be more fair.
 Feedback LSTM uses prediction error as extra input in the forward prop, while dynamic evaluation  backprop it through the network and change the weight accordingly. Also they don't propagate the prediction error in the same way, they both leverage "extra" supervised information through the prediction errors.
In summary:
Pros:
- Interesting idea
- Seems to improve performances
Cons:
- Paper writing
- Weak evaluation (only one dataset)
- Compare only with approaches that does not use the last-timestep error signal


Gatys et al. (2015a) showed that pair-wise products of features in a convolutional network are a very effective representation of image textures. We propose a simple modification to that representation which makes it possible to incorporate longrange structure into image generation, and to render images that satisfy various symmetry constraints. We show how this can greatly improve rendering of regular textures and of images that contain other kinds of symmetric structure. We also present applications to inpainting and season transfer.
1 INTRODUCTION
There are currently two dominant approaches to texture synthesis: non-parametric techniques, which synthesize a texture by extracting pixels (or patches) from a reference image that are resampled for rendering (Efros & Leung, 1999; Kwatra et al., 2003), and parametric statistical models, which optimize reconstructions to match certain statistics computed on filter responses (Heeger & Bergen, 1995; Portilla & Simoncelli, 2000). Recently, the second approach has seen a significant advancement, after Gatys et al. (2015a) showed that a CNN pre-trained on an object classification task, such as ImageNet (Russakovsky et al., 2015), can be very effective at generating textures. Gatys et al. (2015a) propose to minimize with respect to the input image a loss function, that measures how well certain high-level features of a reference image are preserved. The reference image constitutes an example of the texture to be generated. The high-level features to be preserved are pair-wise products of feature responses, averaged over the whole image, referred to as the “Gramian” in that work. In Gatys et al. (2015b), the same authors show that by adding a second term to the cost, which matches the content of another image, one can render that other image in the “style” (texture) of the first. Numerous follow-up works have since then analysed and extended this approach (Ulyanov et al., 2016; Johnson et al., 2016; Ustyuzhaninov et al., 2016).
As shown in Figure 1, this method produces impressive results. However, it fails to take into account non-local structure, and consequently cannot generate results that exhibit long-range correlations in images. An example of the importance of long-range structure is the regular brick wall texture in the middle of the figure. Another example is the task of inpainting, where the goal is to fill in a missing part of an image, such that it is faithful to the non-missing pixels. Our main contribution is to introduce a way to deal with long-range structure using a simple modification to the product-based texture features. Our approach is based on imposing a “Markov-structure” on high-level features, allowing us to establish feature constraints that range across sites instead of being local. Unlike classical approaches to preserving spatial structure in image generation, such as Markov Random Fields and learning-based extensions (Roth & Black, 2005), our approach does not impose any explicit local constraints on pixels themselves. Rather, inspired by Gatys et al. (2015a), it encourages consistency to be satisfied on high-level features and on average across the whole image. We present applications to texture generation, inpainting and season transfer.
2 THE ARTISTIC STYLE ALGORITHM
2.1 SYNTHESIS PROCEDURE
Given a reference texture, x, the algorithm described in Gatys et al. (2015a) permits to synthesize by optimization a new texture x̂ similar to x. To achieve this, the algorithm exploits an ImageNet
pre-trained model to define metrics suitable for describing textures: “Gram” matrices of feature maps, computed on top of L selected layers. Formally, let N l be the number of maps in layer l of a pre-trained CNN. The corresponding Gram matrix Gl is a N l ×N l matrix defined as:
Glij = 1
M l M l∑ k=1 F likF l jk = 1 M l 〈F li:, F lj:〉 (1)
where F li: is the i th vectorized feature map of layer l, M l is the number of elements in each map of this layer, and where 〈 ·, ·〉 denotes the inner product. Equation 1 makes it clear that Gl captures how feature maps from layer l are correlated to each other. Diagonal terms, Glii are the squared Frobenius norm of the ith map
∥∥F li:∥∥2F , so they represent its spatially averaged energy. We will discuss the Gramians in more detail in the next paragraph. Once the Gram matrices {Gl}l∈[1,L] of the reference texture are computed, the synthesis procedure by Gatys et al. (2015a) amounts to constructing an image that produces Gram matrices {Ĝl}l∈[1,L] that match the ones of the reference texture. More precisely, the following loss function is minimized with respect to the image being constructed:
Lstyle = L∑ l=1 wl ∥∥∥Ĝl −Gl∥∥∥2 F = L∑ l=1 wlLlstyle (2)
where wl is a normalizing constant similar to Gatys et al. (2015a).
The overall process is summarized in Figure 2. While the procedure can be computationally expensive, there have been successful attempts reported recently which reduce the generation time (Ulyanov et al., 2016; Johnson et al., 2016).
2.2 WHY GRAM MATRICES WORK
Feature Gram matrices are effective at representing texture, because they capture global statistics across the image due to spatial averaging. Since textures are static, averaging over positions is required and makes Gram matrices fully blind to the global arrangement of objects inside the reference image. This property permits to generate very diverse textures by just changing the starting point of the optimization. Despite averaging over positions, coherence across multiple features needs to be preserved (locally) to model visually sensible textures. This requirement is taken care of by the off-diagonal terms in the Gram matrix, which capture the co-occurence of different features at
a single spatial location. Indeed, Figure 3 shows that restricting the texture representation to the squared Frobenius norm of feature maps (i.e. diagonal terms) makes distinct object-parts from the reference texture encroach on each other in the reconstruction, as local coherence is not captured by the model. Exploiting off-diagonal terms improves the quality of the reconstruction as consistency across feature maps is enforced (on average across the image).
The importance of local coherence can be intuitively understood in the case of linear features (or in the lowest layer of a convolutional network): when decomposing an image using Gabor-like features, local structure can be expressed as the relative offsets in the Fourier phase angles between multiple different filter responses. A sharp step-edge, for example, requires the phases of local Fourier components at different frequencies to align in a different way than a blurry edge or a ridge (Morrone & Burr, 1988; Kovesi, 1999). Also, natural images exhibit very specific phaserelationships across frequency components in general, and destroying these makes the image look unnatural (Wang & Simoncelli, 2003). The same is not true of Fourier amplitudes (represented on the diagonals of the Gramian), which play a much less important role in the visual appearance (Oppenheim & Lim, 1981). In the case of deeper representations, the situation is more complex, but it is still local co-occurrence averaged over the whole image that captures texture.
Unfortunately, average local coherence falls short of capturing long-range structure in images. Spatial consistency is hard to capture within a single filter bank, because of combinatorial effects. Indeed, since Gram matrices capture coherence at a single spatial location, every feature would have to be matched to multiple transformed versions of itself. A corollary is that every feature would have to appear in the form of multiple transformed copies of itself in order to capture spatial consistency. However, this requirement clashes with the limited number of features available in each CNN layer. One way to address this is to use higher-layer features, whose receptive fields are larger. Unfortunately, as illustrated in Figure 3, even if using layers up to pool5 whose input receptive field covers the whole image1 (first row, last column), the reconstruction remains mainly unstructured and the method fails to produce spatial regularities.
3 MODELING SPATIAL CO-OCCURENCES
To account for spatial structure in images, we propose encoding this structure in the feature selfsimilarity matrices themselves. To this end, we suggest that, instead of computing co-occurences between multiple features within a map, we compute co-occurences between feature maps F l and spatially transformed feature maps T (F l), where T denotes a spatial transformation. In the simplest case, T represents local translation, which amounts to measuring similarities between local features and other neighbouring features. We denote by Tx,+δ the operation consisting in horizontally translating feature maps by δ pixels and define the transformed Gramian:
Glx,δ,ij = 1
M l 〈Tx,+δ
( F li: ) , Tx,−δ ( F lj: ) 〉 (3)
1For this experiment, the image size is 264× 264, which is also the size of the pool5 receptive field.
where Tx,−δ performs a translation in the opposite direction. As illustrated in Figure 4, the transformation in practice simply amounts to removing the δ first or last columns from the raw feature maps. Therefore, the inner product now captures how features at position (i, j) are correlated with features located at position (i, j + δ) in average. While Figure 4 illustrates the case where feature maps are horizontally shifted, one would typically use translations along both the x-axis and the y-axis. Our transformed Gramians are related to Gray-Level Co-occurrence Matrices (GLCM) (Haralick et al., 1973) which compute the unnormalized frequencies of pixel values for a given offset in an image. While GLCMs have been mainly used for analysis, some work tried to use these features for texture synthesis (Lohmann, 1995). Usually, GLCMs are defined along 4 directions: 0◦ and 90◦ (i.e. horizontal and vertical offsets), as well as 45◦ and 135◦ (i.e. diagonal offsets). In comparison, our method does not consider diagonal offsets and captures spatial coherence on high-level features, making use of a pre-trained CNN, instead of working directly in the pixel domain.
With this definition for transformed Gram matrices, we propose defining the loss as: L = Lstyle + Lcc, where cc stands for cross-correlation. Like Lstyle, Lcc is a weighted sum of multiple losses Llcc,δ defined for several selected layers as the mean squared error between transformed Gram matrices of the reference texture and the one being constructed:
Llcc,δ = 1
2
( Llcc,x,δ + Llcc,y,δ ) = 1
2 (∥∥∥Ĝlx,δ −Glx,δ∥∥∥2 F + ∥∥∥Ĝly,δ −Gly,δ∥∥∥2 F ) (4)
Although this amounts to adding more terms to a representation that was already high-dimensional and overparametrized, we found that these additional terms do not hurt the diversity of generated textures. Indeed, the new loss remains blind to the global arrangement of objects. In fact, there exists a specific situation where our approach is strictly equivalent to computing Gram matrices of deeper layers: with linear activations and “one-hot” convolution kernels2, deeper layers would simply contain translated versions of lower feature maps. In that case, computing Gram matrices of deeper layers would permit to directly capture cross-correlation statistics in lower ones. Nevertheless, this situation is very unlikely when using a pretrained CNN, as the network probably learned operations more useful than simple translations during its supervised training.
While we focus on translation for most of our results, we shall discuss other types of transformation in the experiments Section.
4 EXPERIMENTS
In our experiments, we exploit the same normalized version of the VGG-19 network3 (Simonyan & Zisserman, 2014) as in Gatys et al. (2015a;b) and layers conv1 1, pool1, pool2, pool3, and pool4 are always used to define the standard Gram matrices. In our method, we did not use conv1 1 to define cross-correlation terms, as the large number of neurons at this stage makes the computation of Gram matrices costly. Corresponding δ values for each layer are discussed in the next paragraph.
2To match our translated Gramians, the only non-zero component would be δ or −δ shifted from the center. 3available at http://bethgelab.org/media/uploads/deeptextures/vgg normalised.caffemodel.
Finally, our implementation4 uses Lasagne (Dieleman et al., 2015). Each image is of size 384×384. Most textures used as references in this paper were taken from textures.com and pixabay.com.
4.1 EXPERIMENTS WITH TRANSLATION-GRAMIANS
The δ parameter is of central importance as it dictates the range of the spatial constraints. We observed that the optimal value depends on both the considered layer in the pre-trained CNN and the reference texture, making it difficult to choose a value automatically.
For instance, Figure 5 shows generated images from the brick wall texture using only the pool2 layer with different δ configurations. The first row depicts the results when considering single values of δ only. While δ = 4 or δ = 8 are good choices, considering extreme long-range correlations does not help for this particular texture: a brick depends mostly on its neighbouring bricks and not the far-away ones. More precisely, a translation of more than 16 pixels in the pool2 layer makes the input receptive field move more than 64 pixels. Therefore δ = 16 or δ = 32 do not capture any information about neighbouring bricks. Unfortunately, this is not true for all textures, and δ = 16 or δ = 32 might be good choices for another image that exhibits longer structures.
Searching systematically for a δ configuration that works well with the reference texture being considered would be a tedious task: even for a very regular texture with a periodic horizontal (or vertical) pattern, it is hard to guess the optimal δ values for each layer (for deeper ones in particular). Instead, we propose to use a fixed but wide set of δ values per layer, by defining the cost to be: Llcc = ∑ k Llcc,δk . A potential concern is that combining many loss terms can hurt the reconstruction or the diversity of generated textures. Figure 5 (second row) shows contrarily that there is no visual effect from using δ values that are not specifically useful for the reference texture being considered: the rendering benefits from using δ ∈ {2, 4, 8} while considering bigger values (δ ∈ {2, 4, 8, 16, 32}, e.g.) does not help, but does not hurt the reconstruction either. We found the same to be true for other textures as well, and our results (shown in the next Section) show that combining the loss terms is able to generate very diverse textures. A drawback from using multiple cross-correlation terms per layers, however, is computational. We found that in our experimental setups, adding cross-correlation terms increases the generation time by roughly 80%.
As a guide-line, for image sizes of roughly 384 × 384 pixels, we recommend the following δ values per layer (which we used in all our following experiments): {2, 4, 8, 16, 32, 64} for pool1, {2, 4, 8, 16, 32} for pool2, {2, 4, 8, 16} for pool3, and {2, 4, 8} for pool4. The number and the range of δ values decrease with depth because feature maps are getting smaller due to 2× 2 pooling layers. This configuration should be sufficient to account for spatial structure in any 384 × 384 image.
4available at https://github.com/guillaumebrg/texture generation
4.2 SYNTHESIS OF STRUCTURED TEXTURE EXAMPLES
Figure 6 shows the result of our approach applied to various structured and unstructured textures. It demonstrates that the method is effective at capturing long-range correlations without simply copying the content of the original texture. For instance, note how our model captures the depth aspect of the reference image in the third and fourth rows.
The problem of synthesizing near-regular structures is challenging because stochasticity and regularity are adversarial properties (Lin et al., 2006). Non-parametric patch-based techniques, such as Efros & Freeman (2001), are better suited for this task because they can tile5 the reference image. On the other hand, regular structures are usually more problematic for parametric statistical models. Nevertheless, the two first rows of Figure 6 demonstrate that our approach can produce good visual results and can reduce the gap to patch-based methods on these kinds of texture.
Even if the reference image is not a texture, the generated images in the last row (Leonardo Dicaprio’s face) provide a good visual illustration of the effect of translation terms. In contrast to Gatys et al., our approach preserves longer-range structure, such as the alignment and similar appearance of the eyes, hair on top of the forehead, the chin below the mouth, etc. Finally, when the reference texture is unstructured (fifth row), our solution does not necessarily provide a benefit, but it also does not hurt the visual quality or the diversity of the generated textures.
4.3 INPAINTING APPLICATION
Modelling long-range correlations can make it possible to apply texture generation to inpainting, because it allows us to impose consistency constraints between the newly rendered region and the unmodified parts of the original image.
To apply our approach to texture inpainting, we extracted two patches from the original image: one that covers the whole area to inpaint, and another one that serves as the reference texture. Then, approximately the same process as for texture generation is used, with the following two
5Copy and paste patches side by side.
modifications: First, instead of random noise, the optimization starts from the masked content patch (the one to inpaint) showing a grey area and its non-missing surrounding. Second, we encourage the borders of the output to not change much with respect to the original image using an L2 penalty. We apply the penalty both in the Gatys et al. rendering and in ours. Some inpainted images are shown in Figure 7. As seen in the figure, our solution significantly outperforms Gatys et al. in terms of visual quality. Further results are shown in the supplementary material.
4.4 SEASON TRANSFER
Figure 8 shows the result of applying our approach to a style transfer task, as in Gatys et al. (2015b): transferring the “season” of a landscape image to another one. On this task, the results from our approach are similar to those from Gatys et al. (2015b). Nevertheless, in contrast to the Gatys et al. results, our approach seems to better capture global information, such as sky color and leaves (bottom row), or the appearance of branches in the winter image (top row).
4.5 INCORPORATING OTHER TYPES OF STRUCTURE
While we focused on feature map translations in most of our experiments, other transformations can be applied as well. To illustrate this point, we explored a way to generate symmetric textures using another simple transformation. To this end, we propose flipping one of the two feature maps before computing the Gram matrices: Gllr,ij = 〈F li:, Tlr ( F lj: ) 〉. Here Tlr corresponds to the left-right flipping operation, but we also considered up-down flipping of feature maps: Ll = Llstyle + Lllr + Llud. As can be seen in Figure 9, in contrast to Gatys et al., the additional loss terms capture which objects are symmetric in the reference texture, and enforce these same objects to be symmetric in the reconstruction as well. Other kinds of transformation could be used, depending on the type of property in the source texture one desires to preserve.
5 CONCLUSION
We presented an approach to satisfying long-range consistency constraints in the generation of images. It is based on a variation of the method by Gatys et al., and considers spatial co-occurences of local features (instead of only co-occurences across features). We showed that the approach permits to generate textures with various global symmetry properties and that it makes it possible to apply texture generation to in-painting. Since it preserves correlations across sites, the approach is reminiscent of an MRF, but in contrast to an MRF or other graphical models, it defines correlationconstraints on high-level features of a (pre-trained) CNN rather than on pixels.

The paper investigates a simple extension of Gatys et al. CNN-based texture descriptors for image generation. Similar to Gatys et al., the method uses as texture descriptor the empirical intra-channel correlation matrix of the CNN feature response at some layer of a deep network. Differently from Gatys et al., longer range correlations are measured by introducing a shift between the correlated feature responses, which translates in a simple modification of the original architecture.
The idea is simple but has interesting effects on the generated textures and can be extended to transformations other than translation. While longer range correlations could be accounted for by considering the response of deeper CNN features in the original method by Gatys et al., the authors show that modelling them explicitly using shallower features is more effective, which is reasonable.
An important limitation that this work shares with most of its peers is the lack of a principled quantitative evaluation protocol, such that judging the effectiveness of the approach remains almost entirely a qualitative affair. While this should not be considered a significant drawback of the paper due to the objective difficulty of solving this open issue, nevertheless it is somewhat limiting that no principled evaluation method could be devised and implemented. The authors suggest that, as future work, a possible evaluation method could be based on a classification task -- this is a potentially interesting approach that merits some further investigation.