REGULARIZING CNNS WITH LOCALLY CONSTRAINED DECORRELATIONS
Regularization is key for deep learning since it allows training more complex models while keeping lower levels of overfitting. However, the most prevalent regularizations do not leverage all the capacity of the models since they rely on reducing the effective number of parameters. Feature decorrelation is an alternative for using the full capacity of the models but the overfitting reduction margins are too narrow given the overhead it introduces. In this paper, we show that regularizing negatively correlated features is an obstacle for effective decorrelation and present OrthoReg, a novel regularization technique that locally enforces feature orthogonality. As a result, imposing locality constraints in feature decorrelation removes interferences between negatively correlated feature weights, allowing the regularizer to reach higher decorrelation bounds, and reducing the overfitting more effectively. In particular, we show that the models regularized with OrthoReg have higher accuracy bounds even when batch normalization and dropout are present. Moreover, since our regularization is directly performed on the weights, it is especially suitable for fully convolutional neural networks, where the weight space is constant compared to the feature map space. As a result, we are able to reduce the overfitting of state-of-the-art CNNs on CIFAR-10, CIFAR-100, and SVHN.
1 INTRODUCTION
Neural networks perform really well in numerous tasks even when initialized randomly and trained with Stochastic Gradient Descent (SGD) (see Krizhevsky et al. (2012)). Deeper models, like Googlenet (Szegedy et al. (2015)) and Deep Residual Networks (Szegedy et al. (2015); He et al. (2015a)) are released each year, providing impressive results and even surpassing human performances in well-known datasets such as the Imagenet (Russakovsky et al. (2015)). This would not have been possible without the help of regularization and initialization techniques which solve the overfitting and convergence problems that are usually caused by data scarcity and the growth of the architectures.
From the literature, two different regularization strategies can be defined. The first ones consist in reducing the complexity of the model by (i) reducing the effective number of parameters with weight decay (Nowlan & Hinton (1992)), and (ii) randomly dropping activations with Dropout (Srivastava et al. (2014)) or dropping weights with DropConnect (Wan et al. (2013)) so as to prevent feature co-adaptation. Due to their nature, although this set of strategies have proved to be very effective, they do not leverage all the capacity of the models they regularize.
The second group of regularizations is those which improve the effectiveness and generality of the trained model without reducing its capacity. In this second group, the most relevant approaches decorrelate the weights or feature maps, e.g. Bengio & Bergstra (2009) introduced a new criterion so as to learn slow decorrelated features while pre-training models. In the same line Bao et al. (2013) presented ”incoherent training”, a regularizer for reducing the decorrelation of the network activations or feature maps in the context of speech recognition. Although regularizations in the second group are promising and have already been used to reduce the overfitting in different tasks, even with the presence of Dropout (as shown by Cogswell et al. (2016)), they are seldom used in the large scale image recognition domain because of the small improvement margins they provide together with the computational overhead they introduce.
Although they are not directly presented as regularizers, there are other strategies to reduce the overfitting such as Batch Normalization (Ioffe & Szegedy (2015)), which decreases the overfitting by reducing the internal covariance shift. In the same line, initialization strategies such as ”Xavier” (Glorot & Bengio (2010)) or ”He” (He et al. (2015b)), also keep the same variance at both input and output of the layers in order to preserve propagated signals in deep neural networks. Orthogonal initialization techniques are another family which set the weights in a decorrelated initial state so as to condition the network training to converge into better representations. For instance, Mishkin & Matas (2016) propose to initialize the network with decorrelated features using orthonormal initialization (Saxe et al. (2013)) while normalizing the variance of the outputs as well.
In this work we hypothesize that regularizing negatively correlated features is an obstacle for achieving better results and we introduce OrhoReg, a novel regularization technique that addresses the performance margin issue by only regularizing positively correlated feature weights. Moreover, OrthoReg is computationally efficient since it only regularizes the feature weights, which makes it very suitable for the latest CNN models. We verify our hypothesis through a series of experiments: first using MNIST as a proof of concept, secondly we regularize wide residual networks on CIFAR-10, CIFAR-100, and SVHN (Netzer et al. (2011)) achieving the lowest error rates in the dataset to the best of our knowledge.
2 DEALING WITH WEIGHT REDUNDANCIES
Deep Neural Networks (DNN) are very expressive models which can usually have millions of parameters. However, with limited data, they tend to overfit. There is an abundant number of techniques in order to deal with this problem, from L1 and L2 regularizations (Nowlan & Hinton (1992)), early-stopping, Dropout or DropConnect. Models presenting high levels of overfitting usually have a lot of redundancy in their feature weights, capturing similar patterns with slight differences which usually correspond to noise in the training data. A particular case where this is evident is in AlexNet (Krizhevsky et al. (2012)), which presents very similar convolution filters and even ”dead” ones, as it was remarked by Zeiler & Fergus (2014).
In fact, given a set of parameters θI,j connecting a set of inputs I = {i1, i2, . . . , in} to a neuron hj , two neurons {hj , hk}, j 6= k will be positively correlated, and thus fire always together if θI,j = θI,k and negatively correlated if θI,j = −θI,k. In other words, two neurons with the same or slightly different weights will produce very similar outputs. In order to reduce the redundancy present in the network parameters, one should maximize the amount of information encoded by each neuron. From an information theory point of view, this means one should not be able to predict the output of a neuron given the output of the rest of the neurons of the layer. However, this measure requires batch statistics, huge joint probability tables, and it would have a high computational cost.
In this paper, we will focus on the weights correlation rather than activation independence since it still is an open problem in many neural network models and it can be addressed without introducing too much overhead, see Table 1. Then, we show that models generalize better when different feature detectors are enforced to be dissimilar. Although it might seem contradictory, CNNs can benefit from having repeated filter weights with different biases, as shown by Li et al. (2016). However, those repeated filters must be shared copies and adding too many unshared filter weights to CNNs increases overfitting and the need for stronger regularization (Zagoruyko & Komodakis (May 2016)). Thus, our proposed method and multi-bias neural networks are complementary since they jointly increase the representation power of the network with fewer parameters.
In order to find a good target to optimize so as to reduce the correlation between weights, it is first required to find a metric to measure it. In this paper, we propose to use the cosine similarity between feature detectors to express how strong is their relationship. Note that the cosine similarity is equivalent to the Pearson correlation for mean-centered normalized vectors, but we will use the term correlation for the sake of clarity.
2.1 ORTHOGONAL WEIGHT REGULARIZATION
This section introduces the orthogonal weight regularization, a regularization technique that aims to reduce feature detector correlation enforcing local orthogonality between all pairs of weight vectors. In order to keep the magnitudes of the detectors unaffected, we have chosen the cosine similarity between the vector pairs in order to solely focus on the vectors angle β ∈ [−π, π]. Then, given any pair of feature vectors of the same size θ1, θ2 the cosine of their relative angle is:
cos(θ1, θ2) = 〈θ1, θ2〉 ||θ1||||θ2||
(1)
Where 〈θ1, θ2〉 denotes the inner product between θ1 and θ2. We then square the cosine similarity in order to define a regularization cost function for steepest descent that has its local minima when vectors are orthogonal:
C(θ) = 1
2 n∑ i=1 n∑ j=1,j 6=i cos2(θi, θj) = 1 2 n∑ i=1 n∑ j=1,j 6=i ( 〈θi, θj〉 ||θi||||θj || )2 (2)
Where θi are the weights connecting the output of the layer l − 1 to the neuron i of the layer l, which has n hidden units. Interestingly, minimizing this cost function relates to the minimization of the Frobenius norm of the cross-covariance matrix without the diagonal. This cost will be added to the global cost of the model J(θ;X, y), where X are the inputs and y are the labels or targets, obtaining J̃(θ;X, y) = J(θ;X, y) + γC(θ). Note that γ is an hyperparameter that weights the relative contribution of the regularization term. We can now define the gradient with respect to the parameters:
δ
δθ(i,j) C(θ) = n∑ k=1,k 6=i θ(k,j)〈θi, θk〉 〈θi, θi〉〈θk, θk〉 − θ(i,j)〈θi, θk〉2 〈θi, θi〉2〈θk, θk〉 (3)
The second term is introduced by the magnitude normalization. As magnitudes are not relevant for the vector angle problem, this equation can be simplified just by assuming normalized feature detectors:
δ
δθ(i,j) C(θ) = n∑ k=1,k 6=i θ(k,j)〈θi, θk〉 (4)
We then add eq. 4 to the backpropagation gradient:
∆θ(i,j) = −α ( ∇Jθ(i,j) + γ n∑ k=1,k 6=i θ(k,j)〈θi, θk〉 )
(5)
Where α is the global learning rate coefficient, J any target loss function for the backpropagation algorithm.
Although this update can be done sequentially for each feature-detector pair, it can be vectorized to speedup computations. Let Θ be a matrix where each row is a feature detector θ(I,j) corresponding to the normalized weights connecting the whole input I of the layer to the neuron j. Then, ΘΘt contains the inner product of each pair of vectors i and j in each position i, j. Subsequently, we
Algorithm 1 Orthogonal Regularization Step.
Require: Layer parameter matrices Θl, regularization coefficient γ, global learning rate α. 1: for each layer l to regularize do 2: η1 = norm rows(Θl) . Keep norm of the rows of Θl. 3: Θl1 = div rows(Θ
l, η1) . Keep a Θl1 with normalized rows. 4: innerProdMat = Θl1transpose(Θ l 1) 5: ∇Θl1 = γ(innerProdMat− diag(innerProdMat))Θl1 . Second term in eq. 6 6: ∆Θl = −α(∇JΘl + γ∇Θl1) . Complete eq. 6. 7: end for
(a) Global loss plot (eq. 2) (b) Local loss plot (eq. 7)
(c) Direction of gradients for the loss in (a). (d) Direction of gradients for loss in (b).
Figure 1: Comparison between the two loss functions represented by eq.2 and 7. (a) is the original loss, (b) is the new loss that discards negative correlations given for different λ values. It can be seen λ = 10 reaches a plateau when approximating to π2 . (c) and (d) shows the directions of the gradients for the two loss functions above. For instance, a red arrow coming from a green ball represents the gradient of the loss between the red and green balls with respect to the green one. In (d) most of the arrows disappear since the loss in (b) only applies to angles smaller than π2 .
subtract the diagonal so as to ignore the angle from each feature with respect to itself and multiply by Θ to compute the final value corresponding to the sum in eq. 5:
∆Θ = −α ( ∇JΘ + γ(ΘΘt − diag(ΘΘt))Θ ) (6)
Where the second term is∇CΘ. Algorithm 1 summarizes the steps in order to apply OrthoReg.
2.2 NEGATIVE CORRELATIONS
Note that the presented algorithm, based on the cosine similarity, penalizes any kind of correlation between all pairs of feature detectors, i.e. the positive and the negative correlations, see Figure 1a. However, negative correlations are related to inhibitory connections, competitive learning, and self-organization. In fact, there is evidence that negative correlations can help a neural population to increase the signal-to-noise ratio (Chelaru & Dragoi (2016)) in the V1. In order to find out
the advantages of keeping negative correlations, we propose to use an exponential to squash the gradients for angles greater than π2 (orthogonal):
C(θ) = n∑ i=1 n∑ j=1,j 6=i log(1 + eλ(cos(θi,θj)−1)) = log(1 + eλ(〈θi,θj〉−1)), ||θi|| = ||θj || = 1 (7)
Where λ is a coefficient that controls the minimum angle-of-influence of the regularizer, i.e. the minimum angle between two feature weights so that there exists a gradient pushing them apart, see Figure 1b. We empirically found that the regularizer worked well for λ = 10, see Figure 2b. Note that when λ ' 10 the loss and the gradients approximate to zero when vectors are at more than π2 (orthogonal). As a result of incorporating the squashing function on the cosine similarity, negatively correlated feature weights will not be regularized. This is different from all previous approaches and the loss presented in eq. 2, where all pairs of weight vectors influence each other. Thus, from now on, the loss in eq. 2 is named as global loss and the loss in eq. 7 is named as local loss.
The derivative of eq. 7 is:
δ
δθ(i,j) C(θ) = n∑ k=1,k 6=i λ eλ〈θi,θk〉θ(k,j) eλ〈θi,θk〉 + eλ (8)
Then, given the element-wise exponential operator exp, we define the following expression in order to simplify the formulas:
Θ̂ = exp(λ(ΘΘt)) (9)
and thus, the ∆ in vectorial form can be formulated as:
∇CΘ = λ (Θ̂− diag(Θ̂))Θ
Θ̂− diag(Θ̂) + eλ (10)
In order to provide a visual example, we have created a 2D toy dataset and used the previous equations for positive and negative γ values, see Figure 2. As expected, it can be seen that the angle between all pairs of adjacent feature weights becomes more uniform after regularization. Note that Figure 2b shows that regularization with the global loss (eq. 2) results in less uniform angles than using the local loss as shown in 2c (which corresponds to the local loss presented in eq. 7) because vectors in opposite quadrants still influence each other. This is why in Figure 2d, it can be seen that the mean nearest neighbor angle using the global loss (b) is more unstable than the local loss (c). As a proof of concept, we also performed gradient ascent, which minimizes the angle between the vectors. Thus, in Figures 2e and 2f, it can be seen that the locality introduced by the local loss reaches a stable configuration where feature weights with angle π2 are too far to attract each other.
The effects of global and local regularizations on Alexnet, VGG-16 and a 50-layer ResNet are shown on Figure 3. As it can be seen, OrthoReg reaches higher decorrelation bounds. Lower decorrelation peaks are still observed when the input dimensionality of the layers is smaller than the output since all vectors cannot be orthogonal at the same time. In this case, local regularization largely outperforms global regularization since it removes interferences caused by negatively correlated feature weights. This suggests why increasing fully connected layers’ size has not improved networks performance.
3 EXPERIMENTS
In this section we provide a set of experiments that verify that (i) training with the proposed regularization increases the performance of naive unregularized models, (ii) negatively correlated feature weights are useful, and (iii) the proposed regularization improves the performance of state-of-the-art models.
3.1 VERIFICATION EXPERIMENTS
As a sanity check, we first train a three-hidden-layer Multi-Layer Perceptron (MLP) with ReLU non-liniarities on the MNIST dataset (LeCun et al. (1998)). Our code is based in the train-a-digit-classifier example included in torch/demos1, which uses an upsampled version of the dataset (32×32). The only pre-processing applied to the data is a global standardization. The model is trained with SGD and a batch size of 200 during 200 epochs. No momentum neither weight decay was applied. By default, the magnitude of the weights of this experiments is recovered after each regularization step in order to prove the regularization only affects their angle.
Sensitivity to hyperparameters. We train a three-hidden-layer MLP with 1024 hidden units, and different γ and λ values so as to verify how they affect the performance of the model. Figure 4a
1https://github.com/torch/demos
shows that the model effectively achieves the best error rate for the highest gamma value (γ = 1), thus proving the advantages of the regularization. On Figure 4b, we verify that higher regularization rates produce more general models. Figure 5a depicts the sensitivity of the model to λ. As expected, the best value is found when lambda corresponds to Orthogonality (λ ' 10). Negative Correlations. Figure 5b highlights the difference between regularizing with the global or the local regularizer. Although both regularizations reach better error rates than the unregularized counterpart, the local regularization is better than the global. This confirms the hypothesis that negative correlations are useful and thus, performance decreases when we reduce them.
Compatibility with initialization and dropout. To demonstrate the proposed regularization can help even when other regularizations are present, we trained a CNN with (i) dropout (c32-c64-l512-d0.5-l10)2 or (ii) LSUV initialization (Mishkin & Matas (2016)). In Table 2, we show that best results are obtained when orthogonal regularization is present. The results are consistent with the hypothesis that OrthoReg, as well as Dropout and LSUV, focuses on reducing the model redundancy. Thus, when one of them is present, the margin of improvement for the others is reduced.
2cxx = convolution with xx filters. lxx = fully-connected with xx units. dxx = dropout with prob xx.
3.2 REGULARIZATION ON CIFAR-10 AND CIFAR-100
We show that the proposed OrthoReg can help to improve the performance of state-of-the-art models such as deep residual networks (He et al. (2015a)). In order to show the regularization is suitable for deep CNNs, we successfuly regularize a 110-layer ResNet3 on CIFAR-10, decreasing its error from 6.55% to 6.29% without data augmentation.
In order to compare with the most recent state-of-the-art, we train a wide residual network (Zagoruyko & Komodakis (November 2016)) on CIFAR-10 and CIFAR-100. The experiment is based on a torch implementation of the 28-layer and 10th width factor wide deep residual model, for which the median error rate on CIFAR-10 is 3.89% and 18.85% on CIFAR-1004. As it can be seen in Figure 6, regularizing with OrthoReg yields the best test error rates compared to the baselines.
The regularization coefficient γ was chosen using grid search although similar values were found for all the experiments, specially if regularization gradients are normalized before adding them to the weights. The regularization was equally applied to all the convolution layers of the (wide) ResNet. We found that, although the regularized models were already using weight decay, dropout, and batch normalization, best error rates were always achieved with OrthoReg.
Table 3 compares the performance of the regularized models with other state-of-the-art results. As it can be seen the regularized model surpasses the state of the art, with a 5.1% relative error improvement on CIFAR-10, and a 1.5% relative error improvement on CIFAR-100.
3.3 REGULARIZATION ON SVHN
For SVHN we follow the procedure depicted in Zagoruyko & Komodakis (May 2016), training a wide residual network of depth=28, width=4, and dropout. Results are shown in Table 4. As it
3https://github.com/gcr/torch-residual-networks 4https://github.com/szagoruyko/wide-residual-networks
can be seen, we reduce the error rate from 1.64% to 1.54%, which is the lowest value reported on this dataset to the best of our knowledge.
4 DISCUSSION
Regularization by feature decorrelation can reduce Neural Networks overfitting even in the presence of other kinds of regularizations. However, especially when the number of feature detectors is higher than the input dimensionality, its decorrelation capacity is limited due to the effects of negatively correlated features. We showed that imposing locality constraints in feature decorrelation removes interferences between negatively correlated feature weights, allowing regularizers to reach higher decorrelation bounds, and reducing the overfitting more effectively.
In particular, we show that the models regularized with the constrained regularization present lower overfitting even when batch normalization and dropout are present. Moreover, since our regularization is directly performed on the weights, it is especially suitable for fully convolutional neural networks, where the weight space is constant compared to the feature map space. As a result, we are able to reduce the overfitting of 110-layer ResNets and wide ResNets on CIFAR-10, CIFAR-100, and SVHN improving their performance. Note that despite OrthoReg consistently improves state of the art ReLU networks, the choice of the activation function could affect regularizers like the one presented in this work. In this sense, the effect of asymmetrical activations on feature correlations and regularizers should be further investigated in the future.
ACKNOWLEDGEMENTS
Authors acknowledge the support of the Spanish project TIN2015-65464-R (MINECO/FEDER), the 2016FI B 01163 grant of Generalitat de Catalunya, and the COST Action IC1307 iV&L Net (European Network on Integrating Vision and Language) supported by COST (European Cooperation in Science and Technology). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of a Tesla K40 GPU and a GTX TITAN GPU, used for this research.

The paper proposes a new regulariser for CNNs that penalises positive correlations between feature weights, but does not affect negative correlations. An alternative version which penalises all correlations regardless of sign is also considered. The paper refers to these as "local" and "global" respectively, which I find a bit confusing as these are very general terms that can mean a plethora of things.
The experimental validation is quite rigorous. Several experiments are conducted on benchmark datasets (MNIST, CIFAR-10, CIFAR-100, SVHN) and improvements are demonstrated in most cases. While these improvements may seem modest, the baselines are already very competitive as the authors pointed out. In some cases it does raise some questions about statistical significance though. More results with the global regulariser (i.e. not just on MNIST) would have been interesting, as the main novelty in the paper seems to be leaving the negative correlations alone, so it would be interesting to see exactly how much of a difference this makes.
One of my main concerns is ambiguity stemming from the fact that the paper sometimes discusses activations and sometimes filter weights, but refers to both as "features". However, the authors have already said they will address this.
The paper somewhat ignores interactions with the choice of nonlinearity, which seems like it could be very important; especially because the goal is to obtain feature activations that are uncorrelated, and this is done only by applying a penalty to the weights (i.e. in a data-agnostic way and also ignoring any nonlinearity). I believe the authors already mentioned in their responses to reviewer questions that this would be addressed, but I think this important and it definitely needs to be discussed.
In response to the authors' answer to my question about the role of biases: as they point out, it is perfectly possible to combine their proposed technique with the "multi-bias" approach, but this was not really my point. Rather, the latter is an example that challenges the idea that features should not be positively correlated / redundant, which seems to be the assumption that this work is built upon. My current intuition is that it's okay to have correlated features, as long as you're not wasting model capacity on them. This is the case for "multi-bias", seeing as the weights are shared across sets of correlated features.
The dichotomy between regularisation methods that reduce capacity and those that don't which is described in the introduction seems a bit arbitrary to me, especially considering that weight decay is counted among the former and the proposed method is counted among the latter. I think this very much depends on ones definition of model capacity (clearly weight decay does not actually reduce the number of parameters in a model).
Overall, the work is perhaps a bit incremental, but it seems to be well-executed. The results are convincing, even if they aren't particularly ground-breaking.


HIERARCHICAL MEMORY NETWORKS
Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which require the network to read from extremely large memories. On the other hand, it is well known that hard attention mechanisms based on reinforcement learning are challenging to train successfully. In this paper, we explore a form of hierarchical memory network, which can be considered as a hybrid between hard and soft attention memory networks. The memory is organized in a hierarchical structure such that reading from it is done with less computation than soft attention over a flat memory, while also being easier to train than hard attention over a flat memory. Specifically, we propose to incorporate Maximum Inner Product Search (MIPS) in the training and inference procedures for our hierarchical memory network. We explore the use of various state-of-the art approximate MIPS techniques and report results on SimpleQuestions, a challenging large scale factoid question answering task.
1 INTRODUCTION
Until recently, traditional machine learning approaches for challenging tasks such as image captioning, object detection, or machine translation have consisted in complex pipelines of algorithms, each being separately tuned for better performance. With the recent success of neural networks and deep learning research, it has now become possible to train a single model end-to-end, using backpropagation. Such end-to-end systems often outperform traditional approaches, since the entire model is directly optimized with respect to the final task at hand. However, simple encode-decode style neural networks often underperform on knowledge-based reasoning tasks like question-answering or dialog systems. Indeed, in such cases it is nearly impossible for regular neural networks to store all the necessary knowledge in their parameters.
Neural networks with memory (Graves et al., 2014; Weston et al., 2015b) can deal with knowledge bases by having an external memory component which can be used to explicitly store knowledge. The memory is accessed by reader and writer functions, which are both made differentiable so that the entire architecture (neural network, reader, writer and memory components) can be trained end-to-end using backpropagation. Memory-based architectures can also be considered as generalizations of RNNs and LSTMs, where the memory is analogous to recurrent hidden states. However they are much richer in structure and can handle very long-term dependencies because once a vector (i.e., a memory) is stored, it is copied from time step to time step and can thus stay there for a very long time (and gradients correspondingly flow back time unhampered).
There exists several variants of neural networks with a memory component: Memory Networks (Weston et al., 2015b), Neural Turing Machines (NTM) (Graves et al., 2014), Dynamic Memory Net-
∗Corresponding author: apsarathchandar@gmail.com
works (DMN) (Kumar et al., 2015). They all share five major components: memory, input module, reader, writer, and output module.
Memory: The memory is an array of cells, each capable of storing a vector. The memory is often initialized with external data (e.g. a database of facts), by filling in its cells with a pre-trained vector representations of that data.
Input module: The input module is to compute a representation of the input that can be used by other modules.
Writer: The writer takes the input representation and updates the memory based on it. The writer can be as simple as filling the slots in the memory with input vectors in a sequential way (as often done in memory networks). If the memory is bounded, instead of sequential writing, the writer has to decide where to write and when to rewrite cells (as often done in NTMs).
Reader: Given an input and the current state of the memory, the reader retrieves content from the memory, which will then be used by an output module. This often requires comparing the input’s representation or a function of the recurrent state with memory cells using some scoring function such as a dot product.
Output module: Given the content retrieved by the reader, the output module generates a prediction, which often takes the form of a conditional distribution over multiple labels for the output.
For the rest of the paper, we will use the name memory network to describe any model which has any form of these five components. We would like to highlight that all the components except the memory are learnable. Depending on the application, any of these components can also be fixed. In this paper, we will focus on the situation where a network does not write and only reads from the memory.
In this paper, we focus on the application of memory networks to large-scale tasks. Specifically, we focus on large scale factoid question answering. For this problem, given a large set of facts and a natural language question, the goal of the system is to answer the question by retrieving the supporting fact for that question, from which the answer can be derived. Application of memory networks to this task has been studied by Bordes et al. (2015). However, Bordes et al. (2015) depended on keyword based heuristics to filter the facts to a smaller set which is manageable for training. However heuristics are invariably dataset dependent and we are interested in a more general solution which can be used when the facts are of any structure. One can design soft attention retrieval mechanisms, where a convex combination of all the cells is retrieved or design hard attention retrieval mechanisms where one or few cells from the memory are retrieved. Soft attention is achieved by using softmax over the memory which makes the reader differentiable and hence learning can be done using gradient descent. Hard attention is achieved by using methods like REINFORCE (Williams, 1992), which provides a noisy gradient estimate when discrete stochastic decisions are made by a model.
Both soft attention and hard attention have limitations. As the size of the memory grows, soft attention using softmax weighting is not scalable. It is computationally very expensive, since its complexity is linear in the size of the memory. Also, at initialization, gradients are dispersed so much that it can reduce the effectiveness of gradient descent. These problems can be alleviated by a hard attention mechanism, for which the training method of choice is REINFORCE. However, REINFORCE can be brittle due to its high variance and existing variance reduction techniques are complex. Thus, it is rarely used in memory networks (even in cases of a small memory).
In this paper, we propose a new memory selection mechanism based on Maximum Inner Product Search (MIPS) which is both scalable and easy to train. This can be considered as a hybrid of soft and hard attention mechanisms. The key idea is to structure the memory in a hierarchical way such that it is easy to perform MIPS, hence the name Hierarchical Memory Network (HMN). HMNs are scalable at both training and inference time. The main contributions of the paper are as follows:
• We explore hierarchical memory networks, where the memory is organized in a hierarchical fashion, which allows the reader to efficiently access only a subset of the memory.
• While there are several ways to decide which subset to access, we propose to pose memory access as a maximum inner product search (MIPS) problem.
• We empirically show that exact MIPS-based algorithms not only enjoy similar convergence as soft attention models, but can even improve the performance of the memory network. • Since exact MIPS is as computationally expensive as a full soft attention model, we propose
to train the memory networks using approximate MIPS techniques for scalable memory access. • We empirically show that unlike exact MIPS, approximate MIPS algorithms provide a
speedup and scalability of training, though at the cost of some performance.
2 HIERARCHICAL MEMORY NETWORKS
In this section, we describe the proposed Hierarchical Memory Network (HMN). In this paper, HMNs only differ from regular memory networks in two of its components: the memory and the reader.
Memory: Instead of a flat array of cells for the memory structure, HMNs leverages a hierarchical memory structure. Memory cells are organized into groups and the groups can further be organized into higher level groups. The choice for the memory structure is tightly coupled with the choice of reader, which is essential for fast memory access. We consider three classes of approaches for the memory’s structure: hashing-based approaches, tree-based approaches, and clustering-based approaches. This is explained in detail in the next section.
Reader: The reader in the HMN is different from the readers in flat memory networks. Flat memorybased readers use either soft attention over the entire memory or hard attention that retrieves a single cell. While these mechanisms might work with small memories, with HMNs we are more interested in achieving scalability towards very large memories. So instead, HMN readers use soft attention only over a selected subset of the memory. Selecting memory subsets is guided by a maximum inner product search algorithm, which can exploit the hierarchical structure of the organized memory to retrieve the most relevant facts in sub-linear time. The MIPS-based reader is explained in more detail in the next section.
In HMNs, the reader is thus trained to create MIPS queries such that it can retrieve a sufficient set of facts. While most of the standard applications of MIPS (Ram & Gray, 2012; Bachrach et al., 2014; Shrivastava & Li, 2014) so far have focused on settings where both query vector and database (memory) vectors are precomputed and fixed, memory readers in HMNs are learning to do MIPS by updating the input representation such that the result of MIPS retrieval contains the correct fact(s).
3 MEMORY READER WITH K-MIPS ATTENTION
In this section, we describe how the HMN memory reader uses Maximum Inner Product Search (MIPS) during learning and inference.
We begin with a formal definition of K-MIPS. Given a set of points X = {x1, . . . , xn} and a query vector q, our goal is to find
argmax (K) i∈X q >xi (1)
where the argmax(K) returns the indices of the top-K maximum values. In the case of HMNs, X corresponds to the memory and q corresponds to the vector computed by the input module.
A simple but inefficient solution for K-MIPS involves a linear search over the cells in memory by performing the dot product of q with all the memory cells. While this will return the exact result for K-MIPS, it is too costly to perform when we deal with a large-scale memory. However, in many practical applications, it is often sufficient to have an approximate result for K-MIPS, trading speed-up at the cost of the accuracy. There exist several approximate K-MIPS solutions in the literature (Shrivastava & Li, 2014; 2015; Bachrach et al., 2014; Neyshabur & Srebro, 2015).
All the approximate K-MIPS solutions add a form of hierarchical structure to the memory and visit only a subset of the memory cells to find the maximum inner product for a given query. Hashingbased approaches (Shrivastava & Li, 2014; 2015; Neyshabur & Srebro, 2015) hash cells into multiple bins, and given a query they search for K-MIPS cell vectors only in bins that are close to the bin
associated with the query. Tree-based approaches (Ram & Gray, 2012; Bachrach et al., 2014) create search trees with cells in the leaves of the tree. Given a query, a path in the tree is followed and MIPS is performed only for the leaf for the chosen path. Clustering-based approaches (Auvolat et al., 2015) cluster cells into multiple clusters (or a hierarchy of clusters) and given a query, they perform MIPS on the centroids of the top few clusters. We refer the readers to (Auvolat et al., 2015) for an extensive comparison of various state-of-the-art approaches for approximate K-MIPS.
Our proposal is to exploit this rich approximate K-MIPS literature to achieve scalable training and inference in HMNs. Instead of filtering the memory with heuristics, we propose to organize the memory based on approximate K-MIPS algorithms and then train the reader to learn to perform MIPS. Specifically, consider the following softmax over the memory which the reader has to perform for every reading step to retrieve a set of relevant candidates:
Rout = softmax(h(q)M T ) (2)
where h(q) ∈ Rd is the representation of the query, M ∈ RN×d is the memory with N being the total number of cells in the memory. We propose to replace this softmax with softmax(K) which is defined as follows:
C = argmax(K) h(q)MT (3)
Rout = softmax (K)(h(q)MT ) = softmax(h(q)M [C]T ) (4)
where C is the indices of top-K MIP candidate cells and M [C] is a sub-matrix of M where the rows are indexed by C.
One advantage of using the softmax(K) is that it naturally focuses on cells that would normally receive the strongest gradients during learning. That is, in a full softmax, the gradients are otherwise more dispersed across cells, given the large number of cells and despite many contributing a small gradient. As our experiments will show, this results in slower training.
One problematic situation when learning with the softmax(K) is when we are at the initial stages of training and the K-MIPS reader is not including the correct fact candidate. To avoid this issue, we always include the correct candidate to the top-K candidates retrieved by the K-MIPS algorithm, effectively performing a fully supervised form of learning.
During training, the reader is updated by backpropagation from the output module, through the subset of memory cells. Additionally, the log-likelihood of the correct fact computed using Ksoftmax is also maximized. This second supervision helps the reader learn to modify the query such that the maximum inner product of the query with respect to the memory will yield the correct supporting fact in the top K candidate set.
Until now, we described the exact K-MIPS-based learning framework, which still requires a linear look-up over all memory cells and would be prohibitive for large-scale memories. In such scenarios, we can replace the exact K-MIPS in the training procedure with the approximate K-MIPS. This is achieved by deploying a suitable memory hierarchical structure. The same approximate K-MIPSbased reader can be used during inference stage as well. Of course, approximate K-MIPS algorithms might not return the exact MIPS candidates and will likely to hurt performance, but at the benefit of achieving scalability.
While the memory representation is fixed in this paper, updating the memory along with the query representation should improve the likelihood of choosing the correct fact. However, updating the memory will reduce the precision of the approximate K-MIPS algorithms, since all of them assume that the vectors in the memory are static. Designing efficient dynamic K-MIPS should improve the performance of HMNs even further, a challenge that we hope to address in future work.
3.1 READER WITH CLUSTERING-BASED APPROXIMATE K-MIPS
Clustering-based approximate K-MIPS was proposed in (Auvolat et al., 2015) and it has been shown to outperform various other state-of-the-art data dependent and data independent approximate KMIPS approaches for inference tasks. As we will show in the experiments section, clustering-based MIPS also performs better when used to training HMNs. Hence, we focus our presentation on the clustering-based approach and propose changes that were found to be helpful for learning HMNs.
Following most of the other approximate K-MIPS algorithms, Auvolat et al. (2015) convert MIPS to Maximum Cosine Similarity Search (MCSS) problem:
argmax (K) i∈X qTxi ||q|| ||xi|| = argmax (K) i∈X qTxi ||xi||
(5)
When all the data vectors xi have the same norm, then MCSS is equivalent to MIPS. However, it is often restrictive to have this additional constraint. Instead, Auvolat et al. (2015) append additional dimensions to both query and data vectors to convert MIPS to MCSS. In HMN terminology, this would correspond to adding a few more dimensions to the memory cells and input representations.
The algorithm introduces two hyper-parameters, U < 1 and m ∈ N∗. The first step is to scale all the vectors in the memory by the same factor, such that maxi ||xi||2 = U . We then apply two mappings, P and Q, on the memory cells and on the input vector, respectively. These two mappings simply concatenate m new components to the vectors and make the norms of the data points all roughly the same (Shrivastava & Li, 2015). The mappings are defined as follows:
P (x) = [x, 1/2− ||x||22, 1/2− ||x||42, . . . , 1/2− ||x||2 m
2 ] (6) Q(x) = [x, 0, 0, . . . , 0] (7)
We thus have the following approximation of MIPS by MCSS for any query vector q:
argmax (K) i q >xi ' argmax(K)i Q(q)>P (xi)
||Q(q)||2 · ||P (xi)||2 (8)
Once we convert MIPS to MCSS, we can use spherical K-means (Zhong, 2005) or its hierarchical version to approximate and speedup the cosine similarity search. Once the memory is clustered, then every read operation requires only K dot-products, where K is the number of cluster centroids.
Since this is an approximation, it is error-prone. As we are using this approximation for the learning process, this introduces some bias in gradients, which can affect the overall performance of HMN. To alleviate this bias, we propose three simple strategies.
• Instead of using only the top-K candidates for a single read query, we also add top-K candidates retrieved for every other read query in the mini-batch. This serves two purposes. First, we can do efficient matrix multiplications by leveraging GPUs since all the K-softmax in a minibatch are over the same set of elements. Second, this also helps to decrease the bias introduced by the approximation error.
• For every read access, instead of only using the top few clusters which has a maximum product with the read query, we also sample some clusters from the rest, based on a probability distribution log-proportional to the dot product with the cluster centroids. This also decreases the bias.
• We can also sample random blocks of memory and add it to top-K candidates.
We empirically investigate the effect of these variations in Section 5.5.
4 RELATED WORK
Memory networks have been introduced in (Weston et al., 2015b) and have been so far applied to comprehension-based question answering (Weston et al., 2015a; Sukhbaatar et al., 2015), large scale question answering (Bordes et al., 2015) and dialogue systems (Dodge et al., 2015). While (Weston et al., 2015b) considered supervised memory networks in which the correct supporting fact is given during the training stage, (Sukhbaatar et al., 2015) introduced semi-supervised memory networks that can learn the supporting fact by itself. (Kumar et al., 2015; Xiong et al., 2016) introduced Dynamic Memory Networks (DMNs) which can be considered as a memory network with two types of memory: a regular large memory and an episodic memory. Another related class of model is the Neural Turing Machine (Graves et al., 2014), which uses softmax-based soft attention. Later (Zaremba & Sutskever, 2015) extended NTM to hard attention using reinforcement learning. (Dodge et al., 2015; Bordes et al., 2015) alleviate the problem of the scalability of soft attention by having
an initial keyword based filtering stage, which reduces the number of facts being considered. Our work generalizes this filtering by using MIPS for filtering. This is desirable because MIPS can be applied for any modality of data or even when there is no overlap between the words in a question and the words in facts.
The softmax arises in various situations and most relevant to this work are scaling methods for large vocabulary neural language modeling. In neural language modeling, the final layer is a softmax distribution over the next word and there exist several approaches to achieve scalability. (Morin & Bengio, 2005) proposes a hierarchical softmax based on prior clustering of the words into a binary, or more generally n-ary tree, that serves as a fixed structure for the learning process of the model. The complexity of training is reduced from O(n) to O(log n). Due to its clustering and tree structure, it resembles the clustering-based MIPS techniques we explore in this paper. However, the approaches differ at a fundamental level. Hierarchical softmax defines the probability of a leaf node as the product of all the probabilities computed by all the intermediate softmaxes on the way to that leaf node. By contrast, an approximate MIPS search imposes no such constraining structure on the probabilistic model, and is better thought as efficiently searching for top winners of what amounts to be a large ordinary flat softmax. Other methods such as Noice Constrastive Estimation (Mnih & Gregor, 2014) and Negative Sampling (Mikolov et al., 2013) avoid an expensive normalization constant by sampling negative samples from some marginal distribution. By contrast, our approach approximates the softmax by explicitly including in its negative samples candidates that likely would have a large softmax value. Jean et al. (2015) introduces an importance sampling approach that considers all the words in a mini-batch as the candidate set. This in general might also not include the MIPS candidates with highest softmax values.
(Spring & Shrivastava, 2016) is the only work that we know of, proposing to use MIPS during learning. It proposes hashing-based MIPS to sort the hidden layer activations and reduce the computation in every layer. However, a small scale application was considered and data-independent methods like hashing will likely suffer as dimensionality increases. Rae et al. (2016) have also independently proposed a model called SAM to use approximate search methods for memory access in NTM-like architectures. However, our motivation is different. While Rae et al. (2016) focus on architectures where the memory is written by the controller itself, we focus on handling memory access to large external knowledge bases. While both the models fix the memory access mechanism (HMN uses MIPS and SAM uses NNS), our controller works in a much more constrained setting. Moreover, our experiments suggest that the performance of SAM could be improved using a clustering-based approach as in our work, instead of tree/hash-based approaches for memory search used by SAM.
5 EXPERIMENTS
In this section, we report experiments on factoid question answering using hierarchical memory networks. Specifically, we use the SimpleQuestions dataset Bordes et al. (2015). The aim of these experiments is not to achieve state-of-the-art results on this dataset. Rather, we aim to propose and analyze various approaches to make memory networks more scalable and explore the achieved tradeoffs between speed and accuracy.
5.1 DATASET
We use SimpleQuestions (Bordes et al., 2015) which is a large scale factoid question answering dataset. SimpleQuestions consists of 108,442 natural language questions, each paired with a corresponding fact from Freebase. Each fact is a triple (subject,relation,object) and the answer to the question is always the object. The dataset is divided into training (75910), validation (10845), and test (21687) sets. Unlike Bordes et al. (2015) who additionally considered FB2M (10M facts) or FB5M (12M facts) with keyword-based heuristics for filtering most of the facts for each question, we only use SimpleQuestions, with no keyword-based heuristics. This allows us to do a direct comparison with the full softmax approach in a reasonable amount of time. Moreover, we would like to highlight that for this dataset, keyword-based filtering is a very efficient heuristic since all questions have an appropriate source entity with a matching word. Nevertheless, our goal is to design a general purpose architecture without such strong assumptions on the nature of the data.
5.2 MODEL
Let Vq be the vocabulary of all words in the natural language questions. Let Wq be a |Vq| ∗ m matrix where each row is some m dimensional embedding for a word in the question vocabulary. This matrix is initialized with random values and learned during training. Given any question, we represent it with a bag-of-words representation by summing the vector representation of each word in the question. Let q = {wi}pi=1,
h(q) = p∑ i=1 Wq[wi]
Then, to find the relevant fact from the memory M, we call the K-MIPS-based reader module with h(q) as the query. This uses Equation 3 and 4 to compute the output of the reader Rout. The reader is trained by minimizing the Negative Log Likelihood (NLL) of the correct fact.
Jθ = N∑ i=1 −log(Rout[fi])
where fi is the index of the correct fact in Wm. We are fixing the memory embeddings to the TransE (Bordes et al., 2013) embeddings and learning only the question embeddings.
This model is simpler than the one reported in (Bordes et al., 2015) so that it is esay to analyze the effect of various memory reading strategies.
5.3 TRAINING DETAILS
We trained the model with the Adam optimizer (Kingma & Ba, 2014), with a fixed learning rate of 0.001. We used mini-batches of size 128. We used 200 dimensional embeddings for the TransE entities, yielding 600 dimensional embeddings for facts by concatenating the embeddings of the subject, relation and object. We also experimented with summing the entities in the triple instead of concatenating, but we found that it was difficult for the model to differentiate facts this way. The only learnable parameters by the HMN model are the question word embeddings. The entity distribution in SimpleQuestions is extremely sparse and hence, following Bordes et al. (2015), we also add artificial questions for all the facts for which we do not have natural language questions. Unlike Bordes et al. (2015), we do not add any other additional tasks like paraphrase detection to the model, mainly to study the effect of the reader. We stopped training for all the models when the validation accuracy consistently decreased for 3 epochs.
5.4 EXACT K-MIPS IMPROVES ACCURACY
In this section, we compare the performance of the full soft attention reader and exact K-MIPS attention readers. Our goal is to verify that K-MIPS attention is in fact a valid and useful attention mechanism and see how it fares when compared to full soft attention. For K-MIPS attention, we tried K ∈ 10, 50, 100, 1000. We would like to emphasize that, at training time, along with K candidates for a particular question, we also add the K-candidates for each question in the minibatch. So the exact size of the softmax layer would be higer than K during training. In Table 1, we report the test performance of memory networks using the soft attention reader and K-MIPS attention reader. We also report the average softmax size during training. From the table, it is clear that the K-MIPS attention readers improve the performance of the network compared to soft attention reader. In fact, smaller the value of K is, better the performance. This result suggests that it is better to use a K-MIPS layer instead of softmax layer whenever possible. It is interesting to see that the convergence of the model is not slowed down due to this change in softmax computation (as shown in Figure 1).
This experiment confirms the usefulness of K-MIPS attention. However, exact K-MIPS has the same complexity as a full softmax. Hence, to scale up the training, we need more efficient forms of K-MIPS attention, which is the focus of next experiment.
Model Test Acc. Avg. Softmax Size Full-softmax 59.5 108442
10-MIPS 62.2 1290 50-MIPS 61.2 6180
100-MIPS 60.6 11928 1000-MIPS 59.6 70941 Clustering 51.5 20006 PCA-Tree 32.4 21108 WTA-Hash 40.2 20008
Table 1: Accuracy in SQ test-set and average size of memory used. 10-softmax has high performance while using only smaller amount of memory.
5.5 APPROXIMATE K-MIPS BASED LEARNING
As mentioned previously, designing faster algorithms for K-MIPS is an active area of research. Auvolat et al. (2015) compared several state-of-the-art data-dependent and data-independent methods for faster approximate K-MIPS and it was found that clustering-based MIPS performs significantly better than other approaches. However the focus of the comparison was on performance during the inference stage. In HMNs, K-MIPS must be used at both training stage and inference stages. To verify if the same trend can been seen during learning stage as well, we compared three different approaches:
Clustering: This was explained in detail in section 3.
WTA-Hash: Winner Takes All hashing (Vijayanarasimhan et al., 2014) is a hashing-based K-MIPS algorithm which also converts MIPS to MCSS by augmenting additional dimensions to the vectors. This method used n hash functions and each hash function does p different random permutations of the vector. Then the prefix constituted by the first k elements of each permuted vector is used to construct the hash for the vector.
PCA-Tree: PCA-Tree (Bachrach et al., 2014) is the state-of-the-art tree-based method, which converts MIPS to NNS by vector augmentation. It uses the principal components of the data to construct a balanced binary tree with data residing in the leaves.
For a fair comparison, we varied the hyper-parameters of each algorithm in such a way that the average speedup is approximately the same. Table 1 shows the performance of all three methods, compared to a full softmax. From the table, it is clear that the clustering-based method performs significantly better than the other two methods. However, performances are lower when compared to the performance of the full softmax.
As a next experiment, we analyze various the strategies proposed in Section 3.1 to reduce the approximation bias of clustering-based K-MIPS:
Top-K: This strategy picks the vectors in the top K clusters as candidates.
Sample-K: This strategy samples K clusters, without replacement, based on a probability distribution based on the dot product of the query with the cluster centroids. When combined with the Top-K strategy, we ignore clusters selected by the Top-k strategy for sampling.
Rand-block: This strategy divides the memory into several blocks and uniformly samples a random block as candidate.
We experimented with 1000 clusters and 2000 clusters. While comparing various training strategies, we made sure that the effective speedup is approximately the same. Memory access to facts per query for all the models is approximately 20,000, hence yielding a 5X speedup.
Results are given in Table 2. We observe that the best approach is to combine the Top-K and SampleK strategies, with Rand-block not being beneficial. Interestingly, the worst performances correspond to cases where the Sample-K strategy is ignored.
6 CONCLUSION
In this paper, we proposed a hierarchical memory network that exploits K-MIPS for its attentionbased reader. Unlike soft attention readers, K-MIPS attention reader is easily scalable to larger memories. This is achieved by organizing the memory in a hierarchical way. Experiments on the SimpleQuestions dataset demonstrate that exact K-MIPS attention is better than soft attention. However, existing state-of-the-art approximate K-MIPS techniques provide a speedup at the cost of some accuracy. Future research will investigate designing efficient dynamic K-MIPS algorithms, where the memory can be dynamically updated during training. This should reduce the approximation bias and hence improve the overall performance.

The paper proposes an algorithm for training memory networks which have very large memories. Training such models in traditional ways, by using soft-attention mechanism over all the memory slots is not only slow, it is also harder to train due to dispersion of gradients. The paper proposes to use the k-mips algorithm over the memories to choose a subset of the memory slots over which the attention is applied. Since the cost of exact k-mips is the same as doing full attention, the authors propose to use approximate k-mips, which while faster to compute, results in inferior performance. An artifact of using k-mips is that one cannot learn the memory slots. Hence they are pre-trained and kept fixed during entire training. The experimental section shows the efficacy of using k-mips using the SimpleQuestions dataset. The exact k-mips results in the same performance as the full attention. The approximate k-mips results in deterioration in performance. The paper is quite clearly written and easy to understand.
I think the ideas proposed in the paper are not super convincing. I have a number of issues with this paper.
1. The k-mips algorithm forces the memories to be fixed. This to me is a rather limiting constraint, especially on problems/dataset which will require multiple hops of training to do compounded reasoning. As a results I'm not entirely sure about the usefulness of this technique.
2. Furthermore, the exact k-mips is the sample complexity as the full attention. The only way to achieve speedup is to use approx k-mips. That, as expected, results in a significant drop in performance.
3. The paper motivates the ideas by proposing solutions to eliminate heuristics used to prune the memories. However in Section 3.1 the authors themselves end up using multiple heuristics to make the training work. Agreed, that the used heuristics are not data dependent, but still, it feels like they are kicking the can down the road as far as heuristics are concerned.
4. The experimental results are not very convincing. First there is no speed comparison. Second, the authors do not compare with methods other than k-mips which do fast nearest neighbor search, such as, FLANN.


One of the main difficulties in applying deep neural nets (DNNs) to new domains is the need to explore multiple architectures in order to discover ones that perform well. We analyze a large set of DNNs across multiple domains and derive insights regarding their effectiveness. We also analyze the characteristics of various DNNs and the general effect they may have on performance. Finally, we explore the application of meta-learning to the problem of architecture ranking. We demonstrate that by using topological features and modeling the changes in its weights, biases and activation functions layers of the initial training steps, we are able to rank architectures based on their predicted performance. We consider this work to be a first step in the important and challenging direction of exploring the space of different neural network architectures.
1 INTRODUCTION
Recent advances in deep neural networks (DNNs) have led to breakthroughs in fields such as image classification (He et al., 2015; Krizhevsky et al., 2012) and speech recognition (Yu et al., 2010; Dahl et al., 2012). One reason for the effectiveness of DNNs is their ability to integrate low, mid and highlevel features in a natural way (Zeiler & Fergus, 2014). While recent work such as (Simonyan & Zisserman, 2014) suggests that in many cases the depth of the architecture is crucial, the emergence of more complex architectures (He et al., 2015; Szegedy et al., 2015) demonstrates that depth alone often does not suffice.
While DNNs have been highly effective in several domains, their application in additional fields is yet to become widespread. We argue that this is the case due to two challenges. The first is the difficulty of designing effective architectures for domains in which there is little or no previous knowledge on the application of deep learning. Moreover, since designing DNN architectures is not intuitive for most people, this task is likely to fall to experts whose time is in high demand. The second challenge, which is strongly coupled with the first, is the large amounts of computing power and time required to evaluate multiple DNNs. These traits constrain the number of DNN architectures that can be evaluated, thus further limiting one’s ability to explore new architectures or respond to changing circumstances.
In this study we explore the possibility of applying architectures that are effective for one domain to another. We do so by generating a large number of architectures and evaluate their performance on multiple tabular datasets in order to determine whether the architectures are transferable. We also explore the feasibility of architectures with parallel layers and compare their effectiveness to that of their “linear” counterparts. Our results show that while architectures do not perform well across multiple datasets, parallel architectures are surprisingly effective.
When attempting to apply DNNs to an unknown domain, one way of approaching the problem would be to randomly “sample” various architectures and analyze their performance distribution. The top-performing architectures found in the sampling can form the base for future exploration while the variance in performance can assist in determining the number of architectures that need to be sampled. We explore a meta-learning approach that may improve the efficiency of this process by ranking the architectures based on their expected performance. Our approach models the topology of the DNN as well as the changes in weights, biases and activation function layers throughout the initial training steps and uses this information to rank the architectures by their relative performance. Preliminary results are encouraging.
While we consider this study to be an important first step, we feel obliged to point out that work is done in a limited setting. To enable the generation of multiple DNN architectures with diverse topologies, we applied uniform and fixed parameters such as layer sizes and learning rates. As a result, the architecture space we explore is limited. Validating our results on a more diverse set of architectures with multiple hyperparameter configuration will require additional experimentation. We plan to address these issues in future work.
Our contributions are as follows:
• We explore DNNs across multiple datasets, evaluate their effectiveness and analyze if some perform best across datasets.
• We systematically evaluate a large number of architectures over multiple supervisedclassification datasets and derive insights regarding the design and application of DNNs with parallel layers for general classification problems.
• We present a novel meta learning-based ranking method that utilizes both topological features as well as weights, biases and activation function layers of the various components of the DNN architecture during the initial training phase. To the best of our knowledge, this is the first time these characteristics have been used in a meta-learning scheme. Preliminary results of this approach are promising.
2 RELATED WORK
We review two areas of research whose aim is to better understand and improve the performance of DNN architectures. The first is area of research focuses on the exploration and analysis of DNN architectures. The second area of research is automatic parameter tuning.
2.1 EXPLORATION AND ANALYSIS OF DNN ARCHITECTURES
Despite their remarkable success in various domains, the inner-workings of DNNs remain to some degree a “black box”. Multiple studies attempted to provide insight into this matter. In Jarrett et al. (2009), the authors analyze convolutional neural networks (CNNs) and derive insights regarding the architecture design and the contribution of its different components. Another work aimed at better understanding CNNs is presented in Shang et al. (2016). The authors analyze widely used CNN architectures and derive insights into their possible shortcomings. To address these shortcomings, they propose a new version of the popular ReLU activation scheme.
The exploration of DNN architectures has also taken place for recurrent neural networks (RNNs). In Zaremba (2015), the authors explore various modifications to LSTM architectures to improve their performance, and propose several enhancements to the architecture. Another study Wu & King (2016) aims to determine the reasons for the effectiveness of LSTMs and identify the contribution of its different elements. Based on their conclusions, the authors proposed a simplified version of LSTM.
2.2 AUTOMATIC DNN PARAMETER TUNING
The ability to automatically tune the hyperparameters of a DNN architecture is important not only because of its ability to improve performance, but also due to the considerable time it can potentially save. In Maclaurin et al. (2015) the authors demonstrate how information extracted from the stochastic gradient descent can efficiently tune multiple parameters in the architecture. An additional work that analyzes the gradient is presented in Duvenaud et al. (2016), where the information is used to determine when to terminate the training of the architecture to avoid over-fitting. A different optimization approach is presented in Mendoza et al., where the authors define a large set of hyperparameters (batch size, learning rate, activation types, etc.) and apply Bayesian optimization on top-performing configurations. The approach is only applied to feed-forward networks and outperforms human experts by 10%, using the AUC measure.
Additional types of optimization have also been proposed in recent years. In Jin et al. (2016), the authors focus on setting the size of hidden layers in RNNs. They accomplish this by converting the optimization problem into a subset selection problem. An important aspect of this approach is
that it takes time constraints into account, thus enabling solutions that are feasible given available resources. Another approach, in which one long-short term memory network (LSTM) is used to optimize another, was proposed by Andrychowicz et al. (2016). The two networks have shared parameters but separate hidden states and the optimizer network is both modifying its own weights and those of the optimized simultaneously. Finally, an approach that automatically adjusts the learning rates of the neural net was presented in Schaul et al. (2013). The approach has been shown to be effective both on convex and non-convex learning tasks.
Recent work by Li et al. (2016) proposes an exploration/exploitation scheme for hyperparameter tuning. The authors apply a multi-arm bandits algorithm, with each arm representing a parameter configuration. A process of successive halving (Jamieson & Talwalkar, 2015), in which a certain percentage of the lowest-performing configurations are dropped every n steps enables the framework to explore promising directions. We consider this approach complementary to our proposed metalearning approach, as the former enables exploration of a large number of configurations while the latter can reduce time required to assess their performance.
3 PROBLEM DEFINITION
As mentioned in Section 1, one of the challenges in applying deep learning to a new field is the need to design and test multiple DNN architectures. Only by iterative testing can practitioners discover the capabilities and limitations of deep learning in the domain. Even with ever-increasing computing power, the high computational cost of this process currently presents a significant barrier for most practitioners.
This limitation leads us to explore the following questions:
1. Would DNN architectures that perform well on one general supervised classification problem also be effective when applied to dataset in other domains?
2. What types of architectures are effective for general supervised learning problems? Should practitioners consider other types architectures besides “deep”?
3. Can DNN architectures outperform “conventional” machine learning classifiers in general supervised problems?
4. Is it possible to identify top-performing networks in the early stages of the training? if possible, such a technique could preserve valuable computing resources.
We attempt to begin addressing these questions in the subsequent sections of this study. We iteratively evaluate a large number of DNN architectures on a set of supervised classification problems. These datasets differ from those of image and speech classification in that they consist of tabular data with both numeric and discrete features. These differences make it unclear what types of architectures are likely to perform well on these domains. The datasets we analyze were selected because of their diversity in terms of size and feature number and composition. These traits also enable us to better understand the difficulties in applying DNN architectures across multiple domains.
In order to provide meaningful results, the set of architectures we evaluate is also diverse. We therefore automatically generate a diverse set of architecture with various topological traits. Because little information is available on the application of deep learning to general supervised classification problems, we choose to explore not only architectures that are linear but also architectures with parallel layers. While the generate set is diverse, additional work is required in order to model additional types of architectures. We elaborate on these points further in the subsequent section.
4 GENERATING MULTIPLE DNN ARCHITECTURES
In order to effectively explore the architecture space, we require a large and diverse set. We create this set by automatically generating a large number of architectures and training each of them on all training set datasets. Our generation algorithm, presented in Algorithm 1, generates both “deep” and “wide” architectures with parallel layers (see Figure 1(b)). Next we describe the generation process.
We consider DNN architectures to consist of components. We define a component as any part of an architecture, be it a layer, normalization or activation function. In this study we consider
the following components: fully-connected layers, softmax, batch normalization, dropout and the ReLU, sigmoid and tanh activation functions.
We begin the generation process with a “basic” architecture consisting only of two components: a fully-connected input layer and an output softmax layer. We then expand the set of possible architectures by iteratively applying the following steps:
1. For each pair of components in the architecture, identify all component that could be inserted between them (Figure 1(a)).
2. For each pair of components in the architecture, identify all component that could be inserted in parallel to one of them (Figure 1(b)).
3. For each of the components identified in the previous steps, generate a new copy of the architecture and perform the corresponding insertion.
Our proposed architecture generation approach enables us to generate the topological representation of every possible neural networks that consist of the predefined components. However, we do not generate multiple hyperparameter configurations for each topology and use fixed parameters for each component. We plan to address this limitation in future work, possibly by using an approach similar to the one presented in Li et al. (2016). It is also important to point out that we currently do not support weight-sharing and therefore do not consider CNN and RNN architectures. Given the characteristics of the analyzed data, we do not consider these architecture types likely to produce meaningful results.
Another important aspect of the our architecture generation approach is that we generate architectures with connections between layers of various depths. An example of this is shown in Figure 1(b), where we connect layers of depths 1 and 2. This setting enables us to systematically explore more complex designs than those commonly used. We analyze these architectures further in Section 6.
As the number of possible architectures grows exponentially, we limit the total number of architectures that we generate by constraining the maximal number of components in a architecture and the number of parallel layers an architecture may contain. The specific settings used in our experiments are presented in Section 6.1. These settings were chosen in order to ensure a diverse set of both deep and wide architectures given the time and computing-power constraints, and we plan to change them in future work to further diversify the set of generated architectures. To select the architectures from which additional ones will be generated, we apply a priority queue. We first sort the architectures by the number of their activation layers (in a descending order) with a secondary sorting based on the total number of components (in an ascending order). This setting prioritizes the creation of deeper architectures with multiple activation layers. For each architecture in the final set, we generate the
meta-features described in Section 5. The algorithm for the architecture generation is presented in Algorithm 1.
Algorithm 1 Automatic architecture generation 1: procedure ARCHITECTUREGENERATION(arcQueue, initArc) 2: architecturesSet← initArc 3: architecturesQueue← initArc 4: while (architecturesQueue 6= ∅) do 5: newarchitectures← ∅ 6: architecture← arcQueue.pop() 7: for each P (ci, cj)i 6= j ∈ {c1, c2, ..., cn} do 8: candidateComponents← proposeInsertBetweenCandidates(P (ci, cj)) 9: for each candidate ∈ candidateComponents do 10: newarchitecture← insertBetween(architecture, P (ci, cj), candidate) 11: newarchitectures← newarchitectures ∪ newarchitecture 12: candidateComponents← proposeInsertAsideCandidates(P (ci, cj)) 13: for each candidate ∈ candidateComponents do 14: newarchitecture← insertAside(architecture, P (ci, cj), candidate) 15: newarchitectures← newarchitectures ∪ newarchitecture 16: newarchitectures← filter(newarchitectures) 17: arcQueue← arcQueue ∪ newarchitectures 18: architecturesSet← architecturesSet ∪ newarchitectures 19: return architecturesSet
5 META-LEARNING FOR ARCHITECTURE RANKING
Our goal is to determine whether by analyzing the topology of DNN architecture as well as the transformations it undergoes in its early training iterations could be used to predict its performance. To this end we develop a novel machine learning-based approach that generates a set of features for each analyzed architecture. Once the features are generated, we use a ranking classifier to assign a score to each architecture. The classifier is trained on a large corpus of datasets (additional information is provided in Section 6.1).
We apply meta-learning (Vilalta & Drissi, 2002) to predict the performance of the DNN architectures. Meta-learning is a branch of machine learning in which an algorithm “learns how to learn” by extracting information on the learning process of another algorithm. The features extracted in this process are called meta-features. We generate three types of meta-features: dataset-based, topologybased and training-based. We hypothesize that these groups represent the elements that affect the performance of the DNN architecture - the data on which it is trained, the structure of the network and the changes in its weights, biases and activation functions during throughout the training process. We provide a full overview of the meta-features groups below and detailed information in Appendix A .
Dataset-based meta-features. As explained in Section 3, the datasets we use in the evaluation vary significantly in size and feature composition. These meta-features attempt to represent the multiple characteristics that may affect the performance of deep learning algorithms. We generate three types of meta-features:
1. General information: general statistics on the analyzed dataset: number of instances and classes, number and type of features and statistics on the correlations among various features.
2. Entropy-based measures: we partition the dataset’s features based on their type (discrete, numeric, etc.) and calculate statistics on the Information Gain (IG) of the features in each group.
3. Feature diversity: we partition the dataset into type-based groups and use the chi-squared and paired-t test to calculate the similarity of each pair in each group. We then generate meta-features using the tests’ statistic values.
Topology-based meta-features. Our generated architectures vary significantly in size, depth and width. Since these traits are likely to affect their performance, we use the meta-features of this group to quantify and model them. The meta-features can be partitioned into two groups:
1. Architecture composition: general statistics on the number and types of layers and functions that make up the architecture, statistics on layer composition as a function of depth etc.
2. Connectivity-based measures: for each layer in the architectures, we calculate various measures that are frequently used for graph-analysis. These measures include statistics on the number and ratio of incoming and outgoing edges (overall, per depth and per type) and node-centrality evaluation measures.
Training-based meta-features. The goal of these meta-features is to model the transformations undergone by the DNN during the course of its training. These meta-features consist of statistics on the weights, biases and activation function layers of the various components in the architecture. These meta-features can be partitioned into two groups:
1. Static evaluation: general statistics on the distribution of the various values across different depths and layer types. These features provide “snapshot” information on the training status of the architecture in multiple training steps.
2. Time series-based evaluation: We compare the values obtained in the various training iterations to those obtained earlier, calculate ratios and modeling the changes in values distribution over time.
A full description of all meta-features is provided in Appendix A.
6 EXPERIMENTS AND ANALYSIS
6.1 EXPERIMENTAL SETUP
We conduct our experiments on 13 supervised classification datasets in a tabular form. We selected these datasets since they represent common supervised-learning problems that are not often addressed by deep learning. In addition, their feature composition consists of both numeric and discrete features, a trait that makes them different from image and speech classification datasets. The datasets vary significantly in size, number and type of features (some contain only numerical features while others also contain discrete features) and class imbalance - traits we hypothesize will make learning across domains more challenging. All datasets are available on the OpenML repository and their properties are represented in Appendix B.
We use the following settings:
• For each dataset, we train the same set of 11,170 architectures, generated as described in Section 4. The maximal width (number of parallel layers) allowed for an architecture was set to 4, and we terminated the generation process upon reaching the predefined number of architectures. This deepest architectures generated by this approach have 8 activation layers and 14 components overall.
• For architectures training, all datasets were randomly partitioned into training, validation and test sets. 80% of the data points was used by the training and the remaining two sets assigned 10% each. The same split was used for all the architectures explored for each dataset. Original class ratios were maintained in all sets.
• All generated architectures were trained until convergence, with the time of termination determined by performance on the validation set.
• The training-based meta-features were only extracted for the following steps: 20, 40, 60, 80 and 100.
• We used a leave-one-out (LOO) cross-validation approach to train the ranking classifier: for each evaluated dataset di, we train the ranking classifier using the meta-features from dj ∈ D where i 6= j. This setting enables to test whether a meta-model trained on one dataset could be effectively applied on another.
• We randomly split the generated architectures into two groups. The first group, consisting of 70% of the architectures, is used for training. We use the remaining 30% to evaluate the performance of our approach on each dataset.
6.2 ANALYSIS
We begin by analyzing the accuracy distribution of the generated architectures across the datasets. We found that the distribution of accuracies varies significantly across the different datasets, with some datasets with ranges of [45%-90%] accuracy while others are in the range [89%-95%]. This difference has significant impact on one’s ability to apply architectures that are effective in one domain to another, as we confirm with the next experiment. An example of accuracies distributions is presented in figures 2 and 3 and plots for all datasets are presented in Appendix D.
Analyzing the performance differences of “parent–child” architectures. In order to determine whether our architecture generation method is effective, we analyzed the differences in accuracy between every architecture and its descendant. Our reason for performing this analysis is as follows: if making incremental additions to an existing architecture does not significantly change its performance, then we simply generate a large number of architecture which are nearly identical in performance.
The results of our analysis are presented in Table 1. For every ”parent–child“ pair we calculate the difference in accuracy on the test set. We then calculate the maximal and average changes in accuracy for each dataset. It is clear from the results that the changes in accuracy are significant, especially given the fact that changes are accumulated over time (deeper architectures are a result of multiple modifications).
Next we analyze the “parent–child” architectures with the maximal differences in order to determine whether the addition of particular component is most likely to induce large changes in accuracy. Our results, presented in Table 2, show that no one component type can be consistently attributed with inducing large changes.
Applying architectures across datasets. We attempt to determine whether it is possible to find architectures that perform well across multiple datasets. For each of the generated architectures, we calculate its performance-based ranking (i.e. position in a list ordered by the accuracy measure) on each of the datasets. Then, for each dataset we test the performance of the architecture with the best average ranking on the remaining datasets. We compare the performance of this architecture to that of the best evaluated architecture and to that of the best architecture found by our meta-learning model (described in the following section). The results, presented in Table 3, show significant differences in performance and lead us to conclude that in most cases DNN architectures do not perform well across multiple datasets.
Comparing the performance of DNN architectures to those of “conventional classifiers”. As a point of reference to “classical” machine learning approaches for classifying tabular data, in Table 3 we also presents the performance of the Random Forest algorithm (using the Weka Hall et al. (2009) implementation with the default parameters ). It is clear that neither Random Forest nor the DNN architectures consistently outperform the other. We intend to explore the factors that cause these differences in performance in future work.
Analyzing the performance of architectures with parallel layers. Next we explore whether architectures with parallel layers outperform similar non-parallel architectures. We analyze the 100 top-performing architectures of each dataset and calculate the percentage of architectures with parallel layers. The results, presented in Appendix C, show that this type of architecture consists on average of 62% of the top-performing architectures.
To determine whether the benefit of applying parallel layers is significant, we randomly choose one of our datasets (Ailerons) and identify the 100 top-performing architectures with parallel layers. From this set we randomly sample 10 architectures and compare the performance of each of them to those of all of their possible serial counterparts, created by iteratively removing all but one of the different parallel layers. Our results, presented in Table 4, show that architectures with parallel layers significantly outperform all of their serial counterparts.
Considering the same sample of parallel architectures, we analyze whether architectures performance can be improved by adding a batch normalization before, after or before and after each activation function. As shown by the results in Table 4, we did not find evidence that the addition of batch normalization improves the performance of architectures with parallel layers. We find this fact surprising and intend to explore this further in future work. An example of one of the parallel architectures is presented in Figure 4 in Appendix C.
Finally, we also analyze the component composition of the 100 top-performing architectures for each dataset. The most interesting conclusion found in this analysis is the fact that a relatively shallow architectures ( 4 fully-connected layers) seem to yield the best performance on average for all datasets. The full analysis of the architecture components is presented in Table 12 in Appendix C.
6.3 EVALUATING THE META-LEARNING APPRAOCH
We analyze the performance of our meta-learning model as a classifier to rank architectures based on their performance. For these experiments, we use the following settings:
• We define the 5% of the top-performing architectures of each dataset as “good” and label the remaining as “bad”. We use this setting due to the large variance in the performance of the DNN architectures on the different datasets (see Appendix D for full details). We also intend to experiment with other labeling methods in future work.
• We use the precision@X measure as the evaluation metric. We calculate it by ranking all architectures according with the confidence of the meta-classifier (i.e. the classifier trained on the meta-features) in them being “good”. Then, for the X top-ranking architectures we calculate the actual percentage of “good” architectures is X .
• We conduct a separate evaluation on the training-based meta-features and the dataset-based and topological meta-features. Since the training-based features are more computationally expensive to compute, we find it interesting to compare their performance to the other types of meta-features. In our experiments we denote the full set as MLfull, the trainingbased meta-features as MLtrain and the topological and dataset-based meta-features as MLdata+top.
• We use the Random Forest algorithm for the training of the meta-model.
The results of our evaluation are presented in Table 5. We show that we are able to identify multiple architectures in the top-ranking spots in a much higher ratio than their share of the population. It is also clear that the joint set of all meta-features outperforms both of the examined subsets.
Next we conduct a random sampling over architectures, and compare the performance of the sampled architectures to those obtained by ranking all architectures using the proposed meta-classifier. Our goal is to determine the probability that N randomly-sampled architectures will consist of at least one architecture that outperforms all the top M items ranked by the meta-classifier. We conduct the experiment as follows: for each dataset, we randomly sample a fixed number of architectures and identify the one with the highest performance among those sampled. We then check if this
architecture outperforms all those in the ranked list provided by the meta-learning model. We repeat this process 50,000 for each dataset and calculate the probability of this scenario. The results, presented in Table 6, show that our model outperforms random sampling for all datasets, often by a large margin. However, further experimentation is required to fully determine the effectiveness of the meta-learning approach.
Finally, we analyze the results in order to determine the effectiveness of the different meta-features used by our model. The analysis was carried out by running LASSO logistic regression and analyzing the weights assigned to the various meta-features. Based on this analysis we reach the following conclusions:
• The dataset-based meta-features had the smallest contribution to the performance. While it is somewhat surprising given the fact that DNNs perform very differently on dataset with different characteristics, we conclude that the model is focused on the way in the architecture is trained on the data (i.e. weights and activations).
• The topological meta-features that had the largest contribution were those modeling the depth of the network, the number of parallel layers and those counting the number of various components.
• The ranking model uses a large number of training-based meta-features and from all types described in Appendix A. However, the model includes only weight and activation-based meta-features among the training-based meta-features. The biases-based meta-features are almost never used.
7 CONCLUSIONS AND FUTURE WORK
In this study we have explored several aspects of applying DNNs to supervised classification problems. Our results demonstrate the difficulty in using DNN architectures that are effective in one domain to another. We also systematically compare the performance of architectures with parallel layers to those of similar linear architectures and demonstrate that the former outperforms the latter in many cases. We present a novel approach for predicting the performance of a DNN architecture by analyzing its topology and the changes in its weights, biases and activation function values during early phases of training. Our aim is that this work can lay the foundation for a better understanding of the DNN architectures space.
For future work we consider several directions. First, we plan to add additional components to the ones currently used in our automatic architecture generation method in order to enable further exploration. In addition, we will seek to enhance our approach adding automatic parameter tuning methods. This will enable us to efficiently explore multiple configurations and possibly identify higher-performing architectures. We are also considering the use of an exploration/exploitation
scheme along the lines presented in Li et al. (2016) to enable us to efficiently explore larger architecture spaces.
Another approach we plan to explore is to make the search over network architectures a fullydifferentiable problem, by encoding the problem only using mechanisms that enable such a search. As an example, let us imagine that we want to decide the best number of internal hidden layers to use in a multi-layer fully-connected neural net. For this, we could create multiple parallel stacks of layers with the same input at the bottom (e.g. the features for each data point) and the same kind of output at the end (e.g. probabilities over the possible classes) and then use a softmax to take a weighted sum of the outputs from each of the parallel stacks. By using a penalty on the negative entropy of this weighted sum, and increasing the penalty over time, the network should learn to produce the output using only one of the parallel stacks which we can then use at inference time. We can also train multiple models simultaneously using this method, and introduce additional penalties to ensure that the multiple models explore different architectures during training, to enable a more diverse search.
A THE META-FEATURES USED BY OUR APPROACH
We extract three types of meta-features for the training: dataset-based, topological and trainingbased. We now provide a complete description of the features used. Note that because calculate the meta-features in Table 9 three times: for the weights, biases and activation functions.
B FULL INFORMATION ON THE DATASETS USED IN THE EVALUATION
C ANALYSIS OF THE PERFORMANCE PARALLEL LAYERS
For each dataset, we analyze the 100 top-performing architectures and determine the percentage of architectures with parallel layers. The results, presented in Table 11, show that the percentage is significant. In table 12 we analyze the component composition of these architectures. The most interesting point (in our view) is that the number of fully-connected layers is about half of the possible maximum. We take this as an indication that the creation of very deep DNNs may not be required for tabular datasets of the type analyzed in this work. In Figure 4 we present an example of an architecture with parallel layers that was among the 100 top-performing on the Ailerons dataset.
Table 12: The average number of component types per architecture for 100 top-performing architectures of each dataset.
Dataset Concat FC BatchnormDropout ReLU Sigmoid Tanh Softmax Contraceptive 0.53 3.54 2.57 0.91 1.71 0.51 0.43 1 Seismic bumps 0.47 3.53 1.75 1.62 1.68 0.64 0.48 1 Page Blocks 0.67 3.46 1.59 0.35 1.22 0.56 0.6 1 Wind 0.57 3.65 2.47 0.3 1.67 0.34 0.52 1 Puma 32 0.55 3.56 1.84 0.94 1.6 0.46 0.57 1 CPU act 0.65 3.95 3.23 0.2 1.91 0.3 0.63 1 Delta elevators 0.64 3.4 1.81 0.49 1.34 0.38 0.59 1 Mammography 0.69 3.6 3 0.22 1.63 0.36 0.49 1 Ailerons 0.62 3.68 2.34 0.69 1.58 0.39 0.55 1 Bank marketing 0.52 3.41 2.23 1.19 1.82 0.53 0.35 1 German Credit 0.5 3.66 2.39 1.1 2 0.42 0.31 1 Space 0.63 3.8 3.32 0.28 2.29 0.35 0.35 1 Cardiography 0.55 3.61 3.29 0.1 2.34 0.26 0.35 1
Input
Batchnorm
Fully Connected
Output
ReLU
Fully Connected
Sigmoid
ReLU
Concat
Dropout
Fully Connected
Figure 4: An example of an architecture with parallel layers.
D ACCURACY DISTRIBUTION OF THE GENERATED ARCHITECTURES ACROSS THE EVALUATED DATASETS

This paper presents an intriguing study of how one can pose architecture search as a meta learning problem. By collecting features from networks trained on various datasets and training a “ranking classifier” (the actual details of the classifier do not seem to be described in detail) one can potentially infer what a good architecture for a new problem could be by simply running the ranker on the extracted features for a new problem setup.
One notable comment from the paper is that the authors fix some important hyper-parameters for all the networks. I am of the opinion that optimizing the learning rate (and its decay schedule) is actually quite important. I hypothesize that a lot of the conclusions of this paper may change quite a bit if the authors did an actual search over the rates instead. I suspect that instead of training 11k nets, one can train 2k nets with 5 learning rates each and get a much better result that is actually compelling.
I am not convinced that the protocol for generating the various architectures is doing a good job at creating a diversity of architecture (simply because of the max depth of 8 layers and 14 components overall). I suspect that most of these generated architectures are actually almost identical performance-wise and that it’s a waste to train so many of them on so many tasks. Unless the authors are already doing this, they should define a pruning mechanism that filters out nets that are too similar to already existing ones.
The batch normalization experiments in Table 2 seem odd and under-explained. It is also well-known that the optimal learning rates when using batch norm vs. not using batch norm can differ by an order of magnitude so given the fixed learning rate throughout all experiments, I take these results with some grain of salt.
I am not sure we got many insights into the kinds of architectures that ended up being at the top. Either visualizations, or trends (or both), would be great.
This work seems to conflate the study of parallel vs. serial architectures with the study of meta learning, which are somewhat distinct issues. I take issue with the table that compares parallel vs. serial performance (table 2) simply because the right way would be to filter the architectures by the same number of parameters / capacity.
Ultimately the conclusion seems to be that when applying deep nets in a new domain, it is difficult to come up with a good architecture in advance. In that sense, it is hard to see the paper as a constructive result, because it’s conclusions are that while the ranker may do a good job often-times, it’s not that reliable. Thus I am not convinced that this particular result will be of practical use to folks who are intending to use deep nets for a new domain.