DESIGNING NEURAL NETWORK ARCHITECTURES
At present, designing convolutional neural network (CNN) architectures requires both human expertise and labor. New architectures are handcrafted by careful experimentation or modified from a handful of existing networks. We introduce MetaQNN, a meta-modeling algorithm based on reinforcement learning to automatically generate high-performing CNN architectures for a given learning task. The learning agent is trained to sequentially choose CNN layers using Qlearning with an -greedy exploration strategy and experience replay. The agent explores a large but finite space of possible architectures and iteratively discovers designs with improved performance on the learning task. On image classification benchmarks, the agent-designed networks (consisting of only standard convolution, pooling, and fully-connected layers) beat existing networks designed with the same layer types and are competitive against the state-of-the-art methods that use more complex layer types. We also outperform existing meta-modeling approaches for network design on image classification tasks.
1 INTRODUCTION
Deep convolutional neural networks (CNNs) have seen great success in the past few years on a variety of machine learning problems (LeCun et al., 2015). A typical CNN architecture consists of several convolution, pooling, and fully connected layers. While constructing a CNN, a network designer has to make numerous design choices: the number of layers of each type, the ordering of layers, and the hyperparameters for each type of layer, e.g., the receptive field size, stride, and number of receptive fields for a convolution layer. The number of possible choices makes the design space of CNN architectures extremely large and hence, infeasible for an exhaustive manual search. While there has been some work (Pinto et al., 2009; Bergstra et al., 2013; Domhan et al., 2015) on automated or computer-aided neural network design, new CNN architectures or network design elements are still primarily developed by researchers using new theoretical insights or intuition gained from experimentation.
In this paper, we seek to automate the process of CNN architecture selection through a metamodeling procedure based on reinforcement learning. We construct a novel Q-learning agent whose goal is to discover CNN architectures that perform well on a given machine learning task with no human intervention. The learning agent is given the task of sequentially picking layers of a CNN model. By discretizing and limiting the layer parameters to choose from, the agent is left with a finite but large space of model architectures to search from. The agent learns through random exploration and slowly begins to exploit its findings to select higher performing models using the - greedy strategy (Mnih et al., 2015). The agent receives the validation accuracy on the given machine learning task as the reward for selecting an architecture. We expedite the learning process through repeated memory sampling using experience replay (Lin, 1993). We refer to this Q-learning based meta-modeling method as MetaQNN, which is summarized in Figure 1.1
We conduct experiments with a space of model architectures consisting of only standard convolution, pooling, and fully connected layers using three standard image classification datasets: CIFAR-10,
1For more information, model files, and code, please visit https://bowenbaker.github.io/metaqnn/
SVHN, and MNIST. The learning agent discovers CNN architectures that beat all existing networks designed only with the same layer types (e.g., Springenberg et al. (2014); Srivastava et al. (2015)). In addition, their performance is competitive against network designs that include complex layer types and training procedures (e.g., Clevert et al. (2015); Lee et al. (2016)). Finally, the MetaQNN selected models comfortably outperform previous automated network design methods (Stanley & Miikkulainen, 2002; Bergstra et al., 2013). The top network designs discovered by the agent on one dataset are also competitive when trained on other datasets, indicating that they are suited for transfer learning tasks. Moreover, we can generate not just one, but several varied, well-performing network designs, which can be ensembled to further boost the prediction performance.
2 RELATED WORK
Designing neural network architectures: Research on automating neural network design goes back to the 1980s when genetic algorithm-based approaches were proposed to find both architectures and weights (Schaffer et al., 1992). However, to the best of our knowledge, networks designed with genetic algorithms, such as those generated with the NEAT algorithm (Stanley & Miikkulainen, 2002), have been unable to match the performance of hand-crafted networks on standard benchmarks (Verbancsics & Harguess, 2013). Other biologically inspired ideas have also been explored; motivated by screening methods in genetics, Pinto et al. (2009) proposed a high-throughput network selection approach where they randomly sample thousands of architectures and choose promising ones for further training. In recent work, Saxena & Verbeek (2016) propose to sidestep the architecture selection process through densely connected networks of layers, which come closer to the performance of hand-crafted networks.
Bayesian optimization has also been used (Shahriari et al., 2016) for automatic selection of network architectures (Bergstra et al., 2013; Domhan et al., 2015) and hyperparameters (Snoek et al., 2012; Swersky et al., 2013). Notably, Bergstra et al. (2013) proposed a meta-modeling approach based on Tree of Parzen Estimators (TPE) (Bergstra et al., 2011) to choose both the type of layers and hyperparameters of feed-forward networks; however, they fail to match the performance of handcrafted networks.
Reinforcement Learning: Recently there has been much work at the intersection of reinforcement learning and deep learning. For instance, methods using CNNs to approximate theQ-learning utility function (Watkins, 1989) have been successful in game-playing agents (Mnih et al., 2015; Silver et al., 2016) and robotic control (Lillicrap et al., 2015; Levine et al., 2016). These methods rely on phases of exploration, where the agent tries to learn about its environment through sampling, and exploitation, where the agent uses what it learned about the environment to find better paths. In traditional reinforcement learning settings, over-exploration can lead to slow convergence times, yet over-exploitation can lead to convergence to local minima (Kaelbling et al., 1996). However, in the case of large or continuous state spaces, the -greedy strategy of learning has been empirically shown to converge (Vermorel & Mohri, 2005). Finally, when the state space is large or exploration is costly,
the experience replay technique (Lin, 1993) has proved useful in experimental settings (Adam et al., 2012; Mnih et al., 2015). We incorporate these techniques—Q-learning, the -greedy strategy and experience replay—in our algorithm design.
3 BACKGROUND
Our method relies on Q-learning, a type of reinforcement learning. We now summarize the theoretical formulation of Q-learning, as adopted to our problem. Consider the task of teaching an agent to find optimal paths as a Markov Decision Process (MDP) in a finite-horizon environment. Constraining the environment to be finite-horizon ensures that the agent will deterministically terminate in a finite number of time steps. In addition, we restrict the environment to have a discrete and finite state space S as well as action space U . For any state si ∈ S , there is a finite set of actions, U(si) ⊆ U , that the agent can choose from. In an environment with stochastic transitions, an agent in state si taking some action u ∈ U(si) will transition to state sj with probability ps′|s,u(sj |si, u), which may be unknown to the agent. At each time step t, the agent is given a reward rt, dependent on the transition from state s to s′ and action u. rt may also be stochastic according to a distribution pr|s′,s,u. The agent’s goal is to maximize the total expected reward over all possible trajectories, i.e., maxTi∈T RTi , where the total expected reward for a trajectory Ti is
RTi = ∑ (s,u,s′)∈Ti Er|s,u,s′ [r|s, u, s ′]. (1)
Though we limit the agent to a finite state and action space, there are still a combinatorially large number of trajectories, which motivates the use of reinforcement learning. We define the maximization problem recursively in terms of subproblems as follows. For any state si ∈ S and subsequent action u ∈ U(si), we define the maximum total expected reward to be Q∗(si, u). Q∗(·) is known as the action-value function and individual Q∗(si, u) are know as Q-values. The recursive maximization equation, which is known as Bellman’s Equation, can be written as
Q∗(si, u) = Esj |si,u [ Er|si,u,sj [r|si, u, sj ] + γmaxu′∈U(sj)Q∗(sj , u′) ] . (2)
In many cases, it is impossible to analytically solve Bellman’s Equation (Bertsekas, 2015), but it can be formulated as an iterative update
Qt+1(si, u) = (1− α)Qt(si, u) + α [ rt + γmaxu′∈U(sj)Qt(sj , u ′) ] . (3)
Equation 3 is the simplest form of Q-learning proposed by Watkins (1989). For well formulated problems, limt→∞Qt(s, u) = Q∗(s, u), as long as each transition is sampled infinitely many times (Bertsekas, 2015). The update equation has two parameters: (i) α is a Q-learning rate which determines the weight given to new information over old information, and (ii) γ is the discount factor which determines the weight given to short-term rewards over future rewards. The Q-learning algorithm is model-free, in that the learning agent can solve the task without ever explicitly constructing an estimate of environmental dynamics. In addition, Q-learning is off policy, meaning it can learn about optimal policies while exploring via a non-optimal behavioral distribution, i.e. the distribution by which the agent explores its environment.
We choose the behavior distribution using an -greedy strategy (Mnih et al., 2015). With this strategy, a random action is taken with probability and the greedy action, maxu∈U(si)Qt(si, u), is chosen with probability 1− . We anneal from 1→ 0 such that the agent begins in an exploration phase and slowly starts moving towards the exploitation phase. In addition, when the exploration cost is large (which is true for our problem setting), it is beneficial to use the experience replay technique for faster convergence (Lin, 1992). In experience replay, the learning agent is provided with a memory of its past explored paths and rewards. At a given interval, the agent samples from the memory and updates its Q-values via Equation 3.
4 DESIGNING NEURAL NETWORK ARCHITECTURES WITH Q-LEARNING
We consider the task of training a learning agent to sequentially choose neural network layers. Figure 2 shows feasible state and action spaces (a) and a potential trajectory the agent may take along with the CNN architecture defined by this trajectory (b). We model the layer selection process as a Markov Decision Process with the assumption that a well-performing layer in one network should
also perform well in another network. We make this assumption based on the hierarchical nature of the feature representations learned by neural networks with many hidden layers (LeCun et al., 2015). The agent sequentially selects layers via the -greedy strategy until it reaches a termination state. The CNN architecture defined by the agent’s path is trained on the chosen learning problem, and the agent is given a reward equal to the validation accuracy. The validation accuracy and architecture description are stored in a replay memory, and experiences are sampled periodically from the replay memory to update Q-values via Equation 3. The agent follows an schedule which determines its shift from exploration to exploitation.
Our method requires three main design choices: (i) reducing CNN layer definitions to simple state tuples, (ii) defining a set of actions the agent may take, i.e., the set of layers the agent may pick next given its current state, and (iii) balancing the size of the state-action space—and correspondingly, the model capacity—with the amount of exploration needed by the agent to converge. We now describe the design choices and the learning process in detail.
4.1 THE STATE SPACE
Each state is defined as a tuple of all relevant layer parameters. We allow five different types of layers: convolution (C), pooling (P), fully connected (FC), global average pooling (GAP), and softmax (SM), though the general method is not limited to this set. Table 1 shows the relevant parameters for each layer type and also the discretization we chose for each parameter. Each layer has a parameter layer depth (shown as Layer 1, 2, ... in Figure 2). Adding layer depth to the state space allows us to constrict the action space such that the state-action graph is directed and acyclic (DAG) and also allows us to specify a maximum number of layers the agent may select before terminating.
Each layer type also has a parameter called representation size (R-size). Convolutional nets progressively compress the representation of the original signal through pooling and convolution. The presence of these layers in our state space may lead the agent on a trajectory where the intermediate signal representation gets reduced to a size that is too small for further processing. For example, five 2× 2 pooling layers each with stride 2 will reduce an image of initial size 32× 32 to size 1× 1. At this stage, further pooling, or convolution with receptive field size greater than 1, would be meaningless and degenerate. To avoid such scenarios, we add the R-size parameter to the state tuple s, which allows us to restrict actions from states with R-size n to those that have a receptive field size less than or equal to n. To further constrict the state space, we chose to bin the representation sizes into three discrete buckets. However, binning adds uncertainty to the state transitions: depending on the true underlying representation size, a pooling layer may or may not change the R-size bin. As a result, the action of pooling can lead to two different states, which we model as stochasticity in state transitions. Please see Figure A1 in appendix for an illustrated example.
4.2 THE ACTION SPACE
We restrict the agent from taking certain actions to both limit the state-action space and make learning tractable. First, we allow the agent to terminate a path at any point, i.e. it may choose a termination state from any non-termination state. In addition, we only allow transitions for a state with layer depth i to a state with layer depth i + 1, which ensures that there are no loops in the graph. This constraint ensures that the state-action graph is always a DAG. Any state at the maximum layer depth, as prescribed in Table 1, may only transition to a termination layer.
Next, we limit the number of fully connected (FC) layers to be at maximum two, because a large number of FC layers can lead to too may learnable parameters. The agent at a state with type FC may transition to another state with type FC if and only if the number of consecutive FC states is less than the maximum allowed. Furthermore, a state s of type FC with number of neurons d may only transition to either a termination state or a state s′ of type FC with number of neurons d′ ≤ d. An agent at a state of type convolution (C) may transition to a state with any other layer type. An agent at a state with layer type pooling (P) may transition to a state with any other layer type other than another P state because consecutive pooling layers are equivalent to a single, larger pooling layer which could lie outside of our chosen state space. Furthermore, only states with representation size in bins (8, 4] and (4, 1] may transition to an FC layer, which ensures that the number of weights does not become unreasonably huge. Note that a majority of these constraints are in place to enable faster convergence on our limited hardware (see Section 5) and not a limitation of the method in itself.
4.3 Q-LEARNING TRAINING PROCEDURE
For the iterativeQ-learning updates (Equation 3), we set theQ-learning rate (α) to 0.01. In addition, we set the discount factor (γ) to 1 to not over-prioritize short-term rewards. We decrease from 1.0 to 0.1 in steps, where the step-size is defined by the number of unique models trained (Table 2). At = 1.0, the agent samples CNN architecture with a random walk along a uniformly weighted Markov chain. Every topology sampled by the agent is trained using the procedure described in Section 5, and the prediction performance of this network topology on the validation set is recorded. We train a larger number of models at = 1.0 as compared to other values of to ensure that the agent has adequate time to explore before it begins to exploit. We stop the agent at = 0.1 (and not at = 0) to obtain a stochastic final policy, which generates perturbations of the global minimum.2 Ideally, we want to identify several well-performing model topologies, which can then be ensembled to improve prediction performance.
During the entire training process (starting at = 1.0), we maintain a replay dictionary which stores (i) the network topology and (ii) prediction performance on a validation set, for all of the sampled
2 = 0 indicates a completely deterministic policy. Because we would like to generate several good models for ensembling and analysis, we stop at = 0.1, which represents a stochastic final policy.
models. If a model that has already been trained is re-sampled, it is not re-trained, but instead the previously found validation accuracy is presented to the agent. After each model is sampled and trained, the agent randomly samples 100 models from the replay dictionary and applies the Q-value update defined in Equation 3 for all transitions in each sampled sequence. The Q-value update is applied to the transitions in temporally reversed order, which has been shown to speed up Q-values convergence (Lin, 1993).
5 EXPERIMENT DETAILS
During the model exploration phase, we trained each network topology with a quick and aggressive training scheme. For each experiment, we created a validation set by randomly taking 5,000 samples from the training set such that the resulting class distributions were unchanged. For every network, a dropout layer was added after every two layers. The ith dropout layer, out of a total n dropout layers, had a dropout probability of i2n . Each model was trained for a total of 20 epochs with the Adam optimizer (Kingma & Ba, 2014) with β1 = 0.9, β2 = 0.999, ε = 10−8. The batch size was set to 128, and the initial learning rate was set to 0.001. If the model failed to perform better than a random predictor after the first epoch, we reduced the learning rate by a factor of 0.4 and restarted training, for a maximum of 5 restarts. For models that started learning (i.e., performed better than a random predictor), we reduced the learning rate by a factor of 0.2 every 5 epochs. All weights were initialized with Xavier initialization (Glorot & Bengio, 2010). Our experiments using Caffe (Jia et al., 2014) took 8-10 days to complete for each dataset with a hardware setup consisting of 10 NVIDIA GPUs.
After the agent completed the schedule (Table 2), we selected the top ten models that were found over the course of exploration. These models were then finetuned using a much longer training schedule, and only the top five were used for ensembling. We now provide details of the datasets and the finetuning process.
The Street View House Numbers (SVHN) dataset has 10 classes with a total of 73,257 samples in the original training set, 26,032 samples in the test set, and 531,131 additional samples in the extended training set. During the exploration phase, we only trained with the original training set, using 5,000 random samples as validation. We finetuned the top ten models with the original plus extended training set, by creating preprocessed training and validation sets as described by Lee et al. (2016). Our final learning rate schedule after tuning on validation set was 0.025 for 5 epochs, 0.0125 for 5 epochs, 0.0001 for 20 epochs, and 0.00001 for 10 epochs.
CIFAR-10, the 10 class tiny image dataset, has 50,000 training samples and 10,000 testing samples. During the exploration phase, we took 5,000 random samples from the training set for validation. The maximum layer depth was increased to 18. After the experiment completed, we used the same validation set to tune hyperparameters, resulting in a final training scheme which we ran on the entire training set. In the final training scheme, we set a learning rate of 0.025 for 40 epochs, 0.0125 for 40 epochs, 0.0001 for 160 epochs, and 0.00001 for 60 epochs, with all other parameters unchanged. During this phase, we preprocess using global contrast normalization and use moderate data augmentation, which consists of random mirroring and random translation by up to 5 pixels.
MNIST, the 10 class handwritten digits dataset, has 60,000 training samples and 10,000 testing samples. We preprocessed each image with global mean subtraction. In the final training scheme, we trained each model for 40 epochs and decreased learning rate every 5 epochs by a factor of 0.2. For further tuning details please see Appendix C.
6 RESULTS
Model Selection Analysis: From Q-learning principles, we expect the learning agent to improve in its ability to pick network topologies as reduces and the agent enters the exploitation phase. In
Figure 3, we plot the rolling mean of prediction accuracy over 100 models and the mean accuracy of models sampled at different values, for the CIFAR-10 and SVHN experiments. The plots show that, while the prediction accuracy remains flat during the exploration phase ( = 1) as expected, the agent consistently improves in its ability to pick better-performing models as reduces from 1 to 0.1. For example, the mean accuracy of models in the SVHN experiment increases from 52.25% at = 1 to 88.02% at = 0.1. Furthermore, we demonstrate the stability of the Q-learning procedure with 10 independent runs on a subset of the SVHN dataset in Section D.1 of the Appendix. Additional analysis of Q-learning results can be found in Section D.2.
The top models selected by the Q-learning agent vary in the number of parameters but all demonstrate high performance (see Appendix Tables 1-3). For example, the number of parameters for the top five CIFAR-10 models range from 11.26 million to 1.10 million, with only a 2.32% decrease in test error. We find design motifs common to the top hand-crafted network architectures as well. For example, the agent often chooses a layer of type C(N, 1, 1) as the first layer in the network. These layers generate N learnable linear transformations of the input data, which is similar in spirit to preprocessing of input data from RGB to a different color spaces such as YUV, as found in prior work (Sermanet et al., 2012; 2013).
Prediction Performance: We compare the prediction performance of the MetaQNN networks discovered by theQ-learning agent with state-of-the-art methods on three datasets. We report the accuracy of our best model, along with an ensemble of top five models. First, we compare MetaQNN with six existing architectures that are designed with standard convolution, pooling, and fully-connected layers alone, similar to our designs. As seen in Table 3, our top model alone, as well as the committee ensemble of five models, outperforms all similar models. Next, we compare our results with six top networks overall, which contain complex layer types and design ideas, including generalized pooling functions, residual connections, and recurrent modules. Our results are competitive with these methods as well (Table 4). Finally, our method outperforms existing automated network de-
sign methods. MetaQNN obtains an error of 6.92% as compared to 21.2% reported by Bergstra et al. (2011) on CIFAR-10; and it obtains an error of 0.32% as compared to 7.9% reported by Verbancsics & Harguess (2013) on MNIST.
The difference in validation error between the top 10 models for MNIST was very small, so we also created an ensemble with all 10 models. This ensemble achieved a test error of 0.28%—which beats the current state-of-the-art on MNIST without data augmentation.
The best CIFAR-10 model performs 1-2% better than the four next best models, which is why the ensemble accuracy is lower than the best model’s accuracy. We posit that the CIFAR-10 MetaQNN did not have adequate exploration time given the larger state space compared to that of the SVHN experiment, causing it to not find more models with performance similar to the best model. Furthermore, the coarse training scheme could have been not as well suited for CIFAR-10 as it was for SVHN, causing some models to under perform.
Transfer Learning Ability: Network designs such as VGGnet (Simonyan & Zisserman, 2014) can be adopted to solve a variety of computer vision problems. To check if the MetaQNN networks provide similar transfer learning ability, we use the best MetaQNN model on the CIFAR-10 dataset for training other computer vision tasks. The model performs well (Table 5) both when training from random initializations, and finetuning from existing weights.
7 CONCLUDING REMARKS
Neural networks are being used in an increasingly wide variety of domains, which calls for scalable solutions to produce problem-specific model architectures. We take a step towards this goal and show that a meta-modeling approach using reinforcement learning is able to generate tailored CNN designs for different image classification tasks. Our MetaQNN networks outperform previous metamodeling methods as well as hand-crafted networks which use the same types of layers.
While we report results for image classification problems, our method could be applied to different problem settings, including supervised (e.g., classification, regression) and unsupervised (e.g., autoencoders). The MetaQNN method could also aid constraint-based network design, by optimizing parameters such as size, speed, and accuracy. For instance, one could add a threshold in the state-action space barring the agent from creating models larger than the desired limit. In addition,
∗Results in this column obtained with the top MetaQNN architecture for CIFAR-10, trained from random initialization with CIFAR-100 data.
one could modify the reward function to penalize large models for constraining memory or penalize slow forward passes to incentivize quick inference.
There are several future avenues for research in reinforcement learning-driven network design as well. In our current implementation, we use the same set of hyperparameters to train all network topologies during the Q-learning phase and further finetune the hyperparameters for top models selected by the MetaQNN agent. However, our approach could be combined with hyperparameter optimization methods to further automate the network design process. Moreover, we constrict the state-action space using coarse, discrete bins to accelerate convergence. It would be possible to move to larger state-action spaces using methods for Q-function approximation (Bertsekas, 2015; Mnih et al., 2015).
ACKNOWLEDGMENTS
We thank Peter Downs for creating the project website and contributing to illustrations. We acknowledge Center for Bits and Atoms at MIT for their help with computing resources. Finally, we thank members of Camera Culture group at MIT Media Lab for their help and support.
A ALGORITHM
We first describe the main components of the MetaQNN algorithm. Algorithm 1 shows the main loop, where the parameter M would determine how many models to run for a given and the parameter K would determine how many times to sample the replay database to update Q-values on each iteration. The function TRAIN refers to training the specified network and returns a validation accuracy. Algorithm 2 details the method for sampling a new network using the -greedy strategy, where we assume we have a function TRANSITION that returns the next state given a state and action. Finally, Algorithm 3 implements theQ-value update detailed in Equation 3, with discounting factor set to 1, for an entire state sequence in temporally reversed order.
Algorithm 1 Q-learning For CNN Topologies Initialize:
replay memory← [ ] Q← {(s, u) ∀s ∈ S, u ∈ U(s) : 0.5}
for episode = 1 to M do S, U ← SAMPLE NEW NETWORK( , Q) accuracy← TRAIN(S) replay memory.append((S, U, accuracy)) for memory = 1 to K do
SSAMPLE , USAMPLE , accuracySAMPLE ← Uniform{replay memory} Q← UPDATE Q VALUES(Q, SSAMPLE , USAMPLE , accuracySAMPLE)
end for end for
Algorithm 2 SAMPLE NEW NETWORK( , Q) Initialize:
state sequence S = [sSTART] action sequence U = [ ]
while U [−1] 6= terminate do α ∼ Uniform[0, 1) if α > then
u = argmaxu∈U(S[−1])Q[(S[−1], u)] s′ = TRANSITION(S[−1], u)
else u ∼ Uniform{U(S[−1])} s′ = TRANSITION(S[−1], u) end if U.append(u) if u != terminate then
S.append(s′) end if
end while return S, U
Algorithm 3 UPDATE Q VALUES(Q, S, U , accuracy) Q[S[−1], U [−1]] = (1− α)Q[S[−1], U [−1]] + α · accuracy for i = length(S)− 2 to 0 do
Q[S[i], U [i]] = (1− α)Q[S[i], U [i]] + αmaxu∈U(S[i+1])Q[S[i+ 1], u] end for return Q
B REPRESENTATION SIZE BINNING
As mentioned in Section 4.1 of the main text, we introduce a parameter called representation size to prohibit the agent from taking actions that can reduce the intermediate signal representation to a size that is too small for further processing. However, this process leads to uncertainties in state transitions, as illustrated in Figure A1, which is handled by the standard Q-learning formulation.
P(2,2)
R-size: 18 R-size bin: 1
R-size: 9 R-size bin: 1
(a)
P(2,2)
R-size: 7 R-size bin: 2
R-size: 14 R-size bin: 1
(b)
States Actions
p 1 2 p
R-size bin: 1
R-size bin: 1 R-size bin: 2
P(2,2)
(c)
Figure A1: Representation size binning: In this figure, we show three example state transitions. The true representation size (R-size) parameter is included in the figure to show the true underlying state. Assuming there are two R-size bins, R-size Bin1: [8,∞) and R-size Bin2: (0, 7], Figure A1a shows the case where the initial state is in R-size Bin1 and true representation size is 18. After the agent chooses to pool with a 2×2 filter with stride 2, the true representation size reduces to 9 but the R-size bin does not change. In Figure A1b, the same 2 × 2 pooling layer with stride 2 reduces the actual representation size of 14 to 7, but the bin changes to R-size Bin2. Therefore, in figures A1a and A1b, the agent ends up in different final states, despite originating in the same initial state and choosing the same action. Figure A1c shows that in our state-action space, when the agent takes an action that reduces the representation size, it will have uncertainty in which state it will transition to.
C MNIST EXPERIMENT
We noticed that the final MNIST models were prone to overfitting, so we increased dropout and did a small grid search for the weight regularization parameter. For both tuning and final training, we warmed the model with the learned weights from after the first epoch of initial training. The final models and solvers can be found on our project website https://bowenbaker.github.io/metaqnn/ . Figure A2 shows the Q-Learning performance for the MNIST experiment.
D FURTHER ANALYSIS OF Q-LEARNING
Figure 3 of the main text and Figure A2 show that as the agent begins to exploit, it improves in architecture selection. It is also informative to look at the distribution of models chosen at each . Figure A4 gives further insight into the performance achieved at each for both experiments.
D.1 Q-LEARNING STABILITY
Because the Q-learning agent explores via a random or semi-random distribution, it is natural to ask whether the agent can consistently improve architecture performance. While the success of the three independent experiments described in the main text allude to stability, here we present further evidence. We conduct 10 independent runs of the Q-learning procedure on 10% of the SVHN dataset (which corresponds to ∼7,000 training examples). We use a smaller dataset to reduce the computation time of each independent run to 10GPU-days, as opposed to the 100GPU-days it would take on the full dataset. As can be seen in Figure A3, the Q-learning procedure with the exploration schedule detailed in Table 2 is fairly stable. The standard deviation at = 1 is notably smaller than at other stages, which we attribute to the large difference in number of samples at each stage.
0 500 1000 1500 2000 2500 3000 3500 Iterations
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
A cc
u ra
cy
Epsilon = 1.0 .9.8.7 .6 .5 .4 .3 .2 .1
MNIST Q-Learning Performance
Average Accuracy Per Epsilon Rolling Mean Model Accuracy
Figure A2: MNIST Q-Learning Performance. The blue line shows a rolling mean of model accuracy versus iteration, where in each iteration of the algorithm the agent is sampling a model. Each bar (in light blue) marks the average accuracy over all models that were sampled during the exploration phase with the labeled . As decreases, the average accuracy goes up, demonstrating that the agent learns to select better-performing CNN architectures.
0.10.20.30.40.50.60.70.80.91.0 Epsilon
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
M ea
n Ac
cu ra
cy
Q-Learning Stability (Across 10 Runs)
(a)
0.10.20.30.40.50.60.70.80.91.0 Epsilon
0.45
0.50
0.55
0.60
0.65
0.70
0.75 0.80 M ea n Ac cu ra cy
Q-Learning Individual Runs
(b)
Figure A3: Figure A3a shows the mean model accuracy and standard deviation at each over 10 independent runs of the Q-learning procedure on 10% of the SVHN dataset. Figure A3b shows the mean model accuracy at each for each independent experiment. Despite some variance due to a randomized exploration strategy, each independent run successfully improves architecture performance.
Furthermore, the best model found during each run had remarkably similar performance with a mean accuracy of 88.25% and standard deviation of 0.58%, which shows that each run successfully found at least one very high performing model. Note that we did not use an extended training schedule to improve performance in this experiment.
D.2 Q-VALUE ANALYSIS
We now analyze the actualQ-values generated by the agent during the training process. The learning agent iteratively updates the Q-values of each path during the -greedy exploration. Each Q-value is initialized at 0.5. After the -schedule is complete, we can analyze the final Q-value associated with each path to gain insights into the layer selection process. In the left column of Figure A5, we plot the average Q-value for each layer type at different layer depths (for both SVHN and CIFAR10) datasets. Roughly speaking, a higher Q-value associated with a layer type indicates a higher probability that the agent will pick that layer type. In Figure A5, we observe that, while the average Q-value is higher for convolution and pooling layers at lower layer depths, the Q-values for fullyconnected and termination layers (softmax and global average pooling) increase as we go deeper into the network. This observation matches with traditional network designs.
We can also plot the averageQ-values associated with different layer parameters for further analysis. In the right column of Figure A5, we plot the averageQ-values for convolution layers with receptive
field sizes 1, 3, and 5 at different layer depths. The plots show that layers with receptive field size of 5 have a higher Q-value as compared to sizes 1 and 3 as we go deeper into the networks. This indicates that it might be beneficial to use larger receptive field sizes in deeper networks.
In summary, the Q-learning method enables us to perform analysis on the relative benefits of different design parameters of our state space, and possibly gain insights for new CNN designs.
E TOP TOPOLOGIES SELECTED BY ALGORITHM
In Tables A1 through A3, we present the top five model architectures selected with Q-learning for each dataset, along with their prediction error reported on the test set, and their total number of parameters. To download the Caffe solver and prototext files, please visit https://bowenbaker.github.io/metaqnn/ .
Model Architecture Test Error (%) # Params (106) [C(512,5,1), C(256,3,1), C(256,5,1), C(256,3,1), P(5,3), C(512,3,1), C(512,5,1), P(2,2), SM(10)] 6.92 11.18 [C(128,1,1), C(512,3,1), C(64,1,1), C(128,3,1), P(2,2), C(256,3,1), P(2,2), C(512,3,1), P(3,2), SM(10)] 8.78 2.17 [C(128,3,1), C(128,1,1), C(512,5,1), P(2,2), C(128,3,1), P(2,2), C(64,3,1), C(64,5,1), SM(10)] 8.88 2.42 [C(256,3,1), C(256,3,1), P(5,3), C(256,1,1), C(128,3,1), P(2,2), C(128,3,1), SM(10)] 9.24 1.10 [C(128,5,1), C(512,3,1), P(2,2), C(128,1,1), C(128,5,1), P(3,2), C(512,3,1), SM(10)] 11.63 1.66
Table A1: Top 5 model architectures: CIFAR-10.
Model Architecture Test Error (%) # Params (106) [C(128,3,1), P(2,2), C(64,5,1), C(512,5,1), C(256,3,1), C(512,3,1), P(2,2), C(512,3,1), C(256,5,1), C(256,3,1), C(128,5,1), C(64,3,1), SM(10)] 2.24 9.81 [C(128,1,1), C(256,5,1), C(128,5,1), P(2,2), C(256,5,1), C(256,1,1), C(256,3,1), C(256,3,1), C(256,5,1), C(512,5,1), C(256,3,1), C(128,3,1), SM(10)] 2.28 10.38 [C(128,5,1), C(128,3,1), C(64,5,1), P(5,3), C(128,3,1), C(512,5,1), C(256,5,1), C(128,5,1), C(128,5,1), C(128,3,1), SM(10)] 2.32 6.83 [C(128,1,1), C(256,5,1), C(128,5,1), C(256,3,1), C(256,5,1), P(2,2), C(128,1,1), C(512,3,1), C(256,5,1), P(2,2), C(64,5,1), C(64,1,1), SM(10)] 2.35 6.99 [C(128,1,1), C(256,5,1), C(128,5,1), C(256,5,1), C(256,5,1), C(256,1,1), P(3,2), C(128,1,1), C(256,5,1), C(512,5,1), C(256,3,1), C(128,3,1), SM(10)] 2.36 10.05
Table A2: Top 5 model architectures: SVHN. Note that we do not report the best accuracy on test set from the above models in Tables 3 and 4 from the main text. This is because the model that achieved 2.28% on the test set performed the best on the validation set.
Model Architecture Test Error (%) # Params (106) [C(64,1,1), C(256,3,1), P(2,2), C(512,3,1), C(256,1,1), P(5,3), C(256,3,1), C(512,3,1), FC(512), SM(10)] 0.35 5.59 [C(128,3,1), C(64,1,1), C(64,3,1), C(64,5,1), P(2,2), C(128,3,1), P(3,2), C(512,3,1), FC(512), FC(128), SM(10)] 0.38 7.43 [C(512,1,1), C(128,3,1), C(128,5,1), C(64,1,1), C(256,5,1), C(64,1,1), P(5,3), C(512,1,1), C(512,3,1), C(256,3,1), C(256,5,1), C(256,5,1), SM(10)] 0.40 8.28 [C(64,3,1), C(128,3,1), C(512,1,1), C(256,1,1), C(256,5,1), C(128,3,1), P(5,3), C(512,1,1), C(512,3,1), C(128,5,1), SM(10)] 0.41 6.27 [C(64,3,1), C(128,1,1), P(2,2), C(256,3,1), C(128,5,1), C(64,1,1), C(512,5,1), C(128,5,1), C(64,1,1), C(512,5,1), C(256,5,1), C(64,5,1), SM(10)] 0.43 8.10 [C(64,1,1), C(256,5,1), C(256,5,1), C(512,1,1), C(64,3,1), P(5,3), C(256,5,1), C(256,5,1), C(512,5,1), C(64,1,1), C(128,5,1), C(512,5,1), SM(10)] 0.44 9.67 [C(128,3,1), C(512,3,1), P(2,2), C(256,3,1), C(128,5,1), C(64,1,1), C(64,5,1), C(512,5,1), GAP(10), SM(10)] 0.44 3.52 [C(256,3,1), C(256,5,1), C(512,3,1), C(256,5,1), C(512,1,1), P(5,3), C(256,3,1), C(64,3,1), C(256,5,1), C(512,3,1), C(128,5,1), C(512,5,1), SM(10)] 0.46 12.42 [C(512,5,1), C(128,5,1), C(128,5,1), C(128,3,1), C(256,3,1), C(512,5,1), C(256,3,1), C(128,3,1), SM(10)] 0.55 7.25 [C(64,5,1), C(512,5,1), P(3,2), C(256,5,1), C(256,3,1), C(256,3,1), C(128,1,1), C(256,3,1), C(256,5,1), C(64,1,1), C(256,3,1), C(64,3,1), SM(10)] 0.56 7.55
Table A3: Top 10 model architectures: MNIST. We report the top 10 models for MNIST because we included all 10 in our final ensemble. Note that we do not report the best accuracy on test set from the above models in Tables 3 and 4 from the main text. This is because the model that achieved 0.44% on the test set performed the best on the validation set.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Validation Accuracy
0
10
20
30
40
50
60
% M
o d e ls
Model Accuracy Distribution (SVHN)
epsilon
0.1 0.2 0.3 0.4 0.5
0.6 0.7 0.8 0.9 1.0
(a)
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Validation Accuracy
0
10
20
30
40
50
60
% M
o d e ls
Model Accuracy Distribution (SVHN)
epsilon
0.1 1.0
(b)
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Validation Accuracy
0
5
10
15
20
% M
o d e ls
Model Accuracy Distribution (CIFAR-10)
epsilon
0.1 0.2 0.3 0.4 0.5
0.6 0.7 0.8 0.9 1.0
(c)
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Validation Accuracy
0
5
10
15 20 % M o d e ls
Model Accuracy Distribution (CIFAR-10)
epsilon
0.1 1.0
(d)
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Validation Accuracy
0
20
40
60
80
100
% M
od el
s
Model Accuracy Distribution (MNIST)
epsilon 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
(e)
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Validation Accuracy
0
20
40
60
80
100
% M
od el
s
Model Accuracy Distribution (MNIST)
epsilon 0.1 1.0
(f)
Figure A4: Accuracy Distribution versus : Figures A4a, A4c, and A4e show the accuracy distribution for each for the SVHN, CIFAR-10, and MNIST experiments, respectively. Figures A4b, A4d, and A4f show the accuracy distributions for the initial = 1 and the final = 0.1. One can see that the accuracy distribution becomes much more peaked in the high accuracy ranges at small for each experiment.
0 2 4 6 8 10 12 14 Layer Depth
0.0
0.2
0.4
0.6
0.8
1.0
A ve
ra ge
Q -V
al ue
Average Q-Value vs. Layer Depth (SVHN)
Convolution Fully Connected Pooling Global Average Pooling Softmax
(a)
0 2 4 6 8 10 12 Layer Depth
0.5
0.6
0.7
0.8
0.9
1.0
A ve
ra ge
Q -V
al ue
Average Q-Value vs. Layer Depth for Convolution Layers (SVHN)
Receptive Field Size 1 Receptive Field Size 3 Receptive Field Size 5
(b)
0 5 10 15 20 Layer Depth
0.0
0.2
0.4
0.6
0.8
1.0
A ve
ra ge
Q -V
al ue
Average Q-Value vs. Layer Depth (CIFAR10)
Convolution Fully Connected Pooling Global Average Pooling Softmax
(c)
0 2 4 6 8 10 12 14 16 18 Layer Depth
0.5
0.6
0.7
0.8
0.9 1.0 A ve ra ge Q -V al ue
Average Q-Value vs. Layer Depth for Convolution Layers (CIFAR10)
Receptive Field Size 1 Receptive Field Size 3 Receptive Field Size 5
(d)
0 2 4 6 8 10 12 14 Layer Depth
0.0
0.2
0.4
0.6
0.8
1.0
A ve
ra ge
Q -V
al ue
Average Q-Value vs. Layer Depth (MNIST)
Convolution Fully Connected Pooling Global Average Pooling Softmax
(e)
0 2 4 6 8 10 12 Layer Depth
0.5
0.6
0.7
0.8
0.9
1.0
A ve
ra ge
Q -V
al ue
Average Q-Value vs. Layer Depth for Convolution Layers (MNIST)
Receptive Field Size 1 Receptive Field Size 3 Receptive Field Size 5
(f)
Figure A5: Average Q-Value versus Layer Depth for different layer types are shown in the left column. Average Q-Value versus Layer Depth for different receptive field sizes of the convolution layer are shown in the right column.

This paper introduces a reinforcement learning framework for designing a neural network architecture. For each time-step, the agent picks a new layer type with corresponding layer parameters (e.g., #filters). In order to reduce the size of state-action space, they used a small set of design choices.
Strengths:
- A novel approach for automatic design of neural network architectures.
- Shows quite promising results on several datasets (MNIST, CIFAR-10).
Weakness:
- Limited architecture design choices due to many prior assumptions (e.g., a set of possible number of convolution filters, at most 2 fully-connected layers, maximum depth, hard-coded dropout, etc.)
- The method is demonstrated in tabular Q-learning setting, but it is unclear whether the proposed method would work in a large state-action space.
Overall, this is an interesting and novel approach for neural network architecture design, and it seems to be worth publication despite some weaknesses.


We propose the Gaussian attention model for content-based neural memory access. With the proposed attention model, a neural network has the additional degree of freedom to control the focus of its attention from a laser sharp attention to a broad attention. It is applicable whenever we can assume that the distance in the latent space reflects some notion of semantics. We use the proposed attention model as a scoring function for the embedding of a knowledge base into a continuous vector space and then train a model that performs question answering about the entities in the knowledge base. The proposed attention model can handle both the propagation of uncertainty when following a series of relations and also the conjunction of conditions in a natural way. On a dataset of soccer players who participated in the FIFA World Cup 2014, we demonstrate that our model can handle both path queries and conjunctive queries well.
We propose the Gaussian attention model for content-based neural memory access. With the proposed attention model, a neural network has the additional degree of freedom to control the focus of its attention from a laser sharp attention to a broad attention. It is applicable whenever we can assume that the distance in the latent space reflects some notion of semantics. We use the proposed attention model as a scoring function for the embedding of a knowledge base into a continuous vector space and then train a model that performs question answering about the entities in the knowledge base. The proposed attention model can handle both the propagation of uncertainty when following a series of relations and also the conjunction of conditions in a natural way. On a dataset of soccer players who participated in the FIFA World Cup 2014, we demonstrate that our model can handle both path queries and conjunctive queries well.
1 INTRODUCTION
There is a growing interest in incorporating external memory into neural networks. For example, memory networks (Weston et al., 2014; Sukhbaatar et al., 2015) are equipped with static memory slots that are content or location addressable. Neural Turing machines (Graves et al., 2014) implement memory slots that can be read and written as in Turing machines (Turing, 1938) but through differentiable attention mechanism.
Each memory slot in these models stores a vector corresponding to a continuous representation of the memory content. In order to recall a piece of information stored in memory, attention is typically employed. Attention mechanism introduced by Bahdanau et al. (2014) uses a network that outputs a discrete probability mass over memory items. A memory read can be implemented as a weighted sum of the memory vectors in which the weights are given by the attention network. Reading out a single item can be realized as a special case in which the output of the attention network is peaked at the desired item. The attention network may depend on the current context as well as the memory item itself. The attention model is called location-based and content-based, if it depends on the location in the memory and the stored memory vector, respectively.
Knowledge bases, such as WordNet and Freebase, can also be stored in memory either through an explicit knowledge base embedding (Bordes et al., 2011; Nickel et al., 2011; Socher et al., 2013) or through a feedforward network (Bordes et al., 2015).
When we embed entities from a knowledge base in a continuous vector space, if the capacity of the embedding model is appropriately controlled, we expect semantically similar entities to be close to each other, which will allow the model to generalize to unseen facts. However the notion of proximity may strongly depend on the type of a relation. For example, Benjamin Franklin was an engineer but also a politician. We would need different metrics to capture his proximity to other engineers and politicians of his time.
In this paper, we propose a new attention model for content-based addressing. Our model scores each item vitem in the memory by the (logarithm of) multivariate Gaussian likelihood as follows:
score(vitem) = log φ(vitem|µcontext,Σcontext)
= −1 2 (vitem − µcontext)Σ −1 context(vitem − µcontext) + const. (1)
where context denotes all the variables that the attention depends on. For example, “American engineers in the 18th century” or “American politicians in the 18th century” would be two contexts that include Benjamin Franklin but the two attentions would have very different shapes.
Compared to the (normalized) inner product used in previous work (Sukhbaatar et al., 2015; Graves et al., 2014) for content-based addressing, the Gaussian model has the additional control of the spread of the attention over items in the memory. As we show in Figure 1, we can view the conventional inner-product-based attention and the proposed Gaussian attention as addressing by an affine energy function and a quadratic energy function, respectively. By making the addressing mechanism more complex, we may represent many entities in a relatively low dimensional embedding space. Since knowledge bases are typically extremely sparse, it is more likely that we can afford to have a more complex attention model than a large embedding dimension.
We apply the proposed Gaussian attention model to question answering based on knowledge bases. At the high-level, the goal of the task is to learn the mapping from a question about objects in the knowledge base in natural language to a probability distribution over the entities. We use the scoring function (1) for both embedding the entities as vectors, and extracting the conditions mentioned in the question and taking a conjunction of them to score each candidate answer to the question.
The ability to compactly represent a set of objects makes the Gaussian attention model well suited for representing the uncertainty in a multiple-answer question (e.g., “who are the children of Abraham Lincoln?”). Moreover, traversal over the knowledge graph (see Guu et al., 2015) can be naturally handled by a series of Gaussian convolutions, which generalizes the addition of vectors. In fact, we model each relation as a Gaussian with mean and variance parameters. Thus a traversal on a relation corresponds to a translation in the mean and addition of the variances.
The proposed question answering model is able to handle not only the case where the answer to a question is associated with an atomic fact, which is called simple Q&A (Bordes et al., 2015), but also questions that require composition of relations (path queries in Guu et al. (2015)) and conjunction of queries. An example flow of how our model deals with a question “Who plays forward for Borussia Dortmund?” is shown in Figure 2 in Section 3.
This paper is structured as follows. In Section 2, we describe how the Gaussian scoring function (1) can be used to embed the entities in a knowledge base into a continuous vector space. We call our model TransGaussian because of its similarity to the TransE model proposed by Bordes et al. (2013). Then in Section 3, we describe our question answering model. In Section 4, we carry out experiments on WorldCup2014 dataset we collected. The dataset is relatively small but it allows us to evaluate not only simple questions but also path queries and conjunction of queries. The proposed TransGaussian embedding with the question answering model achieves significantly higher accuracy than the vanilla TransE embedding or TransE trained with compositional relations Guu et al. (2015) combined with the same question answering model.
2 KNOWLEDGE BASE EMBEDDING
In this section, we describe the proposed TransGaussian model based on the Gaussian attention model (1). While it is possible to train a network that computes the embedding in a single pass (Bordes et al., 2015) or over multiple passes (Li et al., 2015), it is more efficient to offload the embedding as a separate step for question answering based on a large static knowledge base.
2.1 THE TRANSGAUSSIAN MODEL
Let E be the set of entities and R be the set of relations. A knowledge base is a collection of triplets (s, r, o), where we call s ∈ E , r ∈ R, and o ∈ E , the subject, the relation, and the object of the triplet, respectively. Each triplet encodes a fact. For example, (Albert Einstein,has profession,theoretical physicist). All the triplets given in a knowledge base are assumed to be true. However generally speaking a triplet may be true or false. Thus knowledge base embedding aims at training a model that predict if a triplet is true or not given some parameterization of the entities and relations (Bordes et al., 2011; 2013; Nickel et al., 2011; Socher et al., 2013; Wang et al., 2014).
In this paper, we associate a vector vs ∈ Rd with each entity s ∈ E , and we associate each relation r ∈ R with two parameters, δr ∈ Rd and a positive definite symmetric matrix Σr ∈ Rd×d++ . Given subject s and relation r, we can compute the score of an object o to be in triplet (s, r, o) using the Gaussian attention model as (1) with
score(s, r, o) = log φ(vo|µcontext,Σcontext), (2) where µcontext = vs + δr, Σcontext = Σr. Note that if Σr is fixed to the identity matrix, we are modeling the relation of subject vs and object vo as a translation δr, which is equivalent to the TransE model (Bordes et al., 2013). We allow the covariance Σr to depend on the relation to handle one-to-many relations (e.g., profession has person relation) and capture the shape of the distribution of the set of objects that can be in the triplet. We call our model TransGaussian because of its similarity to TransE (Bordes et al., 2013).
Parameterization For computational efficiency, we will restrict the covariance matrix Σr to be diagonal in this paper. Furthermore, in order to ensure that Σr is strictly positive definite, we employ the exponential linear unit (ELU, Clevert et al., 2015) and parameterize Σr as follows:
Σr = diag
( ELU(mr,1)+1+
. . . ELU(mr,d)+1+ ) where mr,j (j = 1, . . . , d) are the unconstrained parameters that are optimized during training and is a small positive value that ensure the positivity of the variance during numerical computation. The ELU is defined as
ELU(x) = { x, x ≥ 0, exp (x)− 1, x < 0.
Ranking loss Suppose we have a set of triplets T = {(si, ri, oi)}Ni=1 from the knowledge base. Let N (s, r) be the set of incorrect objects to be in the triplet (s, r, ·). Our objective function uses the ranking loss to measure the margin between the scores of true answers and those of false answers and it can be written as follows:
min {ve:e∈E},
{δr,Mr,:r∈R̄}
1
N ∑ (s,r,o)∈T Et′∼N (s,r) [ [µ− score(s, r, o) + score(s, r, t′)]+ ]
+ λ ∑ e∈E ‖ve‖22 + ∑ r∈R̄ ( ‖δr‖22 + ‖M r‖2F ) , (3) where, N = |T |, µ is the margin parameter and M r denotes the diagonal matrix with mr,j , j = 1, . . . , d on the diagonal; the function [·]+ is defined as [x]+ = max(0, x). Here, we treat an inverse
relation as a separate relation and denote by R̄ = R∪R−1 the set of all the relations including both relations in R and their inverse relations; a relation r̃ is the inverse relation of r if (s, r̃, o) implies (o, r, s) and vice versa. Moreover, Et′∼N (s,r) denotes the expectation with respect to the uniform distribution over the set of incorrect objects, which we approximate with 10 random samples in the experiments. Finally, the last terms are `2 regularization terms for the embedding parameters.
2.2 COMPOSITIONAL RELATIONS
Guu et al. (2015) has recently shown that training TransE with compositional relations can make it competitive to more complex models, although TransE is much simpler compared to for example, neural tensor networks (NTN, Socher et al. (2013)) and TransH Wang et al. (2014). Here, a compositional relation is a relation that is composed as a series of relations in R, for example, grand father of can be composed as first applying the parent of relation and then the father of relation, which can be seen as a traversal over a path on the knowledge graph.
TransGaussian model can naturally handle and propagate the uncertainty over such a chain of relations by convolving the Gaussian distributions along the path. That is, the score of an entity o to be in the τ -step relation r1/r2/ · · · /rτ with subject s, which we denote by the triplet (s, r1/r2/ · · · /rτ , o), is given as
score(s, r1/r2/ · · · /rτ , o) = log φ(vo|µcontext,Σcontext), (4) with µcontext = vs + ∑τ t=1 δrt , Σcontext = ∑τ t=1 Σrt , where the covariance associated with each relation is parameterized in the same way as in the previous subsection. Training with compositional relations Let P = {( si, ri1/ri2/ · · · /rili , oi )}N ′ i=1
be a set of randomly sampled paths from the knowledge graph. Here relation rik in a path can be a relation in R or an inverse relation in R−1. With the scoring function (4), the generalized training objective for compositional relations can be written identically to (3) except for replacing T with T ∪ P and replacing N with N ′ = |T ∪ P|.
3 QUESTION ANSWERING
Given a set of question-answer pairs, in which the question is phrased in natural language and the answer is an entity in the knowledge base, our goal is to train a model that learns the mapping from the question to the correct entity. Our question answering model consists of three steps, entity recognition, relation composition, and conjunction. We first identify a list of entities mentioned in the question (which is assumed to be provided by an oracle in this paper). If the question is “Who plays Forward for Borussia Dortmund?” then the list would be [Forward, Borussia Dortmund]. The next step is to predict the path of relations on the knowledgegraph starting from each entity in the list extracted in the first step. In the above example, this will be (smooth versions of) /Forward/position played by/ and /Borussia Dortmund/has player/ predicted as series of Gaussian convolutions. In general, we can have multiple relations appearing in each path. Finally, we take a product of all the Gaussian attentions and renormalize it, which is equivalent to Bayes’ rule with independent observations (paths) and a noninformative prior.
3.1 ENTITY RECOGNITION
We assume that there is an oracle that provides a list containing all the entities mentioned in the question, because (1) a domain specific entity recognizer can be developed efficiently (Williams et al., 2015) and (2) generally entity recognition is a challenging task and it is beyond the scope of this paper to show whether there is any benefit in training our question answering model jointly with a entity recognizer. We assume that the number of extracted entities can be different for each question.
3.2 RELATION COMPOSITION
We train a long short-term memory (LSTM, Hochreiter & Schmidhuber, 1997) network that emits an output ht for each token in the input sequence. Then we compute the attention over the hidden
states for each recognized entity e as
pt,e = softmax (f(ve,ht)) (t = 1, . . . , T ),
where ve is the vector associated with the entity e. We use a two-layer perceptron for f in our experiments, which can be written as follows:
f(ve,ht) = u > f ReLU (W f,vve +W f,hht + b1) + b2,
whereW f,v∈RL×d,W f,h∈RL×H , b1 ∈ RL, uf ∈ RL, b2 ∈ R are parameters. Here ReLU(x)= max(0, x) is the rectified linear unit. Finally, softmax denotes softmax over the T tokens.
Next, we use the weights pt,e to compute the weighted sum over the hidden states ht as oe = ∑T
t=1 pt,eht. (5)
Then we compute the weights αr,e over all the relations as αr,e = ReLU ( w>r oe ) (∀r ∈ R ∪ R−1). Here the rectified linear unit is used to ensure the positivity of the weights. Note however that the weights should not be normalized, because we may want to use the same relation more than once in the same path. Making the weights positive also has the effect of making the attention sparse and interpretable because there is no cancellation.
For each extracted entity e, we view the extracted entity and the answer of the question to be the subject and the object in some triplet (e, p, o), respectively, where the path p is inferred from the question as the weights αr,e as we described above. Accordingly, the score for each candidate answer o can be expressed using (1) as:
scoree(vo) = log φ(vo|µe,α,KB,Σe,α,KB) (6) with µe,α,KB = ve + ∑ r∈R̄ αr,eδr, Σe,α,KB = ∑ r∈R̄ α 2 r,eΣr, where ve is the vector associated with entity e and R̄ = R∪R−1 denotes the set of relations including the inverse relations.
3.3 CONJUNCTION
Let E(q) be the set of entities recognized in the question q. The final step of our model is to take the conjunction of the Gaussian attentions derived in the previous step. This step is simply carried out by multiplying the Gaussian attentions as follows:
score(vo|E(q),Θ) = log ∏
e∈E(q)
φ(vo|µe,α,KB,Σe,α,KB)
= −1 2 ∑ e∈E(q) ( vo − µe,α,KB )> Σ−1e,α,KB(vo − µe,α,KB) + const., (7)
which is again a (logarithm of) Gaussian scoring function, where µe,α,KB and Σe,α,KB are the mean and the covariance of the Gaussian attention given in (6). Here Θ denotes all the parameters of the question-answering model.
3.4 TRAINING THE QUESTION ANSWERING MODEL
Suppose we have a knowledge base (E ,R, T ) and a trained TransGaussian model( {ve}e∈E , {(δr,Σr)}r∈R̄ ) , where R̄ is the set of all relations including the inverse relations. During training time, we assume the training set is a supervised question-answer pairs {(qi, E(qi), ai) : i = 1, 2, . . . ,m}. Here, qi is a question formulated in natural language, E(qi) ⊂ E is a set of knowledge base entities that appears in the question, and ai ∈ E is the answer to the question. For example, on a knowledge base of soccer players, a valid training sample could be
(“Who plays forward for Borussia Dortmund?”,[Forward,Borussia Dortmund], Marco Reus).
Note that the answer to a question is not necessarily unique and we allow ai to be any of the true answers in the knowledge base. During test time, our model is shown (qi, E(qi)) and the task is to find ai. We denote the set of answers to qi by A(qi).
To train our question-answering model, we minimize the objective function
1
m m∑ i=1 ( E t′∼N (qi) [ [µ− score(vai |E(qi),Θ) + score(vt′ |E(qi),Θ)]+ ] + ν ∑ e∈E(qi) ∑ r∈R̄ |αr,e| ) + λ‖Θ‖22
where Et′∼N (qi) is expectation with respect to a uniform distribution over of all incorrect answers to qi, which we approximate with 10 random samples. We assume that the number of relations implied in a question is small compared to the total number of relations in the knowledge base. Hence the coefficients αr,e computed for each question qi are regularized by their `1 norms.
4 EXPERIMENTS
As a demonstration of the proposed framework, we perform question and answering on a dataset of soccer players. In this work, we consider two types of questions. A path query is a question that contains only one named entity from the knowledge base and its answer can be found from the knowledge graph by walking down a path consisting of a few relations. A conjunctive query is a question that contains more than one entities and the answer is given as the conjunction of all path queries starting from each entity. Furthermore, we experimented on a knowledge base completion task with TransGaussian embeddings to test its capability of generalization to unseen fact. Since knowledge base completion is not the main focus of this work, we include the results in the Appendix.
4.1 WORLDCUP2014 DATASET
We build a knowledge base of football players that participated in FIFA World Cup 2014 1. The original dataset consists of players’ information such as nationality, positions on the field and ages etc. We picked a few attributes and constructed 1127 entities and 6 atomic relations. The entities include 736 players, 297 professional soccer clubs, 51 countries, 39 numbers and 4 positions. And the six atomic relations are
1The original dataset can be found at https://datahub.io/dataset/fifa-world-cup-2014-all-players.
plays in club: PLAYER→ CLUB, plays position: PLAYER→ POSITION, is aged: PLAYER→ NUMBER, wears number 2: PLAYER→ NUMBER, plays for country: PLAYER→ COUNTRY, is in country: CLUB→ COUNTRY,
where PLAYER, CLUB, NUMBER, etc, denote the type of entities that can appear as the left or right argument for each relation. Some relations share the same type as the right argument, e.g., plays for country and is in country.
Given the entities and relations, we transformed the dataset into a set of 3977 triplets. A list of sample triplets can be found in the Appendix. Based on these triplets, we created two sets of question answering tasks which we call path query and conjunctive query respectively. The answer of every question is always an entity in the knowledge base and a question can involve one or two triplets. The questions are generated as follows.
Path queries. Among the paths on the knowledge graph, there are some natural composition of relations, e.g., plays in country (PLAYER → COUNTRY) can be decomposed as the composition of plays in club (PLAYER→ CLUB) and is in country (CLUB→ COUNTRY). In addition to the atomic relations, we manually picked a few meaningful compositions of relations and formed query templates, which takes the form “find e ∈ E , such that (s, p, e) is true”, where s is the subject and p can be an atomic relation or a path of relations. To formulate a set of path-based question-answer pairs, we manually created one or more question templates for every query template (see Table 5) Then, for a particular instantiation of a query template with subject and object entities, we randomly select a question template to generate a question given the subject; the object entity becomes the answer of the question. See Table 6 for the list of composed relations, sample questions, and answers. Note that all atomic relations in this dataset are many-to-one while these composed relations can be one-to-many or many-to-many as well.
Conjunctive queries. To generate question-and-answer pairs of conjunctive queries, we first picked three pairs of relations and used them to create query templates of the form “Find e ∈ E , such that both (s1, r1, e) and (s2, r2, e) are true.” (see Table 5). For a pair of relations r1 and r2, we enumerated all pairs of entities s1, s2 that can be their subjects and formulated the corresponding query in natural language using question templates as in the same way as path queries. See Table 7 for a list of sample questions and answers.
As a result, we created 8003 question-and-answer pairs of path queries and 2208 pairs of conjunctive queries which are partitioned into train / validation / test subsets. We refer to Table 1 for more statistics about the dataset. Templates for generating the questions are list in Table 5.
4.2 EXPERIMENTAL SETUP
To perform question and answering under our proposed framework, we first train the TransGaussian model on WorldCup2014 dataset. In addition to the atomic triplets, we randomly sampled 50000 paths with length 1 or 2 from the knowledge graph and trained a TransGaussian model compositionally as described in Set 2.2. An inverse relation is treated as a separate relation. Following the naming convention from Guu et al. (2015), we denote this trained embedding by TransGaussian (COMP). We found that the learned embedding possess some interesting properties. Some dimensions of the embedding space dedicate to represent a particular relation. Players are clustered by their attributes when entities’ embeddings are projected to the corresponding lower dimensional subspaces. We elaborate and illustrate such properties in the Appendix.
Baseline methods We also trained a TransGaussian model only on the atomic triplets and denote such a model by TransGaussian (SINGLE). Since no inverse relation was involved when TransGaussian (SINGLE) was trained, to use this embedding in question answering tasks, we represent the inverse relations as follows: for each relation r with mean δr and variance Σr, we model its inverse r−1 as a Gaussian attention with mean −δr and variance equal to Σr. We also trained TransE models on WorldCup2014 dataset by using the code released by the authors of Guu et al. (2015). Likewise, we use TransE (SINGLE) to denote the model trained with atomic triplets only and use TransE (COMP) to denote the model trained with the union of triplets and paths. Note that TransE can be considered as a special case of TransGaussian where the variance matrix is the identity and hence, the scoring formula Eq. (7) is applicable to TransE as well.
Training configurations For all models, dimension of entity embeddings was set to 30. The hidden size of LSTM was set to 80. Word embeddings were trained jointly with the question answering model and dimension of word embedding was set to 40. We employed Adam (Kingma & Ba, 2014) as the optimizer. All parameters were tuned on the validation set. Under the same setting, we experimented with two cases: first, we trained models for path queries and conjunctive queries separately; Furthermore, we trained a single model that addresses both types queries. We present the results of the latter case in the next subsection while the results of the former are included in the Appendix.
Evaluation metrics During test time, our model receives a question in natural language and a list of knowledge base entities contained in the question. Then it predicts the mean and variance of a Gaussian attention formulated in Eq. (7) which is expected to capture the distribution of all positive answers. We rank all entities in the knowledge base by their scores under this Gaussian attention. Next, for each entity which is a correct answer, we check its rank relative to all incorrect answers and call this rank the filtered rank. For example, if a correct entity is ranked above all negative answers except for one, it has filtered rank two. We compute this rank for all true answers and report mean filtered rank and H@1 which is the percentage of true answers that have filtered rank 1.
4.3 EXPERIMENTAL RESULTS
We present the results of joint learning in Table 2. These results show that TransGaussian works better than TransE in general. In fact, TransGaussian (COMP) achieved the best performance in almost all aspects. Most notably, it achieved the highest H@1 rates on challenging questions such as “where is the club that edin dzeko plays for?” (#11, composition of two relations) and “who are the defenders on german national team?” (#14, conjunction of two queries).
The same table shows that TransGaussian benefits remarkably from compositional training. For example, compositional training improved TransGaussian’s H@1 rate by near 60% in queries on players from a given countries (#8) and queries on players who play a particular position (#9). It also boosted TransGaussian’s performance on all conjunctive quries (#13–#15) significantly.
To understand TransGaussian (COMP)’s weak performance on answering queries on the professional football club located in a given country (#10) and queries on professional football club that has players from a particular country (#12), we tested its capability of modeling the composed relation by feeding the correct relations and subjects during test time. It turns out that these two relations were not modeled well by TransGaussian (COMP) embedding, which limits its performance in question answering. (See Table 8 in the Appendix for quantitative evaluations.) The same limit was found in the other three embeddings as well.
Note that all the models compared in Table 2 uses the proposed Gaussian attention model because TransE is the special case of TransGaussian where the variance is fixed to one. Thus the main differences are whether the variance is learned and whether the embedding was trained compositionally. Finally, we refer to Table 9 and 10 in the Appendix for experimental results of models trained on path and conjunctive queries separately.
5 RELATED WORK
The work of Vilnis & McCallum (2014) is similar to our Gaussian attention model. They discuss many advantages of the Gaussian embedding; for example, it is arguably a better way of handling asymmetric relations and entailment. However the work was presented in the word2vec (Mikolov et al., 2013)-style word embedding setting and the Gaussian embedding was used to capture the diversity in the meaning of a word. Our Gaussian attention model extends their work to a more general setting in which any memory item can be addressed through a concept represented as a Gaussian distribution over the memory items.
Bordes et al. (2014; 2015) proposed a question-answering model that embeds both questions and their answers to a common continuous vector space. Their method in Bordes et al. (2015) can combine multiple knowledge bases and even generalize to a knowledge base that was not used during training. However their method is limited to the simple question answering setting in which the answer of each question associated with a triplet in the knowledge base. In contrast, our method can handle both composition of relations and conjunction of conditions, which are both naturally enabled by the proposed Gaussian attention model.
Neelakantan et al. (2015a) proposed a method that combines relations to deal with compositional relations for knowledge base completion. Their key technical contribution is to use recurrent neural networks (RNNs) to encode a chain of relations. When we restrict ourselves to path queries, question answering can be seen as a sequence transduction task (Graves, 2012; Sutskever et al., 2014) in which the input is text and the output is a series of relations. If we use RNNs as a decoder, our model would be able to handle non-commutative composition of relations, which the current weighted convolution cannot handle well. Another interesting connection to our work is that they take the maximum of the inner-product scores (see also Weston et al., 2013; Neelakantan et al., 2015b), which are computed along multiple paths connecting a pair of entities. Representing a set as a collection of vectors and taking the maximum over the inner-product scores is a natural way to represent a set of memory items. The Gaussian attention model we propose in this paper, however, has the advantage of differentiability and composability.
6 CONCLUSION
In this paper, we have proposed the Gaussian attention model which can be used in a variety of contexts where we can assume that the distance between the memory items in the latent space is compatible with some notion of semantics. We have shown that the proposed Gaussian scoring function can be used for knowledge base embedding achieving competitive accuracy. We have also shown that our embedding model can naturally propagate uncertainty when we compose relations together. Our embedding model also benefits from compositional training proposed by Guu et al. (2015). Furthermore, we have demonstrated the power of the Gaussian attention model in a challenging question answering problem which involves both composition of relations and conjunction of queries. Future work includes experiments on natural question answering datasets and end-to-end training including the entity extractor.
ACKNOWLEDGMENTS
The authors would like to thank Daniel Tarlow, Nate Kushman, and Kevin Gimpel for valuable discussions.
A WORDCUP2014 DATASET
B TRANSGAUSSIAN EMBEDDING OF WORLDCUP2014
We trained our TransGaussian model on triplets and paths from WorldCup2014 dataset and illustrated the embeddings in Fig 3 and 4. Recall that we modeled every relation as a Gaussian with diagonal covariance matrix. Fig 3 shows the learned variance parameters of different relations. Each row corresponds to the variances of one relation. Columns are permuted to reveal the block structure. From this figure, we can see that every relation has a small variance in two or more dimensions. This implies that the coordinates of the embedding space are partitioned into semantically coherent clusters each of which represent a particular attribute of a player (or a football club). To verify this further, we picked the two coordinates in which a relation (e.g. plays position) has the least variance and projected the embedding of all valid subjects and objects (e.g. players and positions) of the relation to this 2 dimensional subspace. See Fig. 4. The relation between the subjects and the objects are simply translation in the projection when the corresponding subspace is two dimensional (e.g., plays position relation in Fig. 4 (a)). The same is true for other relations that requires larger dimension but it is more challenging to visualize in two dimensions. For relations that have a large number of unique objects, we only plotted for the eight objects with the most subjects for clarity of illustration.
Furthermore, in order to elucidate whether we are limited by the capacity of the TransGaussian embedding or the ability to decode question expressed in natural language, we evaluated the test question-answer pairs using the TransGaussian embedding composed according to the ground-truth relations and entities. The results were evaluated with the same metrics as in Sec. 4.3. This estimation is conducted for TransE embeddings as well. See Table 8 for the results. Compared to Table 2, the accuracy of TransGaussian (COMP) is higher on the atomic relations and path queries but lower on conjunctive queries. This is natural because when the query is simple there is not much room for the question-answering network to improve upon just combining the relations according to the ground truth relations, whereas when the query is complex the network could combine the embedding in a more creative way to overcome its limitation. In fact, the two queries (#10 and #12) that TransGaussian (COMP) did not perform well in Table 2 pertain to a single relation is in country−1 (#10) and a composition of two relations plays for country−1 / plays in club (#12). The performance of the two queries were low even when the ground truth
relations were given, which indicates that the TransGaussian embedding rather than the questionanswering network is the limiting factor.
C KNOWLEDGE BASE COMPLETION
Knowledge base completion has been a common task for testing knowledge base models on their ability of generalizing to unseen facts. Here, we apply our TransGaussian model to a knowledge completion task and show that it has competitive performance.
We tested on the subset of WordNet released by Guu et al. (2015). The atomic triplets in this dataset was originally created by Socher et al. (2013) and Guu et al. (2015) added path queries that were randomly sampled from the knowledge graph. We build our TransGaussian model by training on these triplets and paths and tested our model on the same link prediction task as done by Socher et al. (2013); Guu et al. (2015).
As done by Guu et al. (2015), we trained TransGaussian (SINGLE) with atomic triplets only and trained TransGaussian (COMP) with the union of atomic triplets and paths. We did not incorporate
word embedding in this task and each entity is assigned its individual vector. Without getting parameters tuned too much, TransGaussian (COMP) obtained accuracy comparable to TransE (COMP). See Table 11.

The contribution of this paper can be summarized as:
1, A TransGaussian model (in a similar idea of TransE) which models the subject / object embeddings in a parameterization of Gaussian distribution.  The model can be naturally adapted to path queries like the formulation of (Guu et al, 2015).
2. Along with the entity / relation representations trained by TransGaussian, an LSTM + attention model is built on natural language questions, aiming at learning a distribution (not normalized though) over relations for question answering.
3. Experiments on a generated WorldCup2014 dataset, focusing on path queries and conjunctive queries.
Overall, I think the Gaussian parameterization exhibits some nice properties, and could be suitable to KB completion and question answering. However, some details and the main experimental results are not convincing enough to me.  The paper writing also needs to be improved. More comments below:
[Major comments]
- My main concern is that that evaluation results are NOT strong. Either knowledge base completion or KB-based question answering, there are many existing and competitive benchmarks (e.g., FB15k / WebQuestions). Experimenting with such a tiny WordCup2014 dataset is not convincing.  Moreover, the questions are just generated by a few templates, which is far from NL questions. I am not even not sure why we need to apply an LSTM in such scenario. The paper would be much stronger if you can demonstrate its effectiveness on the above benchmarks.
- Conjunctive queries:  the current model assumes that all the detected entities in the question could be aligned to one or more relations and we can take conjunctions in the end. This assumption might be not always correct, so it is more necessary to justify this on real QA datasets.
- The model is named as  “Gaussian attention” and I kind of think it is not very closely related to well-known attention mechanism, but more related to KB embedding literature.
[Minor comments]
- I find Figure 2 a bit confusing. The first row of orange blocks denote KB relations, and the second row of those denote every single word of the NL question. Maybe make it clearer?
- Besides “entity recognition”, usually we still need an “entity linker” component which links the text mention to the KB entity.