We present a scalable approach for semi-supervised learning on graph-structured data that is based on an efficient variant of convolutional neural networks which operate directly on graphs. We motivate the choice of our convolutional architecture via a localized first-order approximation of spectral graph convolutions. Our model scales linearly in the number of graph edges and learns hidden layer representations that encode both local graph structure and features of nodes. In a number of experiments on citation networks and on a knowledge graph dataset we demonstrate that our approach outperforms related methods by a significant margin.
We present a scalable approach for semi-supervised learning on graph-structured data that is based on an efficient variant of convolutional neural networks which operate directly on graphs. We motivate the choice of our convolutional architecture via a localized first-order approximation of spectral graph convolutions. Our model scales linearly in the number of graph edges and learns hidden layer representations that encode both local graph structure and features of nodes. In a number of experiments on citation networks and on a knowledge graph dataset we demonstrate that our approach outperforms related methods by a significant margin.
1 INTRODUCTION
We consider the problem of classifying nodes (such as documents) in a graph (such as a citation network), where labels are only available for a small subset of nodes. This problem can be framed as graph-based semi-supervised learning, where label information is smoothed over the graph via some form of explicit graph-based regularization (Zhu et al., 2003; Zhou et al., 2004; Belkin et al., 2006; Weston et al., 2012), e.g. by using a graph Laplacian regularization term in the loss function:
L = L0 + λLreg , with Lreg = ∑ i,j Aij‖f(Xi)− f(Xj)‖2 = f(X)>∆f(X) . (1)
Here, L0 denotes the supervised loss w.r.t. the labeled part of the graph, f(·) can be a neural networklike differentiable function, λ is a weighing factor and X is a matrix of node feature vectors Xi. ∆ = D − A denotes the unnormalized graph Laplacian of an undirected graph G = (V, E) with N nodes vi ∈ V , edges (vi, vj) ∈ E , an adjacency matrix A ∈ RN×N (binary or weighted) and a degree matrix Dii = ∑ j Aij . The formulation of Eq. 1 relies on the assumption that connected nodes in the graph are likely to share the same label. This assumption, however, might restrict modeling capacity, as graph edges need not necessarily encode node similarity, but could contain additional information.
In this work, we encode the graph structure directly using a neural network model f(X,A) and train on a supervised target L0 for all nodes with labels, thereby avoiding explicit graph-based regularization in the loss function. Conditioning f(·) on the adjacency matrix of the graph will allow the model to distribute gradient information from the supervised loss L0 and will enable it to learn representations of nodes both with and without labels.
Our contributions are two-fold. Firstly, we introduce a simple and well-behaved layer-wise propagation rule for neural network models which operate directly on graphs and show how it can be motivated from a first-order approximation of spectral graph convolutions (Hammond et al., 2011). Secondly, we demonstrate how this form of a graph-based neural network model can be used for fast and scalable semi-supervised classification of nodes in a graph. Experiments on a number of datasets demonstrate that our model compares favorably both in classification accuracy and efficiency (measured in wall-clock time) against state-of-the-art methods for semi-supervised learning.
2 FAST APPROXIMATE CONVOLUTIONS ON GRAPHS
In this section, we provide theoretical motivation for a specific graph-based neural network model f(X,A) that we will use in the rest of this paper. We consider a multi-layer Graph Convolutional Network (GCN) with the following layer-wise propagation rule:
H(l+1) = σ ( D̃− 1 2 ÃD̃− 1 2H(l)W (l) ) . (2)
Here, Ã = A + IN is the adjacency matrix of the undirected graph G with added self-connections. IN is the identity matrix, D̃ii = ∑ j Ãij and W
(l) is a layer-specific trainable weight matrix. σ(·) denotes an activation function, such as the ReLU(·) = max(0, ·). H(l) ∈ RN×D is the matrix of activations in the lth layer; H(0) = X . In the following, we show that the form of this propagation rule can be motivated1 via a first-order approximation of localized spectral filters on graphs (Hammond et al., 2011; Defferrard et al., 2016).
2.1 SPECTRAL GRAPH CONVOLUTIONS
We consider spectral convolutions on graphs defined as the multiplication of a signal x ∈ RN (a scalar for every node) with a filter gθ = diag(θ) parameterized by θ ∈ RN in the Fourier domain, i.e.:
gθ ? x = UgθU >x , (3)
where U is the matrix of eigenvectors of the normalized graph Laplacian L = IN −D− 1 2AD− 1 2 = UΛU>, with a diagonal matrix of its eigenvalues Λ and U>x being the graph Fourier transform of x. We can understand gθ as a function of the eigenvalues of L, i.e. gθ(Λ). Evaluating Eq. 3 is computationally expensive, as multiplication with the eigenvector matrix U isO(N2). Furthermore, computing the eigendecomposition of L in the first place might be prohibitively expensive for large graphs. To circumvent this problem, it was suggested in Hammond et al. (2011) that gθ(Λ) can be well-approximated by a truncated expansion in terms of Chebyshev polynomials Tk(x) up to K th order:
gθ′(Λ) ≈ K∑ k=0 θ′kTk(Λ̃) , (4)
with a rescaled Λ̃ = 2λmax Λ − IN . λmax denotes the largest eigenvalue of L. θ ′ ∈ RK is now a vector of Chebyshev coefficients. The Chebyshev polynomials are recursively defined as Tk(x) = 2xTk−1(x) − Tk−2(x), with T0(x) = 1 and T1(x) = x. The reader is referred to Hammond et al. (2011) for an in-depth discussion of this approximation.
Going back to our definition of a convolution of a signal x with a filter gθ′ , we now have:
gθ′ ? x ≈ K∑ k=0 θ′kTk(L̃)x , (5)
with L̃ = 2λmaxL − IN ; as can easily be verified by noticing that (UΛU >)k = UΛkU>. Note that this expression is nowK-localized since it is aK th-order polynomial in the Laplacian, i.e. it depends only on nodes that are at maximum K steps away from the central node (K th-order neighborhood). The complexity of evaluating Eq. 5 is O(|E|), i.e. linear in the number of edges. Defferrard et al. (2016) use this K-localized convolution to define a convolutional neural network on graphs.
2.2 LAYER-WISE LINEAR MODEL
A neural network model based on graph convolutions can therefore be built by stacking multiple convolutional layers of the form of Eq. 5, each layer followed by a point-wise non-linearity. Now, imagine we limited the layer-wise convolution operation to K = 1 (see Eq. 5), i.e. a function that is linear w.r.t. L and therefore a linear function on the graph Laplacian spectrum.
1We provide an alternative interpretation of this propagation rule based on the Weisfeiler-Lehman algorithm (Weisfeiler & Lehmann, 1968) in Appendix A.
In this way, we can still recover a rich class of convolutional filter functions by stacking multiple such layers, but we are not limited to the explicit parameterization given by, e.g., the Chebyshev polynomials. We intuitively expect that such a model can alleviate the problem of overfitting on local neighborhood structures for graphs with very wide node degree distributions, such as social networks, citation networks, knowledge graphs and many other real-world graph datasets. Additionally, for a fixed computational budget, this layer-wise linear formulation allows us to build deeper models, a practice that is known to improve modeling capacity on a number of domains (He et al., 2016).
In this linear formulation of a GCN we further approximate λmax ≈ 2, as we can expect that neural network parameters will adapt to this change in scale during training. Under these approximations Eq. 5 simplifies to:
gθ′ ? x ≈ θ′0x+ θ′1 (L− IN )x = θ′0x− θ′1D− 1 2AD− 1 2x , (6)
with two free parameters θ′0 and θ ′ 1. The filter parameters can be shared over the whole graph. Successive application of filters of this form then effectively convolve the kth-order neighborhood of a node, where k is the number of successive filtering operations or convolutional layers in the neural network model.
In practice, it can be beneficial to constrain the number of parameters further to address overfitting and to minimize the number of operations (such as matrix multiplications) per layer. This leaves us with the following expression:
gθ ? x ≈ θ ( IN +D − 12AD− 1 2 ) x , (7)
with a single parameter θ = θ′0 = −θ′1. Note that IN + D− 1 2AD− 1 2 now has eigenvalues in the range [0, 2]. Repeated application of this operator can therefore lead to numerical instabilities and exploding/vanishing gradients when used in a deep neural network model. To alleviate this problem, we introduce the following renormalization trick: IN +D− 1 2AD− 1 2 → D̃− 12 ÃD̃− 12 , with
Ã = A+ IN and D̃ii = ∑ j Ãij .
We can generalize this definition to a signalX ∈ RN×C withC input channels (i.e. aC-dimensional feature vector for every node) and F filters or feature maps as follows:
Z = D̃− 1 2 ÃD̃− 1 2XΘ , (8)
where Θ ∈ RC×F is now a matrix of filter parameters and Z ∈ RN×F is the convolved signal matrix. This filtering operation has complexity O(|E|FC), as ÃX can be efficiently implemented as a product of a sparse matrix with a dense matrix.
3 SEMI-SUPERVISED NODE CLASSIFICATION
Having introduced a simple, yet flexible model f(X,A) for efficient information propagation on graphs, we can return to the problem of semi-supervised node classification. As outlined in the introduction, we can relax certain assumptions typically made in graph-based semi-supervised learning by conditioning our model f(X,A) both on the data X and on the adjacency matrix A of the underlying graph structure. We expect this setting to be especially powerful in scenarios where the adjacency matrix contains information not present in the data X , such as citation links between documents in a citation network or relations in a knowledge graph. The overall model, a multi-layer GCN for semi-supervised learning, is schematically depicted in Figure 1.
3.1 EXAMPLE
In the following, we consider a two-layer GCN for semi-supervised node classification on a graph with a symmetric adjacency matrix A (binary or weighted). We first calculate Â = D̃− 1 2 ÃD̃− 1 2 in a pre-processing step. Our forward model then takes the simple form:
Z = f(X,A) = softmax ( Â ReLU ( ÂXW (0) ) W (1) ) . (9)
1
30
Here, W (0) ∈ RC×H is an input-to-hidden weight matrix for a hidden layer with H feature maps. W (1) ∈ RH×F is a hidden-to-output weight matrix. The softmax activation function, defined as softmax(xi) = 1 Z exp(xi) with Z = ∑ i exp(xi), is applied row-wise. For semi-supervised multiclass classification, we then evaluate the cross-entropy error over all labeled examples:
L = − ∑ l∈YL F∑ f=1 Ylf lnZlf , (10)
where YL is the set of node indices that have labels. The neural network weights W (0) and W (1) are trained using gradient descent. In this work, we perform batch gradient descent using the full dataset for every training iteration, which is a viable option as long as datasets fit in memory. Using a sparse representation for A, memory requirement is O(|E|), i.e. linear in the number of edges. Stochasticity in the training process is introduced via dropout (Srivastava et al., 2014). We leave memory-efficient extensions with mini-batch stochastic gradient descent for future work.
3.2 IMPLEMENTATION
In practice, we make use of TensorFlow (Abadi et al., 2015) for an efficient GPU-based implementation2 of Eq. 9 using sparse-dense matrix multiplications. The computational complexity of evaluating Eq. 9 is then O(|E|CHF ), i.e. linear in the number of graph edges.
4 RELATED WORK
Our model draws inspiration both from the field of graph-based semi-supervised learning and from recent work on neural networks that operate on graphs. In what follows, we provide a brief overview on related work in both fields.
4.1 GRAPH-BASED SEMI-SUPERVISED LEARNING
A large number of approaches for semi-supervised learning using graph representations have been proposed in recent years, most of which fall into two broad categories: methods that use some form of explicit graph Laplacian regularization and graph embedding-based approaches. Prominent examples for graph Laplacian regularization include label propagation (Zhu et al., 2003), manifold regularization (Belkin et al., 2006) and deep semi-supervised embedding (Weston et al., 2012).
2Code to reproduce our experiments is available at https://github.com/tkipf/gcn.
Recently, attention has shifted to models that learn graph embeddings with methods inspired by the skip-gram model (Mikolov et al., 2013). DeepWalk (Perozzi et al., 2014) learns embeddings via the prediction of the local neighborhood of nodes, sampled from random walks on the graph. LINE (Tang et al., 2015) and node2vec (Grover & Leskovec, 2016) extend DeepWalk with more sophisticated random walk or breadth-first search schemes. For all these methods, however, a multistep pipeline including random walk generation and semi-supervised training is required where each step has to be optimized separately. Planetoid (Yang et al., 2016) alleviates this by injecting label information in the process of learning embeddings.
4.2 NEURAL NETWORKS ON GRAPHS
Neural networks that operate on graphs have previously been introduced in Gori et al. (2005); Scarselli et al. (2009) as a form of recurrent neural network. Their framework requires the repeated application of contraction maps as propagation functions until node representations reach a stable fixed point. This restriction was later alleviated in Li et al. (2016) by introducing modern practices for recurrent neural network training to the original graph neural network framework. Duvenaud et al. (2015) introduced a convolution-like propagation rule on graphs and methods for graph-level classification. Their approach requires to learn node degree-specific weight matrices which does not scale to large graphs with wide node degree distributions. Our model instead uses a single weight matrix per layer and deals with varying node degrees through an appropriate normalization of the adjacency matrix (see Section 3.1).
A related approach to node classification with a graph-based neural network was recently introduced in Atwood & Towsley (2016). They report O(N2) complexity, limiting the range of possible applications. In a different yet related model, Niepert et al. (2016) convert graphs locally into sequences that are fed into a conventional 1D convolutional neural network, which requires the definition of a node ordering in a pre-processing step.
Our method is based on spectral graph convolutional neural networks, introduced in Bruna et al. (2014) and later extended by Defferrard et al. (2016) with fast localized convolutions. In contrast to these works, we consider here the task of transductive node classification within networks of significantly larger scale. We show that in this setting, a number of simplifications (see Section 2.2) can be introduced to the original frameworks of Bruna et al. (2014) and Defferrard et al. (2016) that improve scalability and classification performance in large-scale networks.
5 EXPERIMENTS
We test our model in a number of experiments: semi-supervised document classification in citation networks, semi-supervised entity classification in a bipartite graph extracted from a knowledge graph, an evaluation of various graph propagation models and a run-time analysis on random graphs.
5.1 DATASETS
We closely follow the experimental setup in Yang et al. (2016). Dataset statistics are summarized in Table 1. In the citation network datasets—Citeseer, Cora and Pubmed (Sen et al., 2008)—nodes are documents and edges are citation links. Label rate denotes the number of labeled nodes that are used for training divided by the total number of nodes in each dataset. NELL (Carlson et al., 2010; Yang et al., 2016) is a bipartite graph dataset extracted from a knowledge graph with 55,864 relation nodes and 9,891 entity nodes.
Citation networks We consider three citation network datasets: Citeseer, Cora and Pubmed (Sen et al., 2008). The datasets contain sparse bag-of-words feature vectors for each document and a list of citation links between documents. We treat the citation links as (undirected) edges and construct a binary, symmetric adjacency matrix A. Each document has a class label. For training, we only use 20 labels per class, but all feature vectors.
NELL NELL is a dataset extracted from the knowledge graph introduced in (Carlson et al., 2010). A knowledge graph is a set of entities connected with directed, labeled edges (relations). We follow the pre-processing scheme as described in Yang et al. (2016). We assign separate relation nodes r1 and r2 for each entity pair (e1, r, e2) as (e1, r1) and (e2, r2). Entity nodes are described by sparse feature vectors. We extend the number of features in NELL by assigning a unique one-hot representation for every relation node, effectively resulting in a 61,278-dim sparse feature vector per node. The semi-supervised task here considers the extreme case of only a single labeled example per class in the training set. We construct a binary, symmetric adjacency matrix from this graph by setting entries Aij = 1, if one or more edges are present between nodes i and j.
Random graphs We simulate random graph datasets of various sizes for experiments where we measure training time per epoch. For a dataset with N nodes we create a random graph assigning 2N edges uniformly at random. We take the identity matrix IN as input feature matrix X , thereby implicitly taking a featureless approach where the model is only informed about the identity of each node, specified by a unique one-hot vector. We add dummy labels Yi = 1 for every node.
5.2 EXPERIMENTAL SET-UP
Unless otherwise noted, we train a two-layer GCN as described in Section 3.1 and evaluate prediction accuracy on a test set of 1,000 labeled examples. We provide additional experiments using deeper models with up to 10 layers in Appendix B. We choose the same dataset splits as in Yang et al. (2016) with an additional validation set of 500 labeled examples for hyperparameter optimization (dropout rate for all layers, L2 regularization factor for the first GCN layer and number of hidden units). We do not use the validation set labels for training.
For the citation network datasets, we optimize hyperparameters on Cora only and use the same set of parameters for Citeseer and Pubmed. We train all models for a maximum of 200 epochs (training iterations) using Adam (Kingma & Ba, 2015) with a learning rate of 0.01 and early stopping with a window size of 10, i.e. we stop training if the validation loss does not decrease for 10 consecutive epochs. We initialize weights using the initialization described in Glorot & Bengio (2010) and accordingly (row-)normalize input feature vectors. On the random graph datasets, we use a hidden layer size of 32 units and omit regularization (i.e. neither dropout nor L2 regularization).
5.3 BASELINES
We compare against the same baseline methods as in Yang et al. (2016), i.e. label propagation (LP) (Zhu et al., 2003), semi-supervised embedding (SemiEmb) (Weston et al., 2012), manifold regularization (ManiReg) (Belkin et al., 2006) and skip-gram based graph embeddings (DeepWalk) (Perozzi et al., 2014). We omit TSVM (Joachims, 1999), as it does not scale to the large number of classes in one of our datasets.
We further compare against the iterative classification algorithm (ICA) proposed in Lu & Getoor (2003) in conjunction with two logistic regression classifiers, one for local node features alone and one for relational classification using local features and an aggregation operator as described in Sen et al. (2008). We first train the local classifier using all labeled training set nodes and use it to bootstrap class labels of unlabeled nodes for relational classifier training. We run iterative classification (relational classifier) with a random node ordering for 10 iterations on all unlabeled nodes (bootstrapped using the local classifier). L2 regularization parameter and aggregation operator (count vs. prop, see Sen et al. (2008)) are chosen based on validation set performance for each dataset separately.
Lastly, we compare against Planetoid (Yang et al., 2016), where we always choose their bestperforming model variant (transductive vs. inductive) as a baseline.
6 RESULTS
6.1 SEMI-SUPERVISED NODE CLASSIFICATION
Results are summarized in Table 2. Reported numbers denote classification accuracy in percent. For ICA, we report the mean accuracy of 100 runs with random node orderings. Results for all other baseline methods are taken from the Planetoid paper (Yang et al., 2016). Planetoid* denotes the best model for the respective dataset out of the variants presented in their paper.
We further report wall-clock training time in seconds until convergence (in brackets) for our method (incl. evaluation of validation error) and for Planetoid. For the latter, we used an implementation provided by the authors3 and trained on the same hardware (with GPU) as our GCN model. We trained and tested our model on the same dataset splits as in Yang et al. (2016) and report mean accuracy of 100 runs with random weight initializations. We used the following sets of hyperparameters for Citeseer, Cora and Pubmed: 0.5 (dropout rate), 5 · 10−4 (L2 regularization) and 16 (number of hidden units); and for NELL: 0.1 (dropout rate), 1 · 10−5 (L2 regularization) and 64 (number of hidden units).
In addition, we report performance of our model on 10 randomly drawn dataset splits of the same size as in Yang et al. (2016), denoted by GCN (rand. splits). Here, we report mean and standard error of prediction accuracy on the test set split in percent.
6.2 EVALUATION OF PROPAGATION MODEL
We compare different variants of our proposed per-layer propagation model on the citation network datasets. We follow the experimental set-up described in the previous section. Results are summarized in Table 3. The propagation model of our original GCN model is denoted by renormalization trick (in bold). In all other cases, the propagation model of both neural network layers is replaced with the model specified under propagation model. Reported numbers denote mean classification accuracy for 100 repeated runs with random weight matrix initializations. In case of multiple variables Θi per layer, we impose L2 regularization on all weight matrices of the first layer.
6.3 TRAINING TIME PER EPOCH
Here, we report results for the mean training time per epoch (forward pass, cross-entropy calculation, backward pass) for 100 epochs on simulated random graphs, measured in seconds wall-clock time. See Section 5.1 for a detailed description of the random graph dataset used in these experiments. We compare results on a GPU and on a CPU-only implementation4 in TensorFlow (Abadi et al., 2015). Figure 2 summarizes the results.
7 DISCUSSION
7.1 SEMI-SUPERVISED MODEL
In the experiments demonstrated here, our method for semi-supervised node classification outperforms recent related methods by a significant margin. Methods based on graph-Laplacian regularization (Zhu et al., 2003; Belkin et al., 2006; Weston et al., 2012) are most likely limited due to their assumption that edges encode mere similarity of nodes. Skip-gram based methods on the other hand are limited by the fact that they are based on a multi-step pipeline which is difficult to optimize. Our proposed model can overcome both limitations, while still comparing favorably in terms of efficiency (measured in wall-clock time) to related methods. Propagation of feature information from neighboring nodes in every layer improves classification performance in comparison to methods like ICA (Lu & Getoor, 2003), where only label information is aggregated.
We have further demonstrated that the proposed renormalized propagation model (Eq. 8) offers both improved efficiency (fewer parameters and operations, such as multiplication or addition) and better predictive performance on a number of datasets compared to a naı̈ve 1st-order model (Eq. 6) or higher-order graph convolutional models using Chebyshev polynomials (Eq. 5).
7.2 LIMITATIONS AND FUTURE WORK
Here, we describe several limitations of our current model and outline how these might be overcome in future work.
Memory requirement In the current setup with full-batch gradient descent, memory requirement grows linearly in the size of the dataset. We have shown that for large graphs that do not fit in GPU memory, training on CPU can still be a viable option. Mini-batch stochastic gradient descent can alleviate this issue. The procedure of generating mini-batches, however, should take into account the number of layers in the GCN model, as the K th-order neighborhood for a GCN with K layers has to be stored in memory for an exact procedure. For very large and densely connected graph datasets, further approximations might be necessary.
Directed edges and edge features Our framework currently does not naturally support edge features and is limited to undirected graphs (weighted or unweighted). Results on NELL however show that it is possible to handle both directed edges and edge features by representing the original directed graph as an undirected bipartite graph with additional nodes that represent edges in the original graph (see Section 5.1 for details).
Limiting assumptions Through the approximations introduced in Section 2, we implicitly assume locality (dependence on the K th-order neighborhood for a GCN with K layers) and equal importance of self-connections vs. edges to neighboring nodes. For some datasets, however, it might be beneficial to introduce a trade-off parameter λ in the definition of Ã:
Ã = A+ λIN . (11) 4Hardware used: 16-core Intel R© Xeon R© CPU E5-2640 v3 @ 2.60GHz, GeForce R© GTX TITAN X
This parameter now plays a similar role as the trade-off parameter between supervised and unsupervised loss in the typical semi-supervised setting (see Eq. 1). Here, however, it can be learned via gradient descent.
8 CONCLUSION
We have introduced a novel approach for semi-supervised classification on graph-structured data. Our GCN model uses an efficient layer-wise propagation rule that is based on a first-order approximation of spectral convolutions on graphs. Experiments on a number of network datasets suggest that the proposed GCN model is capable of encoding both graph structure and node features in a way useful for semi-supervised classification. In this setting, our model outperforms several recently proposed methods by a significant margin, while being computationally efficient.
ACKNOWLEDGMENTS
We would like to thank Christos Louizos, Taco Cohen, Joan Bruna, Zhilin Yang, Dave Herman, Pramod Sinha and Abdul-Saboor Sheikh for helpful discussions. This research was funded by SAP.
A RELATION TO WEISFEILER-LEHMAN ALGORITHM
A neural network model for graph-structured data should ideally be able to learn representations of nodes in a graph, taking both the graph structure and feature description of nodes into account. A well-studied framework for the unique assignment of node labels given a graph and (optionally) discrete initial node labels is provided by the 1-dim Weisfeiler-Lehman (WL-1) algorithm (Weisfeiler & Lehmann, 1968):
Algorithm 1: WL-1 algorithm (Weisfeiler & Lehmann, 1968)
Input: Initial node coloring (h(0)1 , h (0) 2 , ..., h (0) N ) Output: Final node coloring (h(T )1 , h (T ) 2 , ..., h (T ) N ) t← 0; repeat
for vi ∈ V do h (t+1) i ← hash (∑ j∈Ni h (t) j ) ;
t← t+ 1; until stable node coloring is reached;
Here, h(t)i denotes the coloring (label assignment) of node vi (at iteration t) and Ni is its set of neighboring node indices (irrespective of whether the graph includes self-connections for every node or not). hash(·) is a hash function. For an in-depth mathematical discussion of the WL-1 algorithm see, e.g., Douglas (2011).
We can replace the hash function in Algorithm 1 with a neural network layer-like differentiable function with trainable parameters as follows:
h (l+1) i = σ ∑ j∈Ni 1 cij h (l) j W (l)  , (12) where cij is an appropriately chosen normalization constant for the edge (vi, vj). Further, we can take h(l)i now to be a vector of activations of node i in the l
th neural network layer. W (l) is a layer-specific weight matrix and σ(·) denotes a differentiable, non-linear activation function.
By choosing cij = √ didj , where di = |Ni| denotes the degree of node vi, we recover the propagation rule of our Graph Convolutional Network (GCN) model in vector form (see Eq. 2)5.
This—loosely speaking—allows us to interpret our GCN model as a differentiable and parameterized generalization of the 1-dim Weisfeiler-Lehman algorithm on graphs.
A.1 NODE EMBEDDINGS WITH RANDOM WEIGHTS
From the analogy with the Weisfeiler-Lehman algorithm, we can understand that even an untrained GCN model with random weights can serve as a powerful feature extractor for nodes in a graph. As an example, consider the following 3-layer GCN model:
Z = tanh ( Â tanh ( Â tanh ( ÂXW (0) ) W (1) ) W (2) ) , (13)
with weight matricesW (l) initialized at random using the initialization described in Glorot & Bengio (2010). Â, X and Z are defined as in Section 3.1.
We apply this model on Zachary’s karate club network (Zachary, 1977). This graph contains 34 nodes, connected by 154 (undirected and unweighted) edges. Every node is labeled by one of four classes, obtained via modularity-based clustering (Brandes et al., 2008). See Figure 3a for an illustration.
5Note that we here implicitly assume that self-connections have already been added to every node in the graph (for a clutter-free notation).
We take a featureless approach by setting X = IN , where IN is the N by N identity matrix. N is the number of nodes in the graph. Note that nodes are randomly ordered (i.e. ordering contains no information). Furthermore, we choose a hidden layer dimensionality6 of 4 and a two-dimensional output (so that the output can immediately be visualized in a 2-dim plot).
Figure 3b shows a representative example of node embeddings (outputs Z) obtained from an untrained GCN model applied to the karate club network. These results are comparable to embeddings obtained from DeepWalk (Perozzi et al., 2014), which uses a more expensive unsupervised training procedure.
A.2 SEMI-SUPERVISED NODE EMBEDDINGS
On this simple example of a GCN applied to the karate club network it is interesting to observe how embeddings react during training on a semi-supervised classification task. Such a visualization (see Figure 4) provides insights into how the GCN model can make use of the graph structure (and of features extracted from the graph structure at later layers) to learn embeddings that are useful for a classification task.
We consider the following semi-supervised learning setup: we add a softmax layer on top of our model (Eq. 13) and train using only a single labeled example per class (i.e. a total number of 4 labeled nodes). We train for 300 training iterations using Adam (Kingma & Ba, 2015) with a learning rate of 0.01 on a cross-entropy loss.
Figure 4 shows the evolution of node embeddings over a number of training iterations. The model succeeds in linearly separating the communities based on minimal supervision and the graph structure alone. A video of the full training process can be found on our website7.
6We originally experimented with a hidden layer dimensionality of 2 (i.e. same as output layer), but observed that a dimensionality of 4 resulted in less frequent saturation of tanh(·) units and therefore visually more pleasing results.
7http://tkipf.github.io/graph-convolutional-networks/
B EXPERIMENTS ON MODEL DEPTH
In these experiments, we investigate the influence of model depth (number of layers) on classification performance. We report results on a 5-fold cross-validation experiment on the Cora, Citeseer and Pubmed datasets (Sen et al., 2008) using all labels. In addition to the standard GCN model (Eq. 2), we report results on a model variant where we use residual connections (He et al., 2016) between hidden layers to facilitate training of deeper models by enabling the model to carry over information from the previous layer’s input:
H(l+1) = σ ( D̃− 1 2 ÃD̃− 1 2H(l)W (l) ) +H(l) . (14)
On each cross-validation split, we train for 400 epochs (without early stopping) using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.01. Other hyperparameters are chosen as follows: 0.5 (dropout rate, first and last layer), 5 · 10−4 (L2 regularization, first layer), 16 (number of units for each hidden layer) and 0.01 (learning rate). Results are summarized in Figure 5.
For the datasets considered here, best results are obtained with a 2- or 3-layer model. We observe that for models deeper than 7 layers, training without the use of residual connections can become difficult, as the effective context size for each node increases by the size of its K th-order neighborhood (for a model with K layers) with each additional layer. Furthermore, overfitting can become an issue as the number of parameters increases with model depth.

The paper introduces a method for semi-supervised learning in graphs that exploits the spectral structure of the graph in a convolutional NN implementation. The proposed algorithm has a limited complexity and it is shown to scale well on a large dataset. The comparison with baselines on different datasets show a clear jump of performance with the proposed method.
The paper is technically fine and clear, the algorithm seems to scale well, and the results on the different datasets compare very favorably with the different baselines. The algorithm is simple and training seems easy. Concerning the originality, the proposed algorithm is a simple adaptation of graph convolutional networks (ref Defferrard 2016 in the paper) to a semi-supervised transductive setting. This is clearly mentioned in the paper, but the authors could better highlight the differences and novelty wrt this reference paper. Also, there is no comparison with the family of iterative classifiers, which usually compare favorably, both in performance and training time, with regularization based approaches, although they are mostly used in inductive settings. Below are some references for this family of methods.
The authors mention that more complex filters could be learned by stacking layers but they limit their architecture to one hidden layer. They should comment on the interest of using more layers for graph classification.
Some references on iterative classification Qing Lu and Lise Getoor. 2003. Link-based classification. In ICML, Vol. 3. 496–503.
Gideon S Mann and Andrew McCallum. 2010. Generalized expectation criteria for semi-supervised learning with weakly labeled data. The Journal of Machine Learning Research 11 (2010), 955–984.
David Jensen, Jennifer Neville, and Brian Gallagher. 2004. Why collective inference improves relational classification. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 593–598.
Joseph J Pfeiffer III, Jennifer Neville, and Paul N Bennett. 2015. Overcoming Relational Learning Biases
to Accurately Predict Preferences in Large Scale Networks. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 853–
863.
Stephane Peters, Ludovic Denoyer, and Patrick Gallinari. 2010. Iterative annotation of multi-relational social networks. In Advances in Social Networks Analysis and Mining (ASONAM), 2010 International Conference on. IEEE, 96–103.


Recently deep neural networks have received considerable attention due to their ability to extract and represent high-level abstractions in data sets. Deep neural networks such as fully-connected and convolutional neural networks have shown excellent performance on a wide range of recognition and classification tasks. However, their hardware implementations currently suffer from large silicon area and high power consumption due to the their high degree of complexity. The power/energy consumption of neural networks is dominated by memory accesses, the majority of which occur in fully-connected networks. In fact, they contain most of the deep neural network parameters. In this paper, we propose sparsely-connected networks, by showing that the number of connections in fullyconnected networks can be reduced by up to 90% while improving the accuracy performance on three popular datasets (MNIST, CIFAR10 and SVHN). We then propose an efficient hardware architecture based on linear-feedback shift registers to reduce the memory requirements of the proposed sparsely-connected networks. The proposed architecture can save up to 90% of memory compared to the conventional implementations of fully-connected neural networks. Moreover, implementation results show up to 84% reduction in the energy consumption of a single neuron of the proposed sparsely-connected networks compared to a single neuron of fully-connected neural networks.
1 INTRODUCTION
Deep neural networks (DNNs) have shown remarkable performance in extracting and representing high-level abstractions in complex data (Lecun et al. (2015)). DNNs rely on multiple layers of interconnected neurons and parameters to solve complex tasks, such as image recognition and classification (Krizhevsky et al. (2012)). While they have been proven very effective in said tasks, their hardware implementations still suffer from high memory and power consumption, due to the complexity and size of their models. Therefore, research efforts have been conducted towards more efficient implementations of DNNs (Han et al. (2016)). In the past few years, the parallel nature of DNNs has led to the use of graphical processing units (GPUs) to execute neural networks tasks (Han et al. (2015)). However, their large latency and power consumption have pushed researchers towards application-specific integrated circuits (ASICs) for hardware implementations (Cavigelli et al. (2015)). For instance, in (Han et al. (2016)), it was shown that a DNN implemented with customized hardware can accelerate the classification task by 189× and 13×, while saving 24,000× and 3,400× energy compared to CPU (Intel i7-5930k) and GPU (GeForce TITAN X), respectively. Convolutional layers in DNNs are used to extract high level abstractions and features of data. In such layers, the connectivity between neurons follows a pattern inspired by the organization of the animal visual cortex. It was shown that the computation in the visual cortex can mathematically be described by a convolution operation (LeCun et al. (1989)). Therefore, each neuron is only connected to a few neurons based on a pattern and a set of weights is shared among all neurons. In contrast, in a fully-connected layer, each neuron is connected to every neuron in the previous and next layers and each connection is associated with a weight. These layers are usually used to learn non-linear
combinations of given data. Fig. 1 shows a two-layer fully-connected network. The main computation kernel performs numerous vector-matrix multiplications followed by non-linear functions in each layer. In (Courbariaux & Bengio (2016); Horowitz (2014); Han et al. (2016)), it was shown that the power/energy consumption of DNNs is dominated by memory accesses. Fully-connected layers, which are widely used in recurrent neural networks (RNNs) and adopted in many state-of-the-art neural network architectures (Krizhevsky et al. (2012); Simonyan & Zisserman (2014); Zeiler & Fergus (2013); Szegedy et al. (2015); Lecun et al. (1998)), independently or as a part of convolutional neural networks, contain most of the weights of a DNN. For instance, the first fully-connected layer of VGGNet (Simonyan & Zisserman (2014)), which is composed of 13 convolution layers and three fully-connected layers, contains 100M weights out of a total of 140M. Such large storage requirements in fully-connected networks result in copious power/energy consumption.
To overcome the aforementioned issue, a pruning technique was first introduced in (Han et al. (2015)) to reduce the memory required by DNN architectures for mobile applications. However, it makes use of an additional training stage, while information addresses identifying the pruned connections still need to be stored in a memory. More recently, several works have focused on the binarization and ternarization of the weights of DNNs (Courbariaux & Bengio (2016); Courbariaux et al. (2015); Lin et al. (2015); Kim & Smaragdis (2016)). While these approaches reduce weight quantization and thus the memory width, the number of weights is unchanged.
In (Shafiee et al. (2016b)), an alternative deep network connectivity named StochasticNet and inspired from the brain synaptic connection between neurons was explored on low-power CPUs. StochasticNet is formed by randomly removing up to 61% connections in both fully-connected and convolution layers of DNNs, speeding up the classification task.
In (Wen et al. (2016)), a method named structured sparsity learning (SSL) was introduced to regularize the convolutional layers’ structures of DNNs. SSL can learn a structured sparsity of DNNs to efficiently speed up the convolutional computations both on CPU and GPU platforms.
In this paper, we propose sparsely-connected networks by randomly removing some of the connections in fully-connected networks. Random connection masks are generated by linear-feedback shift registers (LFSRs), which are also used in the VLSI implementation to disable the connections. Experimental results on three commonly used datasets show that the proposed networks can improve network accuracy while removing up to 90% of the connections. Additionally, we apply the proposed algorithm on top of the binarizing/ternarizing technique achieving a better misclassification rate than the best binarized/ternarized networks reported in literature. Finally, an efficient very large scale integration (VLSI) hardware architecture of a DNN based on sparsely-connected network is proposed, which saves up to 90% memory and 84% energy with respect to the traditional architectures.
The rest of the paper is organized as follows. Section 2 briefly introduces DNNs and their hardware implementation challenges, while Section 3 describes the proposed sparsely-connected network and their training algorithm. In Section 4 the experimental results over three datasets are presented and compared to the state of the art. Section 5 portrays the proposed VLSI architecture for the sparselyconnected network, and conclusions are drawn in Section 6.
2 PRELIMINARIES
2.1 DEEP NEURAL NETWORKS
DNNs are constructed using multiple layers of neurons between the input and output layers. These are usually referred to as hidden layers. They are used in many current image and speech applications to perform complex tasks as recognition or classification. DNNs are trained through an initial phase, called the learning stage, that uses data to prepare the DNN for the task that will follow in the inference stage. Two subcategories of DNNs which are widely used in detection and recognition tasks are convolutional neural networks (CNNs) and RNNs (Han et al. (2016)). Due to parameter reuse in convolutional layers, they are well-studied and can be efficiently implemented with customized hardware platforms (Chen et al. (2016); Shafiee et al. (2016a); Chen et al. (2016)). On the other hand, fully-connected layers, which are widely used in RNNs like long short-term memories and as a part of CNNs, require a large number of parameters to be stored in memories.
DNNs are mostly trained by the backpropagation algorithm in conjunction with stochastic gradient descent (SGD) optimization method (Rumelhart et al. (1986)). This algorithm computes the gradient of a cost function C with respect to all the weights in all the layers. A common choice for the cost function is using the modified hinge loss introduced in (Tang (2013)). The obtained errors are then backward propagated through the layers to update the weights in an attempt to minimize the cost function. Instead of using a whole dataset to update parameters, data are first divided in mini-batches and parameters are updated using each mini-batch several times to speed up the convergence of the training algorithm. The weight updating speed is controlled by a learning rate η. Batch normalization is also commonly used to regularize each mini-batch of data (Ioffe & Szegedy (2015)): it speeds up the training process by allowing the use of a bigger η.
2.2 TOWARDS HARDWARE IMPLEMENTATION OF DNNS
DNNs have shown excellent performance in applications such as computer vision and speech recognition: since the number of neurons has a linear relationship with the ability of a DNN to perform tasks, high-performance DNNs are extremely complex in hardware. AlexNet (Krizhevsky et al. (2012)) and VGGNet (Simonyan & Zisserman (2014)) are two models comprising convolutional layers followed by some fully-connected layers, which are widely used in classification algorithms. Despite their very good classification performance, they require large amounts of memory to store the numerous parameters. Most of these parameters (more than 96%) lie in fully-connected layers. In (Han et al. (2016)), it was shown that the total energy of DNNs is dominated by the required memory accesses. Therefore, the majority of power in a DNN is dissipated through fully-connected layers of DNNs. Moreover, the huge memory requirements make possible only for very small DNNs to be fitted in on-chip RAMs in ASIC/FPGA platforms.
Recently, many works tried to reduce the computational complexity of DNNs. In (Akopyan et al. (2015)), the spiking neural network based on stochastic computing (Smithson et al. (2016)) was introduced, where 1-bit calculations are performed throughout the whole architecture. In (Ardakani et al. (2015)), integral stochastic computing was used to reduce the computation latency, showing that stochastic computing can consume less energy than conventional binary radix implementations. However, both works do not manage to reduce the DNN memory requirements.
Network pruning, compression and weight sharing have been proposed in (Han et al. (2016)), together with weight matrix sparsification and compression. However, additional indexes denoting the pruned connections are required to be stored along with the compressed weight matrices. In (Han et al. (2015)), it was shown that the number of indexes are almost the same as the number of non-zero elements of weight matrices, thus increasing the word length of the required memories. Moreover, the encoding and compression techniques require inverse computations to obtain decoded and decompressed weights, and introduce additional hardware complexity for hardware implementation compared to the conventional computational architectures. Other pruning techniques presented in literature such as (Anwar et al. (2015)) try to reduce the memory required to store the pruned locations by introducing a structured sparsity in DNNs. However, the resulting network yields up to 31.81% misclassification rate on the CIFAR-10 dataset.
Algorithm 1: Training algorithm for the proposed sparsely-connected network Data: Fully-connected network with parameters W , b and M for each layer. Input data x, its corresponding targets t, and learning rate of η. Result: W and b
1 1. Forward computations 2 for each layer i in range(1,N) do 3 Ws ←Wi ·Mi 4 Compute layer output yi according to (3) and its previous layer output yi−1, Ws and bi. 5 end 6 2. Backward Computations
7 Initialize output layers activation gradient ∂C ∂yN 8 for each layer j in range(2,N-1) do 9 Compute ∂C
∂yj 10 end 11 for each layer j in range(1,N-1) do 12 Compute ∂C
∂Ws knowing
∂C ∂yj and yj−1
13 Compute ∂C
∂bj
14 Update Wj :Wj ←Wj − η ∂C
∂Ws
15 Update bj : bj ← bj − η ∂C ∂bj 16 end
3 SPARSELY-CONNECTED NEURAL NETWORKS
Considering a fully-connected neural network layer with n input and m output nodes, the forward computations are performed as follow
y = act(Wx+ b), (1)
where W represents the weights and b the biases, while act() is the non-linear activation function in which ReLU(x) = max(0, x) is used in most cases (Nair & Hinton (2010)). The network’s inputs and outputs are denoted by x and y, respectively.
Let us introduce the sparse weight matrix Ws as the element-wise multiplication
Ws =W ·M, (2)
where Ws and M are sparser than W . The Mask binary matrix M can be defined as
Mn×m =  M11 M12 . . . M1m M21 M22 . . . M2m
... ...
. . . ...
Mn1 Mn2 . . . Mnm  , where each element of Mask Mij ∈ {0, 1}, i ∈ {1, . . . , n} and j ∈ {1, . . . ,m}. Note that the dimensions of M are the same as the weight matrix W . Similarly to a fully-connected network (1), the forward computation of the sparsely-connected network can be expressed as
y = act(Wsx+ b). (3)
We propose the use of LFSRs to form each column of M , similar to the approach used in stochastic computing to generate a binary stream (Gaines (1969)). In general, an nb-bit LFSR serially generates 2nb−1 numbers Si ∈ (0, 1), i ∈ {1, 2, . . . 2nb−1}. A random binary stream with expected value of p ∈ [0 1] can be obtained by comparing Si with a constant value of p. This unit is hereafter referred
100
101
0.125, 0.5, 0.25, 0.625, 0.75, 0.875, 0.375
0.625, 0.75, 0.875, 0.375, 0.125, 0.5, 0.25
in ≥ p
in ≥ p
0, 1, 0, 1, 1, 1, 0
1, 1, 1, 0, 0, 1, 0
=> M = 0 0 1 1 1 0 1 0 1 0 0 1 1 1
p = 0.57
p = 0.57
(a)
to as stochastic number generator (SNG). Therefore, a random binary stream element Xi ∈ {0, 1} is 1 when Si ≥ p, and 0 otherwise. Fig. 2 shows the formation of a small sparsely-connected network using binary streams generated by LFSR units. Fig. 2(a) shows a 3-bit LFSR unit with its 7 different values and a random binary stream with expected value of p = 0.57. A total of m LFSRs of log2(n)-bit length with different seed values are required to form M . By tuning the value of p it is possible to change the sparsity degree of M , and thus of the sparsely-connected network. Fig. 2(b) and Fig. 2(c) show the fully-connected network based on W and the sparsely-connected version based on Ws.
Algorithm 1 summarizes the training algorithm for the proposed sparsely-connected network. The algorithm itself is very similar to what would be used with a fully-connected network, but considers each network layer to have a mask that disables some of the connections. The forward propagation (line 1-5) follows (3), while derivatives in the backward computations (line 6-16) are computed with respect to Ws. It is worth mentioning that most CNNs use fully-connected layers and the proposed training algorithm can still be used for those layers in CNNs.
4 EXPERIMENTAL RESULTS
We have validated the effectiveness of the proposed sparsely-connected network and its training algorithm on three datasets: MNIST (LeCun & Cortes (2010)), CIFAR10 (Krizhevsky (2009)) and SVHN (Netzer et al. (2011)) using the Theano library (Team (2016)) in Python.
4.1 EXPERIMENTAL RESULTS ON MNIST
The MNIST dataset contains 60000 gray-scale 28 × 28 images (50000 for training and 10000 for testing), falling into 10 classes. A deep fully-connected neural network is used for evaluation and the hinge loss is considered as the cost function. The training set is divided into two separate parts. The first 40000 images are used as the training set and the rest for the validation and test sets. All models are trained using SGD without momentum, a batch size of 100, 500 epochs and the batch normalization method.
Table 1 summarizes the misclassification rate of sparsely-connected neural networks compared to fully-connected neural networks for different network configurations, using single-precision floating-point format. We adopted a fully-connected network with 784-512-512-10 network configuration as a reference network, in which each number represent the number of inputs to each fully-connected layer. From this, we formed sparse weight matrices Ws with different sparsity degrees. For instance, sparsely-connected 90% denotes sparse weight matrices containing 90% zero elements. Case 1 shows that a sparsely-connected neural network with 50% fewer connections achieves approximately the same accuracy as the fully-connected network using the same network configuration. In Cases 2 and 3, the sparsely-connected networks with 60% and 80% fewer connections achieve a better misclassification rate than the fully-connected network while having approximately the same number of parameters. Case 4 shows no gain in performance and number of parameters for a sparsely-connected 90% and network configuration of 784-512-512-10 compared to the fully-connected at the same number of parameters. However, we can still reduce the connections up to 90% using a smaller network, as shown in Case 5.
Recently, BinaryConnect and TernaryConnect neural networks have outperformed the state-of-theart on different datasets (Courbariaux et al. (2015); Lin et al. (2015)). In BinaryConnect, weights are represented with either -1 or 1, whereas they can be -1, 0 or 1 in TernaryConnect. These networks have emerged to facilitate hardware implementations of neural networks by reducing the memory requirements and removing multiplications. We applied our training method to BinaryConnect and TernaryConnect training algorithms: the obtained results are provided in Table 2. The source Python codes used for comparison are the same used in (Courbariaux et al. (2015); Lin et al. (2015)), available online (Lin et al. (2015)). The simulation results show that up to 70% and 80% of connections can be dropped by the proposed method from BinaryConnect and TernaryConnect networks without any compromise in performance without using data augmentation, respectively. Moreover, the binarized and ternarized sparsely-connected 50% improve the accuracy compared to the conventional binarized and ternarized fully-connected networks. Considering data augmentation (affine transformation), our method can drop up to 50% and 70% of connections from BinaryConnect and TernaryConnect networks without any compromise in performance, respectively. However, using data augmentation results in a better misclassification rate when it is used on networks trained with single-precision floating-point weights as shown in Table 2. In this case, our method still can drop up to 90% of connections without any performance degradation. It is worth specifying that we only
used the binarized/ternarized algorithm during the learning phase, and we used single-precision floating-point weights during the test run in Section 4, similar to the approach used in (Lin et al. (2015)).
4.2 EXPERIMENTAL RESULTS ON CIFAR10
The CIFAR10 dataset consists of a total number of 60, 000 32×32 RGB images. Similar to MNIST, we split the images into 40, 000, 10, 000 and 10, 000 training, validation and test datasets, respectively. As our model, we adopt a convolutional network comprising {128-128-256-256-512-512} channels for six convolution/pooling layers and two 1024-node fully-connected layers followed by a classification layer. This architecture is inspired by VGGNet (Simonyan & Zisserman (2014)) and was also used in (Courbariaux et al. (2015)). Hinge loss is used for training with batch normalization and a batch size of 50.
In order to show the performance of the proposed technique, we use sparsely-connected networks instead of fully-connected networks in the convolutional network. Again, we compare our results with the binarized and ternarized models since they are the most hardware-friendly models reported to-date. As summarized in Table 3, simulation results show significant improvement in accuracy compared to the ordinary network while having significantly fewer parameters.
4.3 EXPERIMENTAL RESULTS ON SVHN
SVHN dataset contains 32 × 32 RGB images (600, 000 images for training and roughly 26, 000 images for testing) of street house numbers. Also, 6, 000 images are separated from the training part for validation. Similar to the CIFAR10 case, we use a convolutional network comprising {128-128- 256-256-512-512} channels for six convolution/pooling layers and two 1024 fully-connected layers followed by a classification layer. Hinge loss is used as the cost function with batch normalization and batch size of 50.
Table 4 summarizes the accuracy performance of using the proposed sparsely-connected network in the convolutional network model, compared to the hardware-friendly binarized and ternarized
models. Despite the fewer parameters that the proposed sparsely-connected network provides, it also yields state-of-the-art results in terms of accuracy performance.
4.4 COMPARISON WITH THE STATE OF THE ART
The proposed sparsely-connected network has been compared to other networks in literature in terms of misclassification rate in Table 5. In Section 4.1 to 4.3, we used the binarization/ternarization algorithm to train our models in the learning phase while using single-precision floating-point weights during the test run (i.e. inference phase). The first part of Table 5 applies the same technique, while in the second part we use binarized/ternarized weights also during the test run. We thus exploit a deterministic method introduced in (Courbariaux et al. (2015)) to perform the test run using binarized/ternarized weights. The weights are obtained as follows:
Wb = { 1 if W ≥ 0 -1 otherwise ,
Wt =  1 if W ≥ 1 3 0 otherwise
-1 if W ≤ -1 3
,
where Wb and Wt denote binarized and ternarized weights, respectively.
From the results presented in Table 5, we can see that our proposed work outperforms the state-ofthe-art models with binarized/ternarized weights during the test run while achieving performance
close to the state-of-the-art result of the model with no binarization/ternarization in the test run. The former are the most suitable and hardware-friendly models for hardware implementation of DNNs: our model shows a better performance in terms of both accuracy/misclassification rate and memory requirements. The obtained results suggest that the proposed network acts as a regularizer to prevent models from overfitting. Similar conclusions were also obtained in (Courbariaux et al. (2015)). It is worth noting that no data augmentation was used in our simulations throughout this paper except for the results reported in Table 2 and Table 3.
5 VLSI IMPLEMENTATION OF SPARSELY-CONNECTED NEURAL NETWORKS
In this Section, we propose an efficient hardware architecture for the proposed sparsely-connected network. In fully-connected networks, the main computational core is the matrix-vector multiplication that computes (1). This computation is usually implemented in parallel on GPUs. However, parallel implementation of this unit requires parallel access to memories and causes routing congestion, leading to large silicon area and power/energy consumption in customized hardware. Thus, VLSI architectures usually opt for semi-parallel implementations of such networks. In this approach, each neuron performs its computations serially, and a certain number of neurons are instantiated in parallel (Moreno et al. (2008)). Every neuron is implemented using multiply-and-accumulate (MAC) units as shown in Fig. 3(a). The number of inputs of each neuron determines the latency of this architecture. For example, considering a hidden layer with 1024 inputs and 1024 outputs, 1024 MACs are required in parallel and each MAC requires 1024 clock cycles to perform computations of this layer. In general, a counter is required to count from 0 to N − 1 where N is the number of inputs of each neuron. It provides the addresses for the memory in which a column of the weight matrix W is stored. In this way, each input and its corresponding weight are fed to the multiplier every clock cycle (see Fig. 3(a)). For binarized/ternarized networks, the multiplier in 3(a) is substituted with a multiplexer.
In Section 3, we described the formation of the Mask matrix M using an SNG unit (see Fig. 2(a)). The value of p, through which it is possible to tune the sparsity degree of networks, also corresponds to the occurrence of 1 in a binary stream generated by SNG. Therefore, we can save up to 90% of memory by storing only the weights corresponding to the 1s in the SNG stream. For instance,
considering a Mask matrix M in Fig. 2(a), Ws is formed as
Ws =
0 0 W21 W22 W31 0 W41 0 0 W52 W61 W62 0 W72
 ,
and the compressed matrix Wc stored in on-chip memories is
Wc = W21 W22W31 W52W41 W62 W61 W72  . The smaller memory can significantly reduce the silicon area and the power consumption of DNNs architectures. Depending on the value of p, the size of the memory varies. In general, the depth of the weight memory in each neuron is (1− p)×N . Fig. 3(b) depicts the architecture of a single neuron of the proposed sparsely-connected network. Decompression is performed using an SNG generating the enable signal of the counter and accumulator. Inputs are fed into each neuron sequentially in each clock cycle. If the output of the SNG is 1, the counter counts upward and provides an address for the memory. Then, the multiplication of an input and its corresponding weight is computed, the result stored in the internal register of the accumulator. If instead the output of the SNG is 0, the counter holds its previous value, while the internal register of the accumulator is not enabled, and does not load a new value. The latency of the proposed architecture is the same as that of the conventional architecture.
Table 6 shows the ASIC implementation results of the neuron in Fig. 3(b) supposing 1024 inputs. The proposed architectures were described in VHDL and synthesized in TSMC 65 nm CMOS technology with Cadence RTL compiler, for different sparsity degrees p. For the provided syntheses we used a binarized network. Implementation results show up to 84% decrement in energy consumption and up to 90% less area compared to the conventional fully-connected architecture.
6 CONCLUSION
DNNs are capable of solving complex tasks: their ability to do so depends on the number of neurons and their connections. Fully-connected layers in DNNs contain more than 96% of the total neural network parameters, pushing the designers to use off-chip memories which are band-width limited and consume large amounts of energy. In this paper, we proposed sparsely-connected networks and their training algorithm to substantially reduce the memory requirements of DNNs. The sparsity degree of the proposed network can be tuned by an SNG, which is implemented using an LFSR unit and a comparator. We used the proposed sparsely-connected network instead of fully-connected networks in a VGG-like network on three commonly used datasets: we achieved better accuracy results with up to 90% fewer connections than the state of the art. Moreover, our simulation results confirm that the proposed network can be used as a regularizer to prevent models from overfitting. Finally, we implemented a single neuron of the sparsely-connected network in in 65 nm CMOS technology for different sparsity degrees. The implementation results show that the proposed architecture can save up to 84% energy and 90% silicon area compared to the conventional fully-connected network while having a lower misclassification rate.

The paper proposes a sparsely connected network and an efficient hardware architecture that can save up to 90% of memory compared to the conventional implementations of fully connected neural networks.
The paper removes some of the connections in the fully connected layers and shows performance and computational efficiency increase in networks on three different datasets. It is also a good addition that the authors combine their method with binary and ternary connect studies and show further improvements.
The paper was hard for me to understand because of this misleading statement: In this paper, we propose sparsely-connected networks by reducing the number of connections of fully-connected networks using linear-feedback shift registers (LFSRs). It led me to think that LFSRs reduced the connections by keeping some of the information in the registers. However, LFSR is only used as a random binary generator. Any random generator could be used but LFSR is chosen for the convenience in VLSI implementation.
This explanation would be clearer to me: In this paper, we propose sparsely-connected networks by randomly removing some of the connections in fully-connected networks. Random connection masks are generated by LFSR, which is also used in the VLSI implementation to disable the connections.
Algorithm 1 is basically training a network with back-propogation where each layer has a binary mask that disables some of the connections. This explanation can be added to the text.
Using random connections is not a new idea in CNNs. It was used between CNN layers in a 1998 paper by Yann LeCun and others:


We investigate deep generative models that can exchange multiple modalities bidirectionally, e.g., generating images from corresponding texts and vice versa. Recently, some studies handle multiple modalities on deep generative models, such as variational autoencoders (VAEs). However, these models typically assume that modalities are forced to have a conditioned relation, i.e., we can only generate modalities in one direction. To achieve our objective, we should extract a joint representation that captures high-level concepts among all modalities and through which we can exchange them bi-directionally. As described herein, we propose a joint multimodal variational autoencoder (JMVAE), in which all modalities are independently conditioned on joint representation. In other words, it models a joint distribution of modalities. Furthermore, to be able to generate missing modalities from the remaining modalities properly, we develop an additional method, JMVAE-kl, that is trained by reducing the divergence between JMVAE’s encoder and prepared networks of respective modalities. Our experiments show that our proposedmethod can obtain appropriate joint representation frommultiple modalities and that it can generate and reconstruct themmore properly than conventional VAEs. We further demonstrate that JMVAE can generate multiple modalities bidirectionally.
We investigate deep generative models that can exchange multiple modalities bidirectionally, e.g., generating images from corresponding texts and vice versa. Recently, some studies handle multiple modalities on deep generative models, such as variational autoencoders (VAEs). However, these models typically assume that modalities are forced to have a conditioned relation, i.e., we can only generate modalities in one direction. To achieve our objective, we should extract a joint representation that captures high-level concepts among all modalities and through which we can exchange them bi-directionally. As described herein, we propose a joint multimodal variational autoencoder (JMVAE), in which all modalities are independently conditioned on joint representation. In other words, it models a joint distribution of modalities. Furthermore, to be able to generate missing modalities from the remaining modalities properly, we develop an additional method, JMVAE-kl, that is trained by reducing the divergence between JMVAE’s encoder and prepared networks of respective modalities. Our experiments show that our proposedmethod can obtain appropriate joint representation frommultiple modalities and that it can generate and reconstruct themmore properly than conventional VAEs. We further demonstrate that JMVAE can generate multiple modalities bidirectionally.
1 INTRODUCTION
In our world, information is represented through various modalities. While images are represented by pixel information, these can also be described with text or tag information. People often exchange such information bi-directionally. For instance, we can not only imagine what “a young female with a smile who does not wear glasses” looks like, but also add this caption to a corresponding photograph. To do so, it is important to extract a joint representation that captures high-level concepts among all modalities. Then we can bi-directionally generate modalities through the joint representations. However, each modality typically has a different kind of dimension and structure, e.g., images (real-valued and dense) and texts (discrete and sparse). Therefore, the relations between each modality and the joint representations might become high nonlinearity. To discover such relations, deep neural network architectures have been used widely for multimodal learning (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2012). The common approach with these models to learn joint representations is to share the top of hidden layers in modality specific networks. Among them, generative approaches using deep Boltzmannmachines (DBMs) (Srivastava & Salakhutdinov, 2012; Sohn et al., 2014) offer the important advantage that these can generate modalities bi-directionally.
Recently, variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) have been proposed to estimate flexible deep generative models by variational inference methods. These models use back-propagation during training, so that it can be trained on large-scale and highdimensional dataset compared with DBMs with MCMC training. Some studies have addressed to handle such large-scale and high-dimensionalmodalities on VAEs, but they are forced to model conditional distribution (Kingma et al., 2014; Sohn et al., 2015; Pandey & Dukkipati, 2016). Therefore, it can only generate modalities in one direction. For example, we cannot obtain generated images from texts if we train the likelihood of texts given images. To generate modalities bi-directionally,
all modalities should be treated equally under the learned joint representations, which is the same as previous multimodal learning models before VAEs.
As described in this paper, we develop a novel multimodal learning model with VAEs, which we call a joint multimodal variational autoencoder (JMVAE). The most significant feature of our model is that all modalities, x and w (e.g., images and texts), are conditioned independently on a latent variable z corresponding to joint representation, i.e., the JMVAE models a joint distribution of all modalities, p(x,w). Therefore, we can extract a high-level representation that contains all information of modalities. Moreover, since it models a joint distribution, we can draw samples from both p(x|w) and p(w|x). Because, at this time, modalities that we want to generate are usually missing, the inferred latent variable becomes incomplete and generated samples might be collapsed in the testing time when missing modalities are high-dimensional and complicated. To prevent this issue, we propose a method of preparing the new encoders for each modality, p(z|x) and p(z|w), and reducing the divergence between the multimodal encoder p(z|x,w), which we call JMVAE-kl. This contributes to more effective bi-directional generation of modalities, e.g., from face images to texts (attributes) and vice versa (see Figure 1).
The main contributions of this paper are as follows:
• We introduce a joint multimodal variational autoencoder (JMVAE), which is the first study to train joint distribution of modalities with VAEs.
• We propose an additional method (JMVAE-kl), which prevents generated samples from being collapsed when some modalities are missing. We experimentally confirm that this method solves this issue.
• We show qualitatively and quantitatively that JMVAE can extract appropriate joint distribution and that it can generate and reconstruct modalities similarly or more properly than conventional VAEs.
• We demonstrate that the JMVAE can generate multiple modalities bi-directionally even if these modalities have completely different kinds of dimensions and structures, e.g., highdimentional color face images and low-dimentional binary attributes.
2 RELATED WORK
The common approach of multimodal learning with deep neural networks is to share the top of hidden layers in modality specific networks. Ngiam et al. (2011) proposed this approach with deep autoencoders (AEs) and found that it can extract better representations than single modality settings. Srivastava & Salakhutdinov (2012) also took this idea but used deep Boltzmann machines (DBMs) (Salakhutdinov & Hinton, 2009). DBMs are generative models with undirected connections based on maximum joint likelihood learning of all modalities. Therefore, this model can generate modalities bi-directionally. Sohn et al. (2014) improved this model to exchange multiple modalities effectively, which are based on minimizing the variation of information and JMVAE-kl in ours can be regarded as minimizing it with variational learning on parameterized distributions (see Section 3.3
and Appendix A). However, it is computationally difficult for DBMs to train high-dimensional data such as natural images because of MCMC training.
Recently, VAEs (Kingma & Welling, 2013; Rezende et al., 2014) are used to train such highdimensional modalities. Kingma et al. (2014); Sohn et al. (2015) propose conditional VAEs (CVAEs), which maximize a conditional log-likelihood by variational methods. Many studies are based on CVAEs to train various multiple modalities such as handwriting digits and labels (Kingma et al., 2014; Sohn et al., 2015), object images and degrees of rotation (Kulkarni et al., 2015), face images and attributes (Larsen et al., 2015; Yan et al., 2015), and natural images and captions (Mansimov et al., 2015). The main features of CVAEs are that the relation between modalities is one-way and a latent variable does not contain the information of a conditioned modality1, which are unsuitable for our objective.
Pandey & Dukkipati (2016) proposed a conditional multimodal autoencoder (CMMA), which also maximizes the conditional log-likelihood. The difference between CVAEs is that a latent variable is connected directly from a conditional variable, i.e., these variables are not independent. Moreover, this model forces the latent representation from an input to be close to the joint representation from multiple inputs, which is similar to JMVAE-kl. However, the CMMA still considers that modalities are generated in fixed direction. This is the most different part from ours.
3 METHODS
This section first introduces the algorithm of VAEs briefly and then proposes a novel multimodal learning model with VAEs, which we call the joint multimodal variational autoencoder (JMVAE).
3.1 VARIATIONAL AUTOENCODERS
Given observation variables x and corresponding latent variables z, their generating processes are definable as z ∼ p(z) = N (0, I) and x ∼ pθ(x|z), where θ is the model parameter of p. The objective of VAEs is maximization of the marginal distribution p(x) = ∫ pθ(x|z)p(z)dx. Because this distribution is intractable, we instead train the model to maximize the following lower bound of the marginal distribution LV AE(x) as
log p(x) ≥ −DKL(qφ(z|x)||p(z)) + Eqφ(z|x)[log pθ(x|z)] = LV AE(x), (1)
where qφ(z|x) is an approximate distribution of posterior p(z|x) and φ is the model parameter of q. We designate qφ(z|x) as encoder and pθ(x|z) as decoder. Moreover, in Equation 1, the first term represents a regularization. The second one represents a negative reconstruction error.
To optimize the lower bound L(x) with respect to parameters θ, φ, we estimate gradients of Equation 1 using stochastic gradient variational Bayes (SGVB). If we consider qφ(z|x) as Gaussian distribution N (z;µ, diag(σ2)), where φ = {µ,σ2}, then we can reparameterize z ∼ qφ(z|x) to z = µ + σ ⊙ ǫ, where ǫ ∼ N (0, I). Therefore, we can estimate the gradients of the negative reconstruction term in Equation 1 with respect to θ and φ as ∇θ,φEqφ(z|x)[log pθ(x|z)] = EN (ǫ;0,I)[∇θ,φ log pθ(z|µ + σ ⊙ ǫ)]. Because the gradients of the regularization term are solvable analytically, we can optimize Equation 1 with standard stochastic optimization methods.
3.2 JOINT MULTIMODAL VARIATIONAL AUTOENCODERS
Next, we consider i.i.d. dataset (X,W) = {(x1,w1), ..., (xN ,wN )}, where two modalities x and w have different kinds of dimensions and structures2. Our objective is to generate two modalities bi-directionally. For that reason, we assume that these are conditioned independently on the same latent concept z: joint representation. Therefore, we assume their generating processes as z ∼ p(z) and x,w ∼ p(x,w|z) = pθx(x|z)pθw (w|z), where θx and θw represent the model parameters of each independent p. Figure 2(a) shows a graphical model that represents generative processes. One can see that this models joint distribution of all modalities, p(x,w). Therefore, we designate this model as a joint multimodal variational autoencoder (JMVAE).
1According to Louizos et al. (2015), this independence might not be satisfied strictly because the encoder in CVAEs still has the dependence.
2In our experiment, these depend on dataset, see Section 4.2.
Considering an approximate posterior distribution as qφ(z|x,w), we can estimate a lower bound of the log-likelihood log p(x,w) as follows:
LJM (x,w) = Eqφ(z|x,w)[log pθ(x,w, z)
qφ(z|x,w) ] (2)
= −DKL(qφ(z|x,w)||p(z))
+Eqφ(z|x,w)[log pθx(x|z)] + Eqφ(z|x,w)[log pθw(w|z)]. (3)
Equation 3 has two negative reconstruction terms which are correspondent to each modality. As with VAEs, we designate qφ(z|x,w) as the encoder and both pθx(x|z) and pθw(w|z) as decoders.
We can apply the SGVB to Equation 3 just as Equation 1, so that we can parameterize the encoder and decoder as deterministic deep neural networks and optimize them with respect to their parameters, θx, θw, and φ. Because each modality has different feature representation, we should set different networks for each decoder, pθx(x|z) and pθw(w|z). The type of distribution and corresponding network architecture depends on the representation of each modality, e.g., Gaussian when the representation of modality is continuous, and a Bernoulli when it is a binary value.
Unlike original VAEs and CVAEs, the JMVAE models joint distribution of all modalities. In this model, modalities are conditioned independently on a joint latent variable. Therefore, we can extract better representation that includes all information of modalities. Moreover, we can estimate both marginal distribution and conditional distribution in bi-directional, so that we can not only obtain images reconstructed themselves but also draw texts from corresponding images and vice versa. Additionally, we can extend JMVAEs to handle more than two modalities such as p(x,w1,w2, ...) in the same learning framework.
3.3 INFERENCE MISSING MODALITIES
In the JMVAE, we can extract joint latent features by sampling from the encoder qφ(z|x,w) at testing time. Our objective is to exchange modalities bi-directionally, e.g., images to texts and vice versa. In this setting, modalities that we want to sample are missing, so that inputs of such modalities are set to zero (the left panel of Figure 2(b)). The same is true of reconstructing a modality only from itself. This is a natural way in discriminative multimodal settings to estimate samples from unimodal information (Ngiam et al., 2011). However, if missing modalities are high-dimensional and complicated such as natural images, then the inferred latent variable becomes incomplete and generated samples might collapse.
We propose a method to solve this issue, which we designate as JMVAE-kl. Moreover, we describe the former way as JMVAE-zero to distinguish it. Suppose that we have encoders with a single input, qφx(z|x) and qφw(z|w), where φx and φw are parameters. We would like to train them by bringing their encoders close to an encoder qφ(z|x,w) (the right panel of Figure 2(b)). Therefore, the object function of JMVAE-kl becomes
LJMkl(α)(x,w) = LJM (x,w) − α · [DKL(qφ(z|x,w)||qφx (z|x)) +DKL(qφ(z|x,w)||qφw (z|w))], (4)
where α is a factor that regulates the KL divergence terms.
From another viewpoint, maximizing Equation 4 can be regarded as minimizing the variation of information with variational learning on parameterized distributions (proven and derived in Appendix A). The variation of information, a measure of the distance between two variables, is written as
−EpD(x,w)[log p(x|w) + log p(w|x)], where pD is the data distribution. It is apparent that the variation of information is the sum of two negative conditional log-likelihoods. Therefore, minimizing the variation of information contributes to appropriate bi-directional exchange of modalities. Sohn et al. (2014) also train their model to minimize the VI for the same objective as ours. However, they use DBMs with MCMC training.
4 EXPERIMENTS
This section presents evaluation of the qualitative and quantitative performance and confirms the JMVAE functionality in practice.
4.1 DATASETS
As described herein, we used two datasets: MNIST and CelebA (Liu et al., 2015).
MNIST is not a dataset for multimodal setting. In this work, we used this dataset for toy problem of multimodal learning. We consider handwriting images and corresponding digit labels as two different modalities. We used 50,000 as training set and the remaining 10,000 as a test set.
CelebA consists of 202,599 color face images and corresponding 40 binary attributes such as male, eyeglasses, and mustache. In this work, we regard them as two modalities. This dataset is challenging because these have completely different kinds of dimensions and structures. Beforehand, we cropped the images to squares and resized to 64 × 64 and normalized. From the dataset, we chose 191,899 images that are identifiable face by OpenCV and used them for our experiment. We used 90% out of all the dataset contains as training set and the remaining 10% of them as test set.
4.2 MODEL ARCHITECTURES
For MNIST, we considered images as x ∈ R28×28 and corresponding labels as w ∈ {0, 1}10. We prepared two networks each with two dense layers of 512 hidden units and using leaky rectifiers and shared the top of each layers and mapped them into 64 hidden units. Moreover, we prepared two networks each with three dense layers of 512 units and set p(x|z) as Bernoulli and p(w|z) as categorical distribution whose output layer is softmax. We used warm-up (Bowman et al., 2015; Sønderby et al., 2016), which first forces training only of the term of the negative reconstruction error and then gradually increases the effect of the regularization term to prevent local minima during early training. We increased this term linearly during the first Nt epochs as with Sønderby et al. (2016). We set Nt = 200 and trained for 500 epochs on MNIST. Moreover, same as Burda et al. (2015); Sønderby et al. (2016), we resampled the binarized training values randomly from MNIST for each epoch to prevent over-fitting.
For CelebA, we considered face images as x ∈ R32×32×3 and corresponding attributes as w ∈ {−1, 1}40. We prepared two networks with layers (four convolutional and a flattened layers for x and two dense layers forw) with ReLU and shared the top of each layers and mapped them into 128 units. For the decoder, we prepared two networks, with a dense and four deconvolutional layers for x and three dense layers for w, and set Gaussian distribution for decoder of both modalities, where the variance of Gaussian was fixed to 1 for the decoder of w. In CelebA settings, we combined JMVAE with generative adversarial networks (GANs) (Goodfellow et al., 2014) to generate clearer images. We considered the network of p(x|z) as generator in GAN, then we optimized the GAN loss with the lower bound of the JMVAE, which is the same way as a VAE-GAN model (Larsen et al., 2015). As presented herein, we describe this model as JMVAE-GAN. We set Nt = 20 and trained for 100 epochs on CelebA.
We used the Adam optimization algorithm (Kingma & Ba, 2014) with a learning rate of 10−3 on MNIST and 10−4 on CelebA. The models were implemented using Theano (Team et al., 2016) and Lasagne (Dieleman et al., 2015).
4.3 QUANTITATIVE EVALUATION
4.3.1 EVALUATION METHOD
For this experiment, we estimated test log-likelihood to evaluate the performance of model. This estimate roughly corresponds to negative reconstruction error. Therefore, higher is better. From this performance, we can find that not only whether the JMVAE can generate samples properly but also whether it can obtain joint representation properly. If the log-likelihood of a modality is low, representation for this modality might be hurt by other modalities. By contrast, if it is the same or higher than model trained on a single modality, then other modalities contribute to obtaining appropriate representation.
We estimate the test marginal log-likelihood and test conditional log-likelihood on JMVAE. We compare the test marginal log-likelihood against VAEs (Kingma & Welling, 2013; Rezende et al., 2014) and the test conditional log-likelihood against CVAEs (Kingma et al., 2014; Sohn et al., 2015) and CMMAs (Pandey & Dukkipati, 2016). On CelebA, we combine all competitive models with GAN and describe them as VAE-GAN, CVAE-GAN, and CMMA-GAN. For fairness, architectures and parameters of these competitive models were set to be as close as possible to those of JMVAE.
We calculate the importance weighted estimator (Burda et al., 2015) from lower bounds at testing time because we would like to estimate the true test log-likelihood from lower bounds. To estimate the test marginal log-likelihood p(w) of the JMVAE, we use two possible lower bounds: sampling from qφ(z|x,w) or qφx(z|x). We describe the former lower bound as the multiple lower bound and the latter one as the single lower bound. When we estimate the test conditional loglikelihood log p(x|w), we also use two lower bounds, each of which is estimated by sampling from qφ(z|x,w) (multiple) or qφw(z|w) (single) (see Appendix B for more details). To estimate the single lower bound, we should approximate the single encoder (qφx(z|x) or qφw(z|w)) by JMVAE-zero or JMVAE-kl. When the value of log-likelihood with the single lower bound is the same or larger than that with the multiple lower bound, the approximation of the single encoder is good. Note that original VAEs use a single lower bound and that CVAEs and CMMAs use a multiple lower bound.
4.3.2 MNIST
Our first experiment evaluated the test marginal log-likelihood and compared it with that of the VAE on MNIST dataset. We trained the model with both JMVAE-zero and JMVAE-kl and confirmed these differences. As described in Section 4.3.1, we have two possible ways of estimating the marginal log-likelihood of the JMVAE, i.e., multiple and single lower bounds. The left of Table 1 shows the test marginal log-likelihoods of the VAE and JMVAE. It is apparent that log-likelihood of the JMVAE-zero is the same or slightly better than that of the VAE. In the case of the loglikelihood of JMVAE-kl, the log-likelihood becomes better as α is small. Especially, JMVAE-kl with α = 0.01 and single lower bound archives the highest log-likelihood in Table 1. If α is 1, however, then the test log-likelihood on JMVAE-kl becomes much lower. This is because the influence of the regularization term becomes strong as α is large.
Next, we evaluated the test conditional log-likelihood and compared it with that of the CVAE and CMMA conditioned on w. As in the case of the marginal log-likelihood, we can estimate the JMVAE’s conditional log-likelihood by both the single and multiple lower bound. The single bound can be estimated using JMVAE-zero or JMVAE-kl. The right of Table 1 shows the test conditional log-likelihoods of the JMVAE, CVAE, and CMMA. It is apparent that the CVAE achieves the highest log-likelihood. Even so, in the case of multiple bound, log-likelihoods with both JMVAE-zero and JMVAE-kl (except α = 1) outperform that of the CMMA.
It should be noted that the log-likelihood with JMVAE-zero and single bound is significantly low. As described in Section 3.3, this is because a modality w is missing as input. By contrast, it is apparent that the log-likelihood with JMVAE-kl is improved significantly from that with JMVAEzero. It shows that JMVAE-kl solves the issue of missing modalities (we can also find this result in generated images, see Appendix E). Moreover, we find that this log-likelihood becomes better as α is large, which is opposite to the other results. Therefore, there is a trade-off between whether each modality can be reconstructed properly and whether multiple modalities can be exchanged properly and it can be regulated by α.
4.3.3 CELEBA
In this section, we used CelebA dataset to evaluate the JMVAE. Table 2 presents the evaluations of marginal and conditional log-likelihood. From this table, it is apparent that values of both marginal and conditional log-likelihood with JMVAEs are larger than those with other competitive methods. Moreover, comparison with Table 1 shows that the improvement on CelebA is greater than that on MNIST, which suggests that joint representation with multiple modalities contributes to improvement of the quality of the reconstruction and generation in the case in which an input modality is large-dimensioned and complicated.
4.4 QUALITATIVE EVALUATION
4.4.1 JOINT REPRESENTATION ON MNIST
In this section, we first evaluated that the JMVAE can obtain joint representation that includes the information of modalities. Figure 3 shows the visualization of latent representation with the VAE, CVAE, and JMVAE on MNIST. It is apparent that the JMVAE obtains more discriminable latent representation by adding digit label information. Figure 3(b) shows that, in spite of using multimodal information as with the JMVAE, points in CVAE are distributed irrespective of labels because CVAEs force latent representation to be independent of label information, i.e., it is not objective for CVAEs to obtain joint representation.
4.4.2 GENERATING FACES FROM ATTRIBUTES AND JOINT REPRESENTATION ON CELEBA
Next, we confirm that JMVAE-GAN on CelebA can generate images from attributes. Figure 4(a) portrays generated faces conditioned on various attributes. We find that we can generate an average face of each attribute and various random faces conditioned on a certain attributes. Figure 4(b) shows that samples are gathered for each attribute and that locations of each variation are the same irrespective of attributes. From these results, we find that manifold learning of joint representation with images and attributes works well.
4.4.3 BI-DIRECTIONAL GENERATION BETWEEN FACES AND ATTRIBUTES ON CELEBA
Finally, we demonstrate that JMVAE-GAN can generate bi-directionally between faces and attributes. Figure 5 shows that MVAE-GAN can generate both attributes and changed images conditioned on various attributes from images which had no attribute information. This way of generating an image by varying attributes is similar to the way of the CMMA (Pandey & Dukkipati, 2016). However, the CMMA cannot generate attributes from an image because it only generates images from attributes in one direction.
5 CONCLUSION AND FUTURE WORK
In this paper, we introduced a novel multimodal learning model with VAEs, the joint multimodal variational autoencoders (JMVAE). In this model, modalities are conditioned independently on joint representation, i.e., it models a joint distribution of all modalities. We further proposed the method (JMVAE-kl) of reducing the divergence between JMVAE’s encoder and a prepared encoder of each modality to prevent generated samples from collapsing when modalities are missing. We confirmed that the JMVAE can obtain appropriate joint representations and high log-likelihoods on MNIST
4https://en.wikipedia.org/wiki/Mona Lisa 4https://en.wikipedia.org/wiki/Wolfgang Amadeus Mozart
and CelebA datasets. Moreover, we demonstrated that the JMVAE can generate multiple modalities bi-directionally on the CelebA dataset.
In future work, we would like to evaluate the multimodal learning performance of JMVAEs using various multimodal datasets such as containing three or more modalities.
A RELATION BETWEEN THE OBJECTIVE OF JMVAE-KL AND THE VARIATION
OF INFORMATION
The variation of information can be expressed as −EpD(x,w)[log p(x|w) + log p(w|x)], where pD is the data distribution. In this equation, we specifically examine the sum of two negative loglikelihoods and do not consider the expectation in this derivation. We can calculate the lower bounds of these log-likelihoods as follows:
log p(x|w) + log p(w|x) ≥ Eq(z|x,w)[log p(x|z)p(z|w)
q(z|x,w) ] + Eq(z|x,w)[log
p(w|z)p(z|x)
q(z|x,w) ]
= Eq(z|x,w)[log p(x|z)] + Eq(z|x,w)[log p(w|z)]
−DKL(q(z|x,w)||p(z|x)) −DKL(q(z|x,w)||p(z|w))
= LJM (x,w)− [DKL(q(z|x,w)||p(z|x)) +DKL(q(z|x,w)||p(z|w))]
+DKL(q(z|x,w)||p(z)). (5)
If all the probability distributions are parameterized in neural networks, we can consider each p(z|x) and p(z|w) as q(z|x) and q(z|w), respectively. This is because both p(z|x) and q(z|x) (or both p(z|w) and q(z|w)) can be expressed as same network architectures. Therefore, the replaced Equation 5 can be calculated as follows:
LJM (x,w) − [DKL(q(z|x,w)||q(z|x)) +DKL(q(z|x,w)||q(z|w))] +DKL(q(z|x,w)||p(z))
= LJMkl(1)(x,w) +DKL(q(z|x,w)||p(z)) ≥ LJMkl(1)(x,w), (6)
where LJMkl(1) is Equation 4 with α = 1. Therefore, maximizing Equation 4 is regarded as minimizing the variation of information with variational learning on parameterized distributions, i.e., maximizing the lower bounds of the negative variation of information.
B TEST LOWER BOUNDS
Two lower bounds used to estimate test marginal log-likelihood p(x) of the JMVAE are as follows:
Lsingle(x) = Eqφx (z|x)[log pθx(x|z)p(z)
qφx(z|x) ], (7)
Lmultiple(x) = Eqφ(z|x,w)[log pθx(x|z)p(z)
qφ(z|x,w) ]. (8)
We can also estimate test conditional log-likelihood p(x|w) from these two lower bounds as
Lsingle(x|w) = Eqφw (z|w)[log p(x, z|w)
qφw (z|w) ] = Eqφw (z|w)[log
pθx(x|z)pθw (w|z)p(z)
qφw(z|w) ]− log p(w), (9)
Lmultiple(x|w) = Eqφ(z|x,w)[log pθx(x|z)pθw (w|z)p(z)
qφ(z|x,w) ]− log p(w), (10)
where log p(w) = logEp(z)[pθw(w|z)] = log 1
Nw
∑Nw i pθw(w|z
(i)) and z(i) ∼ p(z). In this paper, we set Nw = 5, 000 on MNIST and Nw = 10 on CelebA.
We can obtain a tighter bound on the log-likelihood by k-fold importance weighted sampling. For example, we obtain an importance weighted bound on log p(x) from Equation 11 as follows:
log p(x) ≥ Ez1,...,zk∼qφx (z|x)[log 1
k
k∑
i=1
pθx(x|z)p(z)
qφx(z|x) ] = Lksingle(x). (11)
Strictly speaking, these two lower bounds are not equal. However, if the number of importance samples is extremely large, the difference of these two lower bounds converges to 0.
Proof. Let the multiple and single k-hold importanceweighted lower bounds asLksingle andL k single. From the theorem of the importance weighted bound, both Lksingle and L k multiple converge to log p(x) as k → ∞.
Therefore,
limk→∞|L k multiple − L k single| ≤ |limk→∞L k multiple − limk→∞L k single| = 0
C RECONSTRUCTED IMAGES
Figure 6 presents a comparison of the original image and reconstructed image by the JMVAE on both MNIST and CelebA datasets. It is apparent that the JMVAE can reconstruct the original image properly with either a multiple or single encoder.
D TEST JOINT LOG-LIKELIHOOD ON MNIST
Table 3 shows the joint log-likelihood of the JMVAE on MNIST dataset by both JMVAE-zero and JMVAE-kl. It is apparent that the log-likelihood test on both approaches is almost identical (strictly, JMVAE-zero is slightly lower). The test log-likelihood on JMVAE-kl becomes much lower if α is large.
E IMAGE GENERATION FROM CONDITIONAL DISTRIBUTION ON MNIST
Figure 7 presents generation samples of x conditioned on single input w. It is apparent that the JMVAE with JMVAE-kl generates conditioned digit images properly, although that with JMVAEzero cannot generate them. As results showed, we also confirmed qualitatively that JMVAE-kl can model qφx(z|x) properly compared to JMVAE-zero.

This paper proposes an alternative to Conditional Variational Auto-Encoders and Conditional MultiModal Auto-Encoders to perform inference of missing modalities in dataset with multiple modalities. The proposed approach is a Variational Auto-Encoder jointly on all the modalities  with additional KL divergence penalties between the approximate posterior given all the modalities and the approximate posterior given a subset of the modalities. The approach is named Joint Multimodal Variational Auto-Encoder.
The authors make a connection between this approach and the Variation of Information. It is unclear why the authors chose the JMVAE approach instead of a more elegant Variation of Information approach.
Another unaddressed issue is the scalability of the method. As far as I can tell (given that no code is provided and the specification of the encoder is missing), this approach requires a new encoder per subset of missing modalities. Right now this approach seems to scale since there are only two modalities.
The fact that the estimating the log-likelihood log(p(x)) using multiple modalities provide a lower value than with just one in Table 1 is a bit odd. Could you explain that ?
The comparison with between the representation learned by JMVAE and CVAE might be unfair given that the representation of CVAE is learned conditionally, on the label in the case of MNIST, and should therefore not consider the label in this representation. Intuitively, this representation could represent "style" as shown in (Kingma et al., 2014) in their conditional generation figure.
For CelebA, comparing log-likelihood on models that use GANs is probably not significant since GAN does not optimizes log-likelihood.
Overall this is an interesting problem and there are also interesting ideas worth exploring further, but the execution of the paper requires more work.