TRAINED TERNARY QUANTIZATION
Deep neural networks are widely used in machine learning applications. However, the deployment of large neural networks models can be difficult to deploy on mobile devices with limited power budgets. To solve this problem, we propose Trained Ternary Quantization (TTQ), a method that can reduce the precision of weights in neural networks to ternary values. This method has very little accuracy degradation and can even improve the accuracy of some models (32, 44, 56-layer ResNet) on CIFAR-10 and AlexNet on ImageNet. And our AlexNet model is trained from scratch, which means it’s as easy as to train normal full precision model. We highlight our trained quantization method that can learn both ternary values and ternary assignment. During inference, only ternary values (2-bit weights) and scaling factors are needed, therefore our models are nearly 16× smaller than fullprecision models. Our ternary models can also be viewed as sparse binary weight networks, which can potentially be accelerated with custom circuit. Experiments on CIFAR-10 show that the ternary models obtained by trained quantization method outperform full-precision models of ResNet-32,44,56 by 0.04%, 0.16%, 0.36%, respectively. On ImageNet, our model outperforms full-precision AlexNet model by 0.3% of Top-1 accuracy and outperforms previous ternary models by 3%.
1 INTRODUCTION
Deep neural networks are becoming the preferred approach for many machine learning applications. However, as networks get deeper, deploying a network with a large number of parameters on a small device becomes increasingly difficult. Much work has been done to reduce the size of networks. Halfprecision networks (Amodei et al., 2015) cut sizes of neural networks in half. XNOR-Net (Rastegari et al., 2016), DoReFa-Net (Zhou et al., 2016) and network binarization (Courbariaux et al.; 2015; Lin et al., 2015) use aggressively quantized weights, activations and gradients to further reduce computation during training. While weight binarization benefits from 32× smaller model size, the extreme compression rate comes with a loss of accuracy. Hubara et al. (2016) and Li & Liu (2016) propose ternary weight networks to trade off between model size and accuracy.
In this paper, we propose Trained Ternary Quantization which uses two full-precision scaling coefficients W pl , W n l for each layer l, and quantize the weights to {−Wnl , 0, +W p l } instead of traditional {-1, 0, +1} or {-E, 0, +E} where E is the mean of the absolute weight value, which is not learned. Our positive and negative weights have different absolute values W pl and W n l that are trainable parameters. We also maintain latent full-precision weights at training time, and discard them at test time. We back propagate the gradient to both W pl , W n l and to the latent full-precision weights. This makes it possible to adjust the ternary assignment (i.e. which of the three values a weight is assigned).
Our quantization method, achieves higher accuracy on the CIFAR-10 and ImageNet datasets. For AlexNet on ImageNet dataset, our method outperforms previously state-of-art ternary network(Li &
∗Work done while at Stanford CVA lab.
Liu, 2016) by 3.0% of Top-1 accuracy and the full-precision model by 1.6%. By converting most of the parameters to 2-bit values, we also compress the network by about 16x. Moreover, the advantage of few multiplications still remains, because W pl and W n l are fixed for each layer during inference. On custom hardware, multiplications can be pre-computed on activations, so only two multiplications per activation are required.
2 MOTIVATIONS
The potential of deep neural networks, once deployed to mobile devices, has the advantage of lower latency, no reliance on the network, and better user privacy. However, energy efficiency becomes the bottleneck for deploying deep neural networks on mobile devices because mobile devices are battery constrained. Current deep neural network models consist of hundreds of millions of parameters. Reducing the size of a DNN model makes the deployment on edge devices easier.
First, a smaller model means less overhead when exporting models to clients. Take autonomous driving for example; Tesla periodically copies new models from their servers to customers’ cars. Smaller models require less communication in such over-the-air updates, making frequent updates more feasible. Another example is on Apple Store; apps above 100 MB will not download until you connect to Wi-Fi. It’s infeasible to put a large DNN model in an app. The second issue is energy consumption. Deep learning is energy consuming, which is problematic for battery-constrained mobile devices. As a result, iOS 10 requires iPhone to be plugged with charger while performing photo analysis. Fetching DNN models from memory takes more than two orders of magnitude more energy than arithmetic operations. Smaller neural networks require less memory bandwidth to fetch the model, saving the energy and extending battery life. The third issue is area cost. When deploying DNNs on Application-Specific Integrated Circuits (ASICs), a sufficiently small model can be stored directly on-chip, and smaller models enable a smaller ASIC die.
Several previous works aimed to improve energy and spatial efficiency of deep networks. One common strategy proven useful is to quantize 32-bit weights to one or two bits, which greatly reduces model size and saves memory reference. However, experimental results show that compressed weights usually come with degraded performance, which is a great loss for some performancesensitive applications. The contradiction between compression and performance motivates us to work on trained ternary quantization, minimizing performance degradation of deep neural networks while saving as much energy and space as possible.
3 RELATED WORK
3.1 BINARY NEURAL NETWORK (BNN)
Lin et al. (2015) proposed binary and ternary connections to compress neural networks and speed up computation during inference. They used similar probabilistic methods to convert 32-bit weights into binary values or ternary values, defined as:
wb ∼ Bernoulli( w̃ + 1 2 )× 2− 1
wt ∼ Bernoulli(|w̃|)× sign(w̃) (1)
Here wb and wt denote binary and ternary weights after quantization. w̃ denotes the latent full precision weight.
During back-propagation, as the above quantization equations are not differentiable, derivatives of expectations of the Bernoulli distribution are computed instead, yielding the identity function:
∂L ∂w̃ = ∂L ∂wb = ∂L ∂wt (2)
Here L is the loss to optimize.
For BNN with binary connections, only quantized binary values are needed for inference. Therefore a 32× smaller model can be deployed into applications.
3.2 DOREFA-NET
Zhou et al. (2016) proposed DoReFa-Net which quantizes weights, activations and gradients of neural networks using different widths of bits. Therefore with specifically designed low-bit multiplication algorithm or hardware, both training and inference stages can be accelerated.
They also introduced a much simpler method to quantize 32-bit weights to binary values, defined as:
wb = E(|w̃|)× sign(w̃) (3)
Here E(|w̃|) calculates the mean of absolute values of full precision weights w̃ as layer-wise scaling factors. During back-propagation, Equation 2 still applies.
3.3 TERNARY WEIGHT NETWORKS
Li & Liu (2016) proposed TWN (Ternary weight networks), which reduce accuracy loss of binary networks by introducing zero as a third quantized value. They use two symmetric thresholds ±∆l and a scaling factor Wl for each layer l to quantize weighs into {−Wl, 0,+Wl}:
wtl =  Wl : w̃l > ∆l
0 : |w̃l| ≤ ∆l −Wl : w̃l < −∆l
(4)
They then solve an optimization problem of minimizing L2 distance between full precision and ternary weights to obtain layer-wise values of Wl and ∆l:
∆l = 0.7×E(|w̃l|) Wl = E
i∈{i|w̃l(i)|>∆} (|w̃l(i)|) (5)
And again Equation 2 is used to calculate gradients. While an additional bit is required for ternary weights, TWN achieves a validation accuracy that is very close to full precision networks according to their paper.
3.4 DEEP COMPRESSION
Han et al. (2015) proposed deep compression to prune away trivial connections and reduce precision of weights. Unlike above models using zero or symmetric thresholds to quantize high precision weights, Deep Compression used clusters to categorize weights into groups. In Deep Compression, low precision weights are fine-tuned from a pre-trained full precision network, and the assignment of each weight is established at the beginning and stay unchanged, while representative value of each cluster is updated throughout fine-tuning.
4 METHOD
Our method is illustrated in Figure 1. First, we normalize the full-precision weights to the range [-1, +1] by dividing each weight by the maximum weight. Next, we quantize the intermediate full-resolution weights to {-1, 0, +1} by thresholding. The threshold factor t is a hyper-parameter that is the same across all the layers in order to reduce the search space. Finally, we perform trained quantization by back propagating two gradients, as shown in the dashed lines in Figure 1. We back-propagate gradient1 to the full-resolution weights and gradient2 to the scaling coefficients. The former enables learning the ternary assignments, and the latter enables learning the ternary values.
At inference time, we throw away the full-resolution weights and only use ternary weights.
4.1 LEARNING BOTH TERNARY VALUES AND TERNARY ASSIGNMENTS
During gradient descent we learn both the quantized ternary weights (the codebook), and choose which of these values is assigned to each weight (choosing the codebook index).
To learn the ternary value (codebook), we introduce two quantization factors W pl and W n l for positive and negative weights in each layer l. During feed-forward, quantized ternary weights wtl are calculated as:
wtl =  W pl : w̃l > ∆l
0 : |w̃l| ≤ ∆l −Wnl : w̃l < −∆l
(6)
Unlike previous work where quantized weights are calculated from 32-bit weights, the scaling coefficients W pl and W n l are two independent parameters and are trained together with other parameters. Following the rule of gradient descent, derivatives of W pl and W n l are calculated as:
∂L
∂W pl = ∑ i∈Ipl ∂L ∂wtl (i) , ∂L ∂Wnl = ∑ i∈Inl ∂L ∂wtl (i) (7)
Here Ipl = {i|w̃l(i) > ∆l} and Inl = {i|(i)w̃l < −∆l}. Furthermore, because of the existence of two scaling factors, gradients of latent full precision weights can no longer be calculated by Equation 2. We use scaled gradients for 32-bit weights:
∂L ∂w̃l =  W pl × ∂L ∂wtl : w̃l > ∆l 1× ∂L ∂wtl : |w̃l| ≤ ∆l Wnl × ∂L
∂wtl : w̃l < −∆l
(8)
Note we use scalar number 1 as factor of gradients of zero weights. The overall quantization process is illustrated as Figure 1. The evolution of the ternary weights from different layers during training is shown in Figure 2. We observe that as training proceeds, different layers behave differently: for the first quantized conv layer, the absolute values of W pl and W n l get smaller and sparsity gets lower, while for the last conv layer and fully connected layer, the absolute values of W pl and W n l get larger and sparsity gets higher.
We learn the ternary assignments (index to the codebook) by updating the latent full-resolution weights during training. This may cause the assignments to change between iterations. Note that the thresholds are not constants as the maximal absolute values change over time. Once an updated weight crosses the threshold, the ternary assignment is changed.
The benefits of using trained quantization factors are: i) The asymmetry of W pl 6= Wnl enables neural networks to have more model capacity. ii) Quantized weights play the role of "learning rate multipliers" during back propagation.
4.2 QUANTIZATION HEURISTIC
In previous work on ternary weight networks, Li & Liu (2016) proposed Ternary Weight Networks (TWN) using ±∆l as thresholds to reduce 32-bit weights to ternary values, where ±∆l is defined as Equation 5. They optimized value of ±∆l by minimizing expectation of L2 distance between full precision weights and ternary weights. Instead of using a strictly optimized threshold, we adopt
different heuristics: 1) use the maximum absolute value of the weights as a reference to the layer’s threshold and maintain a constant factor t for all layers:
∆l = t×max(|w̃|) (9)
and 2) maintain a constant sparsity r for all layers throughout training. By adjusting the hyperparameter r we are able to obtain ternary weight networks with various sparsities. We use the first method and set t to 0.05 in experiments on CIFAR-10 and ImageNet dataset and use the second one to explore a wider range of sparsities in section 5.1.1.
We perform our experiments on CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (Russakovsky et al., 2015). Our network is implemented on both TensorFlow (Abadi & et. al o, 2015) and Caffe (Jia et al., 2014) frameworks.
4.3 CIFAR-10
5 EXPERIMENTS
CIFAR-10 is an image classification benchmark containing images of size 32×32RGB pixels in a training set of 50000 and a test set of 10000. ResNet (He et al., 2015) structure is used for our experiments.
We use parameters pre-trained from a full precision ResNet to initialize our model. Learning rate is set to 0.1 at beginning and scaled by 0.1 at epoch 80, 120 and 300. A L2-normalized weight decay
177 0.08179999887943270.100699998438358 0.089199997484684
178 0.08139999955892560.128600001335144 0.0901999995112419
179 0.08060000091791150.110799998044968 0.086499996483326
Full precision Binary weight (DoReFa-Net) Ternary weight (Ours)
5
of 0.0002 is used as regularizer. Most of our models converge after 160 epochs. We take a moving average on errors of all epochs to filter off fluctuations when reporting error rate.
We compare our model with the full-precision model and a binary-weight model. We train a a full precision ResNet (He et al., 2016) on CIFAR-10 as the baseline (blue line in Figure 3). We fine-tune the trained baseline network as a 1-32-32 DoReFa-Net where weights are 1 bit and both activations and gradients are 32 bits giving a significant loss of accuracy (green line) . Finally, we fine-tuning the baseline with trained ternary weights (red line). Our model has substantial accuracy improvement over the binary weight model, and our loss of accuracy over the full precision model is small. We also compare our model to Tenary Weight Network (TWN) on ResNet-20. Result shows our model improves the accuracy by ∼ 0.25% on CIFAR-10. We expand our experiments to ternarize ResNet with 32, 44 and 56 layers. All ternary models are fine-tuned from full precision models. Our results show that we improve the accuracy of ResNet-32, ResNet-44 and ResNet-56 by 0.04%, 0.16% and 0.36% . The deeper the model, the larger the improvement. We conjecture that this is due to ternary weights providing the right model capacity and preventing overfitting for deeper networks.
5.1 IMAGENET
We further train and evaluate our model on ILSVRC12(Russakovsky et al. (2015)). ILSVRC12 is a 1000-category dataset with over 1.2 million images in training set and 50 thousand images in validation set. Images from ILSVRC12 also have various resolutions. We used a variant of AlexNet(Krizhevsky et al. (2012)) structure by removing dropout layers and add batch normalization(Ioffe & Szegedy, 2015) for all models in our experiments. The same variant is also used in experiments described in the paper of DoReFa-Net.
Our ternary model of AlexNet uses full precision weights for the first convolution layer and the last fully-connected layer. Other layer parameters are all quantized to ternary values. We train our model on ImageNet from scratch using an Adam optimizer (Kingma & Ba (2014)). Minibatch size is set to 128. Learning rate starts at 10−4 and is scaled by 0.2 at epoch 56 and 64. A L2-normalized weight decay of 5 × 10−6 is used as a regularizer. Images are first resized to 256 × 256 then randomly cropped to 224× 224 before input. We report both top 1 and top 5 error rate on validation set. We compare our model to a full precision baseline, 1-32-32 DoReFa-Net and TWN. After around 64 epochs, validation error of our model dropped significantly compared to other low-bit networks as well as the full precision baseline. Finally our model reaches top 1 error rate of 42.5%, while DoReFa-Net gets 46.1% and TWN gets 45.5%. Furthermore, our model still outperforms full precision AlexNet (the batch normalization version, 44.1% according to paper of DoReFa-Net) by 1.6%, and is even better than the best AlexNet results reported (42.8%1). The complete results are listed in Table 2.
1https://github.com/BVLC/caffe/wiki/Models-accuracy-on-ImageNet-2012-val
1
We draw the process of training in Figure 4, the baseline results of AlexNet are marked with dashed lines. Our ternary model effectively reduces the gap between training and validation performance, which appears to be quite great for DoReFa-Net and TWN. This indicates that adopting trainable W pl and Wnl helps prevent models from overfitting to the training set.
We also report the results of our methods on ResNet-18B in Table 3. The full-precision error rates are obtained from Facebook’s implementation. Here we cite Binarized Weight Network(BWN)Rastegari et al. (2016) results with all layers quantized and TWN finetuned based on a full precision network, while we train our TTQ model from scratch. Compared with BWN and TWN, our method obtains a substantial improvement.
6 DISCUSSION
In this section we analyze performance of our model with regard to weight compression and inference speeding up. These two goals are achieved through reducing bit precision and introducing sparsity. We also visualize convolution kernels in quantized convolution layers to find that basic patterns of edge/corner detectors are also well learned from scratch even precision is low.
6.1 SPATIAL AND ENERGY EFFICIENCY
We save storage for models by 16× by using ternary weights. Although switching from a binaryweight network to a ternary-weight network increases bits per weight, it brings sparsity to the weights, which gives potential to skip the computation on zero weights and achieve higher energy efficiency.
6.1.1 TRADE-OFF BETWEEN SPARSITY AND ACCURACY
Figure 5 shows the relationship between sparsity and accuracy. As the sparsity of weights grows from 0 (a pure binary-weight network) to 0.5 (a ternary network with 50% zeros), both the training and validation error decrease. Increasing sparsity beyond 50% reduces the model capacity too far, increasing error. Minimum error occurs with sparsity between 30% and 50%.
We introduce only one hyper-parameter to reduce search space. This hyper-parameter can be either sparsity, or the threshold t w.r.t the max value in Equation 6. We find that using threshold produces better results. This is because fixing the threshold allows the sparsity of each layer to vary (Figure reffig:weights).
Published as a conference paper at ICLR 2017
Table 1
Table 2 w/o pruning 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Validation Error 0.105744999647141 0.09665999934077260.09309500008821490.0913449995219708 0.09231499992311 0.091499999538064 0.09421999976038930.099015000090003 0.113474999368191 0.150045001879334 Train Error 0.03142703464254740.008692236384376880.007409938820637760.007549929246306420.007867254689335820.009407590329647060.0125876291189343 0.0196611762978137 0.03617928531020880.0824047848582268
6.1.2 SPARSITY AND EFFICIENCY OF ALEXNET
We further analyze parameters from our AlexNet model. We calculate layer-wise density (complement of sparsity) as shown in Table 4. Despite we use different W pl and W n l for each layer, ternary weights can be pre-computed when fetched from memory, thus multiplications during convolution and inner product process are still saved. Compared to Deep Compression, we accelerate inference speed using ternary values and more importantly, we reduce energy consumption of inference by saving memory references and multiplications, while achieving higher accuracy.
We notice that without all quantized layers sharing the same t for Equation 9, our model achieves considerable sparsity in convolution layers where the majority of computations takes place. Therefore we are able to squeeze forward time to less than 30% of full precision networks.
As for spatial compression, by substituting 32-bit weights with 2-bit ternary weights, our model is approximately 16× smaller than original 32-bit AlexNet.
6.2 KERNEL VISUALIZATION
We visualize quantized convolution kernels in Figure 6. The left matrix is kernels from the second convolution layer (5× 5) and the right one is from the third (3× 3). We pick first 10 input channels and first 10 output channels to display for each layer. Grey, black and white color represent zero, negative and positive weights respectively.
We observe similar filter patterns as full precision AlexNet. Edge and corner detectors of various directions can be found among listed kernels. While these patterns are important for convolution neural networks, the precision of each weight is not. Ternary value filters are capable enough extracting key features after a full precision first convolution layer while saving unnecessary storage.
Furthermore, we find that there are a number of empty filters (all zeros) or filters with single non-zero value in convolution layers. More aggressive pruning can be applied to prune away these redundant kernels to further compress and speed up our model.
Conv2 Conv3
7 CONCLUSION
We introduce a novel neural network quantization method that compresses network weights to ternary values. We introduce two trained scaling coefficients W lp and W l n for each layer and train these coefficients using back-propagation. During training, the gradients are back-propagated both to the latent full-resolution weights and to the scaling coefficients. We use layer-wise thresholds that are proportional to the maximum absolute values to quantize the weights. When deploying the ternary network, only the ternary weights and scaling coefficients are needed, which reducing parameter size by at least 16×. Experiments show that our model reaches or even surpasses the accuracy of full precision models on both CIFAR-10 and ImageNet dataset. On ImageNet we exceed the accuracy of prior ternary networks (TWN) by 3%.
9

This work presents a novel ternary weight quantization approach which quantizes weights to either 0 or one of two layer specific learned values. Unlike past work, these quantized values are separate and learned stochastically alongside all other network parameters. This approach achieves impressive quantization results while retaining or surpassing corresponding full-precision networks on CIFAR10 and ImageNet.
Strengths:
- Overall well written and algorithm is presented clearly.
- Approach appears to work well in the experiments, resulting in good compression without loss (and sometimes gain!) of performance.
- I enjoyed the analysis of sparsity (and how it changes) over the course of training, though it is uncertain if any useful conclusion can be drawn from it.
Some points:
- The energy analysis in Table 3 assumes dense activations due to the unpredictability of sparse activations. Can the authors provide average activation sparsity for each network to help verify this assumption. Even if the assumption does not hold, relatively close values for average activation between the networks would make the comparison more convincing.
- In section 5.1.1, the authors suggest having a fixed t (threshold parameter set at 0.05) for all layers allows for varying sparsity (owed to the relative magnitude of different layer weights with respect to the maximum). In Section 5.1.2 paragraph 2, this is further developed by suggesting additional sparsity can be achieved by allowing each layer a different values of t. How are these values set? Does this multiple threshold style network appear in any of the tables or figures? Can it be added?
- The authors claim "ii) Quantized weights play the role of "learning rate multipliers" during back propagation." as a benefit of using trained quantization factors. Why is this a benefit?
- Figure and table captions are not very descriptive.
Preliminary Rating:
I think this is an interesting paper with convincing results but is somewhat lacking in novelty.
Minor notes:
- Table 3 lists FLOPS rather than Energy for the full precision model. Why?
- Section 5 'speeding up'
- 5.1.1 figure reference error last line


DEEP VARIATIONAL CANONICAL CORRELATION ANALYSIS
We present deep variational canonical correlation analysis (VCCA), a deep multiview learning model that extends the latent variable model interpretation of linear CCA (Bach and Jordan, 2005) to nonlinear observation models parameterized by deep neural networks (DNNs). Computing the marginal data likelihood, as well as inference of the latent variables, are intractable under this model. We derive a variational lower bound of the data likelihood by parameterizing the posterior density of the latent variables with another DNN, and approximate the lower bound via Monte Carlo sampling. Interestingly, the resulting model resembles that of multiview autoencoders (Ngiam et al., 2011), with the key distinction of an additional sampling procedure at the bottleneck layer. We also propose a variant of VCCA called VCCA-private which can, in addition to the “common variables” underlying both views, extract the “private variables” within each view. We demonstrate that VCCA-private is able to disentangle the shared and private information for multi-view data without hard supervision.
1 INTRODUCTION
In the multi-view representation learning setting, we have multiple views/measurements of the same underlying signal, and the goal is to learn useful features of each view using complementary information contained in the views. The intuition underlying this setting is that the learned features can help uncover the common sources of variation in the views, which can be helpful for exploratory analysis or for downstream tasks.
A classical approach in this setting is canonical correlation analysis (CCA, Hotelling, 1936) and its nonlinear extensions, including the kernel extension (Lai and Fyfe, 2000; Akaho, 2001; Melzer et al., 2001; Bach and Jordan, 2002) and the deep neural network (DNN) extension (Andrew et al., 2013; Wang et al., 2015b). CCA projects two random vectors x ∈ Rdx and y ∈ Rdy into a lowerdimensional subspace so that the projections are maximally correlated. There is a probabilistic latent variable model interpretation of linear CCA (Bach and Jordan, 2005) as shown in Figure 1 (left). Assume that x and y are linear functions of some lower-dimensional random variable z ∈ Rdz , where dz ≤ min(dx, dy). When the prior distribution of the latent variable p(z) and the conditional distributions p(x|z) and p(y|z) are Gaussian, Bach and Jordan (2005) showed that E[z|x] (resp. E[z|y]) lives in the same space as the linear CCA projection for x (resp. y). This generative interpretation of CCA is often lost in nonlinear extensions of CCA. For example, in deep CCA (DCCA, (Andrew et al., 2013)), to extend CCA to nonlinear mappings with greater representation power, one extracts nonlinear features from the original inputs of each view using two DNNs, f for x and g for y, so that the canonical correlation of the DNN outputs (measured by a linear CCA with projection matrices U and V) is maximized. Formally, given a dataset of N pairs of observations (x1,y1), . . . , (xN ,yN ) of the random vectors (x,y), DCCA optimizes
max Wf ,Wg
U,V
tr ( U>f(X)g(Y)>V ) s.t. U> ( f(X)f(X)> ) U = V> ( g(Y)g(Y)> ) V = NI, (1)
where f(X) = [f(x1), . . . , f(xN )] and g(Y) = [g(y1), . . . ,g(yN )], and Wf denotes all weight parameters of the DNN f (and similarly for g).
DCCA has achieved good performance in the multi-view representation learning setting across different domains (Wang et al., 2015b,a; Lu et al., 2015; Yan and Mikolajczyk, 2015). However, a disadvantage of DCCA is that it directly looks for DNNs that can map inputs into the low-dimensional space, without a model for generating samples from the latent space. Although Wang et al. (2015b)’s deep canonically correlated autoencoders (DCCAE) model optimizes the combination of the autoencoder objective (reconstruction errors) and the canonical correlation objective, the authors found that in practice, the canonical correlation term tends to dominate the reconstruction error terms in the DCCAE objective when tuning performance for a downstream task (especially when the inputs are noisy), and as a result the inputs are not reconstructed well. At the same time, optimization of the DCCA and DCCAE objectives is challenging due to the constraints that couple all training samples.
The main contribution of this paper is the proposal of a new deep multi-view learning model named deep variational CCA (VCCA), which extends the latent variable model interpretation of linear CCA to nonlinear observation models parameterized by DNNs. Computing the marginal data likelihood, as well as inference of the latent variables, are intractable under this model. Inspired by variational autoencoders (VAE, Kingma and Welling, 2014), we parameterize the posterior distribution of the latent variables with another DNN, and derive a variational lower bound of the data likelihood as the objective of VCCA, which is further approximated by Monte Carlo sampling. With the reparameterization trick, sampling for the Monte Carlo approximation is trivial and all DNN weights in VCCA can be optimized jointly via stochastic gradient descent, using unbiased gradient estimates from small minibatches. Interestingly, VCCA is related to multi-view autoencoders (Ngiam et al., 2011), with the key distinctions of additional regularization on the posterior distribution and the sampling procedure at the bottleneck layer.
We also propose a variant of VCCA called VCCA-private that can, in addition to the “common variables” underlying both views, extract the “private variables” within each view. We demonstrate that VCCA-private is able to disentangle the shared and private information for multi-view data without hard supervision. Last but not least, as generative models, VCCA and VCCA-private enable us to obtain high-quality samples for the input of each view.
2 VARIATIONAL CCA
The probabilistic latent variable model of CCA (Bach and Jordan, 2005) defines the following joint distribution over the random variables (x,y):
p(x,y, z) = p(z)p(x|z)p(y|z), p(x,y) = ∫ p(x,y, z)dz. (2)
The assumption underlying this model is that, conditioned on the latent variables z ∈ Rdz , the two views x and y are independent. However, linear observation models (p(x|z) and p(y|z) as shown in Figure 1 (left)) have limited representation power. In this paper, we consider nonlinear
observation models pθ(x|z;θx) and pθ(y|z;θy), parameterized by θx and θy respectively, which can be the collections of weights of DNNs. In this case, the marginal likelihood pθ(x,y) does not have a closed form. In addition, the inference problem pθ(z|x)—the problem of inferring the latent variables given one of the views—is also intractable.
Inspired by Kingma and Welling (2014)’s work on variational autoencoders (VAE), we approximate pθ(z|x) with the conditional density qφ(z|x;φz), where φz is the collection of parameters of another DNN.1 We can derive a lower bound on the marginal data likelihood using qφ(z|x):
log pθ(x,y) = log pθ(x,y) ∫ qφ(z|x)dz = ∫ log pθ(x,y)qφ(z|x)dz
= ∫ qφ(z|x) ( log
qφ(z|x) pθ(z|x,y) + log pθ(x,y, z) qφ(z|x)
) dz
= DKL(qφ(z|x)||pθ(z|x,y)) + Eqφ(z|x) [ log pθ(x,y, z)
qφ(z|x) ] ≥ Eqφ(z|x) [ log pθ(x,y, z)
qφ(z|x)
] =: L(x,y;θ,φ) (3)
where we used the fact that KL divergence is nonnegative in the last step. As a result, L(x,y;θ,φ) is a lower bound on the data log-likelihood logθ p(x,y). Substituting (2) into (3), we have
L(x,y;θ,φ) = ∫ qφ(z|x) ( log p(z)
qφ(z|x) + log pθ(x|z) + log pθ(y|z)
) dz
= −DKL(qφ(z|x)||p(z)) + Eqφ(z|x) [log pθ(x|z) + log pθ(y|z)] . (4) VCCA maximizes this variational lower bound on the data likelihood on the training set:
max θ,φ
1
N N∑ i=1 L(xi,yi;θ,φ). (5)
The first term in (4) measures the KL divergence between the approximate posterior distribution and the prior distribution of the latent variables z. When the parameterization qφ(z|x) is chosen properly, this term can be computed exactly in closed form. As a concrete example, let the variational approximate posterior be a multivariate Gaussian with diagonal covariance. That is, for a sample pair (xi,yi), we have
log qφ(zi|xi) = logN (zi;µi,Σi), Σi = diag ( σ2i1, . . . , σ 2 idz ) , (6)
where the mean µi and covariance Σi are outputs of an encoding DNN f (and thus [µi,Σi] = f(xi;φz) are deterministic nonlinear functions of xi). In this case, we have
DKL(qφ(zi|xi)||p(zi)) = − 1
2 dz∑ j=1 ( 1 + log σ2ij − σ2ij − µ2ij ) .
The second term of (4) corresponds to the expected complete data likelihood under the approximate posterior distribution. Though still intractable, this term can be approximated by Monte Carlo sampling. In particular, we draw L samples z(l)i ∼ qφ(zi|xi):
z (l) i = µi + Σi (l), where (l) ∼ N (0, I), for l = 1, . . . , L, (7) and have
Eqφ(zi|xi) [log pθ(xi|zi) + log pθ(yi|zi)] ≈ 1
L L∑ l=1 log pθ ( xi|z(l)i ) + log pθ ( yi|z(l)i ) . (8)
Notice that we parameterized qφ(zi|xi) above to obtain the VCCA objective; this is useful when the first view is available for downstream tasks, in which case we can directly apply qφ(zi|xi) to obtain its projection (as features). One could also derive likelihood lower bounds by parameterizing the approximate posteriors qφ(zi|yi) and qφ(zi|xi,yi), and optimize their convex combinations for training. We give a sketch of VCCA in Figure 1 (right).
1For notational simplicity, we denote by θ the collection of parameters associated with the model probabilities pθ(·), and φ the collection of parameters associated with the variational approximate probabilities qφ(·), and often omit specific parameters inside the probabilities.
Connection to multi-view autoencoder (MVAE) If we use the Gaussian observation models
log pθ(x|z) = logN (gx(z;θx), I), log pθ(y|z) = logN (gy(z;θy), I), we observe that log pθ ( xi|z(l)i ) and log pθ ( yi|z(l)i ) measure the reconstruction errors of each
view’s inputs from samples z(l)i using the two DNNs gx and gy respectively. In this case, maximizing L(x,y;θ,φ) is equivalent to
min θ,φ
1
N N∑ i=1 DKL(qφ(zi|xi)||p(zi)) + 1 2NL N∑ i=1 L∑ l=1 ∥∥∥xi − gx (z(l)i ;θx)∥∥∥2 + ∥∥∥yi − gy (z(l)i ;θy)∥∥∥2 (9)
s.t. z(l)i = µi + Σi (l), where (l) ∼ N (0, I), l = 1, . . . , L.
Now, consider the case of Σi → 0, for i = 1, . . . , N , and we have z(l)i → µi which is a deterministic function of x (and there is no need for sampling). In the limit, the second term of (9) reduces to
1
2N N∑ i=1 ‖xi − gx(f(xi;φz);θx)‖ 2 + ‖yi − gy(f(xi;φz);θy)‖ 2 , (10)
which is the objective of the multi-view autoencoder (MVAE, Ngiam et al., 2011). Note, however, that Σi → 0 is prevented by the VCCA objective as it results in a large penalty in DKL(qφ(zi|xi)||p(zi)). Compared with the MVAE objective, in the VCCA objective we are creating L different “noisy” versions of the latent representation and enforce that these versions reconstruct the original inputs well. The “noise” distribution (the variances Σi) are also learned and regularized by the KL divergence DKL(qφ(zi|xi)||p(zi)). Using the VCCA objective, we expect to learn different representations from those of MVAE, due to these regularization effects.
2.1 EXTRACTING PRIVATE VARIABLES
So far, VCCA aims at extracting only the latent variables z that are common to both views. A potential disadvantage of this model is that it assumes the common variables are sufficient by themselves to generate the views, which can be too restrictive in practice. Consider the example of audio and articulatory measurements as two views for speech. Although the transcription is a common variable behind the views, it combines with the physical environment and the vocal tract anatomy to generate the individual views. In other words, there might be large variations in the input space that can not be explained by the common variables, making the objective (4) hard to optimize. It may then be beneficial to explicitly model the private variables within each view.
We therefore propose a new probabilistic graphical model, shown in Figure 2, that we refer to as VCCA-private. We introduce two sets of hidden variables hx ∈ Rdhx and hy ∈ Rdhy to explain the aspects of x and y not captured by the common variables z. Under this model, the data likelihood
is defined by
pθ(x,y, z,hx,hy) = p(z)p(hx)p(hy)pθ(x|z,hx;θx)pθ(y|z,hy;θy), (11)
pθ(x,y) = ∫ ∫ ∫ pθ(x,y, z,hx,hy)dz dhx dhy.
To obtain tractable inference, we introduce the following factored variational posterior
qφ(z,hx,hy|x,y) = qφ(z|x;φz)qφ(hx|x;φx)qφ(hy|y;φy), (12)
where each factor is parameterized by a different DNN. Similarly to VCCA, we can derive a variational lower bound on the data likelihood for VCCA-private as
log pθ(x,y) ≥ ∫ ∫ ∫ qφ(z,hx,hy|x,y) log pθ(x,y, z,hx,hy)
qφ(z,hx,hy|x,y) dz dhx dhy
= ∫ ∫ ∫ qφ(z,hx,hy|x,y) [ log p(z)
qφ(z|x) + log
p(hx)
qφ(hx|x) + log
p(hy)
qφ(hy|y) + log pθ(x|z,hx) + log pθ(y|z,hy) ] dz dhx dhy
= −DKL(qφ(z|x)||p(z))−DKL(qφ(hx|x)||p(hx))−DKL(qφ(hy|y)||p(hy))
+ ∫ ∫ qφ(z|x)qφ(hx|x) log pθ(x|z,hx)dz dhx + ∫ ∫ qφ(z|x)qφ(hy|y) log pθ(y|z,hy)dz dhy
=: Lprivate(x,y;θ,φ). (13)
As in VCCA, the last two terms of (14) can be approximated by Monte Carlo sampling. For example, we draw samples of z and hx from their corresponding approximate posteriors, and concatenate their samples as inputs to the DNN parameterizing pθ(x|z,hx). In this paper, we use simple Gaussian prior distributions for the private variables, i.e., hx ∼ N (0, I) and hy ∼ N (0, I). We leave to future work to examine the effect of more sophisticated prior distributions for the latent variables.
VCCA-private maximizes this lower bound on the training set, i.e.,
max θ,φ
1
N N∑ i=1 Lprivate(xi,yi;θ,φ). (14)
Optimization The objectives (5) and (14) decouple over the training samples and can be trained efficiently using stochastic gradient descent. Enabled by the reparameterization trick, unbiased gradient estimates are obtained by Monte Carlo sampling and the standard backpropagation procedure on minibatches of training samples. We apply the ADAM algorithm (Kingma and Ba, 2015) for optimizing our objectives.
3 RELATED WORK
Recently, there has been much interest in unsupervised deep generative models (Kingma and Welling, 2014; Rezende et al., 2014; Goodfellow et al., 2014; Gregor et al., 2015; Makhzani et al., 2016; Burda et al., 2016; Alain et al., 2016). A common motivation behind these models is that, with the expressive power of DNNs, the generative models can capture distributions for complex inputs. Additionally, if we are able to generate realistic samples from the learned distribution, we can infer that we have discovered the underlying structure of the data, which may allow us to reduce the sample complexity for learning for downstream tasks. These previous models have mostly focused on single-view data. Here we focus on the multi-view setting where multiple views of the data are present for feature extraction but only one view is available at test time (in downstream tasks).
Some recent work has explored deep generative models for (semi-)supervised learning. Kingma et al. (2014) built a generative model based on variational autoencoders (VAEs) for semi-supervised classification, where the authors model the input distribution with two set of latent variables: the class label (if it is missing) and another set that models the intra-class variabilities (styles). Sohn
et al. (2015) proposed a conditional generative model for structured output prediction, where the authors explicitly model the uncertainty in the input/output using Gaussian latent variables. While there are two set of observations (input and output labels) in these work, their graphical models are different from that of VCCA.
Our work is also related to the deep multi-view probabilistic models based on restricted Boltzmann machines (Srivastava and Salakhutdinov, 2014; Sohn et al., 2014). We note that these are undirected graphical models for which both inference and learning are difficult, and one typically resorts to carefully designed variational approximation and Gibbs sampling procedures for training such models. In contrast, our models only require sampling from simple, standard distributions (such as Gaussians), and all parameters can be learned end-to-end by standard stochastic gradient methods. Therefore, our models are more scalable than the previous multi-view probabilistic models.
On the other hand, there is a rich literature in modeling multi-view data using the same or similar graphical models behind VCCA/VCCA-private (Wang, 2007; Jia et al., 2010; Salzmann et al., 2010; Virtanen et al., 2011; Memisevic et al., 2012; Klami et al., 2013). Our methods differ from previous work in parameterizing the probability distributions using DNNs. This makes the model more powerful, while still having tractable objectives and efficient end-to-end training using the local reparameterization technique. We note that, unlike earlier work on probabilistic models of linear CCA (Bach and Jordan, 2005), VCCA does not optimize the same criterion, nor produce the same solution, as any linear or nonlinear CCA. However, we retain the terminology in order to clarify the connection with earlier work on probabilistic models for CCA, which we are extending with DNN models for the observations and for the variational posterior distribution approximation.
4 EXPERIMENTAL RESULTS
In this section, we compare different multi-view representation learning algorithms on three tasks involving several domains: image-image, speech-articulation, and image-text. The algorithms we choose to compare below are closely related to the proposed model or have been shown to have strong empirical performance under similar settings.
• Linear CCA: its probabilistic interpretation motivates this work.
• Deep CCA (DCCA) (Andrew et al., 2013): see its objective in (1).
• Deep canonically correlated autoencoders (DCCAE) (Wang et al., 2015b): combination of the DCCA objective and the reconstruction errors of each view.
• Multi-view autoencoder (MVAE) (Ngiam et al., 2011): see its objective in (10).
• Multi-view contrastive loss (Hermann and Blunsom, 2014): based on the intuition that the distance between embeddings of paired examples x+ and y+ should be smaller than the distance between embeddings of x+ and an unmatched negative example y− by a margin:
min f,g Lcontrast :=
1
N N∑ i max ( 0, m+ dis ( f(x+i ), g(y + i ) ) − dis ( f(x+i ), g(y − i ) )) ,
where y−i is a randomly sampled view 2 example, and m is a margin hyperparameter. We use the cosine distance dis (a,b) = 1− 〈
a ‖a‖ , b ‖b‖
〉 .
4.1 NOISY MNIST DATASET
We first demonstrate our algorithms on the noisy MNIST dataset used by Wang et al. (2015b). The dataset is generated using the MNIST dataset (LeCun et al., 1998), which consists of 28 × 28 grayscale digit images, with 60K/10K images for training/testing. We first linearly rescale the pixel values to the range [0, 1]. Then, we randomly rotate the images at angles uniformly sampled from [−π/4, π/4] and the resulting images are used as view 1 inputs. For each view 1 image, we randomly select an image of the same identity (0-9) from the original dataset, add independent random noise uniformly sampled from [0, 1] to each pixel, and truncate the pixel final values to [0, 1] to obtain the corresponding view 2 sample. Selection of input images are given in Figure 3 (left). The original
training set is further split into training/tuning sets of size 50K/10K. The data generation process ensures that the digit identity is the only common variable underlying both views.
To evaluate the amount of class information extracted by different methods, after unsupervised learning of latent representations, we reveal the labels and train a linear SVM on the projected view 1 training data (using the one-versus-all scheme), and use it to classify the projected test set. This experiment simulates the typical usage of multi-view learning methods, which is to extract useful representations for downstream discriminative tasks.
Note that this synthetic dataset perfectly satisfies the multi-view assumption that the two views are independent given the class label, so the latent representation should contain precisely the class information. This is indeed achieved by CCA-based and contrastive loss-based multi-view approaches. In Figure 3 (right), we show 2D t-SNE (van der Maaten and Hinton, 2008) visualizations of the original view 1 inputs and view 1 projections by various deep multi-view methods.
We use DNNs with 3 hidden layers of 1024 rectified linear units (ReLUs, Nair and Hinton, 2010) each to parameterize the distributions: qφ(z|x), pθ(x|z), pθ(y|z) in VCCA, and additionally qφ(hx|x) and qφ(hy|y) in VCCA-private. The capacities of these networks are the same as those of their counterparts in DCCA and DCCAE from Wang et al. (2015b). The reconstruction networks pθ(x|z) or pθ(x|z,hx) model each pixel of x as an independent Bernoulli variable and parameterize its mean (using a sigmoid activation); pθ(y|z) and pθ(y|z,hy) model y with diagonal Gaussians and parameterize the mean (using a sigmoid activation) and standard deviation for each pixel dimension. We tune the dimensionality dz over {10, 20, 30, 40, 50}, and fix dhx = dhy = 30 for VCCA-private. We select the hyperparameter combination that yields the best SVM classification accuracy on the projected tuning set, and report the corresponding accuracy on the projected test set.
The effect of dropout We add dropout (Srivastava et al., 2014) to all intermediate layers and the input layers and find it to be very useful in our models, with most of the gain coming from dropout applied to the samples of z, hx and hy . This is because dropout encourages each latent dimension to reconstruct the inputs well in the absence of other dimensions, and therefore avoids learning coadapted features. Intuitively, in VCCA-private dropout also helps to prevent the degenerate situation where the pathways x → hx → x and y → hy → y achieve good reconstruction while ignoring z (e.g., by setting it to a constant). We use the same dropout rate for all layers and tune it over {0, 0.1, 0.2, 0.3, 0.4}. We show the 2D t-SNE embeddings of the common variables z learned by VCCA and VCCA-private on test set in Figure 4. We observe that in general, VCCA/VCCA-private tend to separate the classes
in the projection well; dropout significantly improves the performance of both VCCA and VCCAprivate, with the latter slightly outperforming the former. While such class separation can also be achieved by DCCA/contrastive loss as well, these methods can not naturally generate samples in the input space. On the other hand, such separation is not achieved by multi-view autoencoders.
The effect of private variables on reconstructions We show sample reconstructions (mean and standard deviation) by VCCA for the view 2 images from the test set in Figure 5 (columns 2 and 3). We observe that for each input, the mean reconstruction of yi by VCCA is a prototypical image of the same digit, regardless of the individual style in yi. This is to be expected, as yi contains an arbitrary image of the same digit as xi, and the variation in background noise in yi does not appear in xi and can not be reflected in qφ(z|x); thus the best way for pθ(y|z) to model yi is to output a prototypical image of that class to achieve on average small reconstruction error. On the other hand, since yi contains little rotation of the digits, this variation is suppressed to a large extent in qφ(z|x) (it is no longer the major variation in z as in the original inputs).
We show sample reconstructions by VCCA-private for the same set of view 2 images in Figure 5 (columns 4 and 5). With the help of private variables hy (as part of the input to pθ(y|z,hy)), the model does a much better job in reconstructing the styles of y. And by disentangling the private variables from the shared variables, qφ(z|x) achieves even better class separation than VCCA does.
We also note that the standard deviation of the reconstruction is low within the digit and high outside the digit, implying that pθ(y|z,hy) is able to separate the background noise from the digit image.
Disentanglement of private/shared variables In Figure 6 (in Appendix) we provide the 2D tSNE embeddings of the shared variables z (top row) and the private variables hx (bottom row) learned by VCCA-private. In the embedding of hx, digits with different identities but the same rotation are mapped close together, and the rotation varies smoothly from left to right, confirming that the private variables contain little class information but mainly style information.
Finally, we give the test error rates of linear SVMs applied to the features learned with different models in Table 1. VCCA-private is comparable in performance to the best previous approach (DCCAE), while having the advantage that it can also generate.
4.2 XRMB SPEECH-ARTICULATION DATASET
We now consider the task of learning acoustic features for speech recognition. We use data from the Wisconsin X-ray microbeam (XRMB) corpus (Westbury, 1994), which contains simultaneously recorded speech and articulatory measurements from 47 American English speakers. We follow the setup of Wang et al. (2015a,b) and use the learned features for speaker-independent phonetic recognition.2 The two input views are standard 39D acoustic features (13 mel frequency cepstral coefficients (MFCCs) and their first and second derivatives) and 16D articulatory features (horizontal/vertical displacement of 8 pellets attached to several parts of the vocal tract), each then concatenated over a 7-frame window around each frame to incorporate context. The speakers are split into disjoint sets of 35/8/2/2 speakers for feature learning/recognizer training/tuning/testing. The 35 speakers for feature learning are fixed; the remaining 12 are used in a 6-fold experiment (recognizer training on 8 speakers, tuning on 2 speakers, and testing on the remaining 2 speakers). Each speaker has roughly 50K frames. We remove the per-speaker mean and variance of the articulatory measurements for each training speaker, and remove the mean of the acoustic measurements for each utterance. All learned feature types are used in a “tandem” speech recognizer (Hermansky et al., 2000), i.e., they are appended to the original 39D features and used in a standard hidden Markov model (HMM)-based recognizer with Gaussian mixture observation distributions.
Each algorithm uses up to 3 ReLU hidden layers, each of 1500 units, for the projection and reconstruction mappings. For VCCA/VCCA-private, we use Gaussian observation models as the inputs are real-valued. In contrast to the MNIST experiments, we do not learn the standard deviations of each output dimension on training data, as this leads to poor downstream task performance. Instead, we use isotropic covariances for each view, and tune the standard deviations by grid search. The best model uses a smaller standard deviation (0.1) for the view 2 than for view 1 (1.0), effectively putting more emphasis on the reconstruction of articulatory measurements. Our best performing VCCA model uses dz = 70, while the best performing VCCA-private model uses dz = 70 and dhx = dhy = 10.
2As in Wang and Livescu (2016), we use the Kaldi toolkit (Povey et al., 2011) for feature extraction and recognition with hidden Markov models. Our results do not match Wang et al. (2015a,b) (who instead used the HTK toolkit (Young et al., 1999)) for the same types of features, but the relative results are consistent.
The mean phone error rates (PER) over 6 folds obtained by different algorithms are given in Table 1. Our methods achieve competitive performance in comparison to previous deep multi-view methods.
4.3 MIR-FLICKR DATASET
Finally, we consider the task of learning cross-modality features for topic classification on the MIRFlickr database (Huiskes and Lew, 2008). The Flickr database contains 1 million images accompanied by user tags, among which 25000 images are labeled with 38 topic classes (each image may be categorized as multiple topics). We use the same image and text features as in previous work (Srivastava and Salakhutdinov, 2014; Sohn et al., 2014): the image feature is 3857 dimensional real-valued vector, composed of Pyramid Histogram of Words (PHOW) (Bosch et al., 2007), GIST (Oliva and Torralba, 2001), and MPEG-7 descriptors (Manjunath et al., 2001), while the text feature is a 2000-dimensional binary vector of frequent tags.
Following the same protocol as Sohn et al. (2014), we train multi-view representations using the unlabelled data,3 and use projected image features of the labeled data (further divided into splits of 10000/5000/10000 samples for training/tuning/testing) for training and evaluating a classifier that predicts the topic labels, corresponding to the unimodal query task in Srivastava and Salakhutdinov (2014); Sohn et al. (2014). For each algorithm, we select the model which achieves the highest mean average precision (mAP) on the validation set, and report its performance on the test set.
Each algorithm uses up to 4 ReLU hidden layers, each of 1024 units, for the projection and reconstruction mappings. For VCCA/VCCA-private, we use Gaussian observation models with isotropic covariance for image features, with standard deviation tuned by grid search, and a Bernoulli model for text features. In this experiment, we also found it helpful to tune an additional trade-off parameter for the text-view likelihood (cross-entropy); the best VCCA/VCCA-private models prefer a large trade-off parameter of the level 104, emphasizing the reconstruction of the sparse text-view inputs. Our best performing VCCA model uses dz = 1024, while the best performing VCCA-private model uses dz = 1024 and dhx = dhy = 16.
As shown in Table 1, VCCA/VCCA-private achieve significantly higher mAPs than other methods considered here. Being much easier to train, the performance of our methods are competitive with the previous state-of-the-art mAP result of 0.607 achieved by the multi-view RBMs of Sohn et al. (2014) under the same setting.
5 CONCLUSIONS
We have proposed variational canonical correlation analysis (VCCA), a deep generative method for multi-view representation learning. Our method embodies a natural idea for multi-view learning: the multiple views can be generated from a small set of shared latent variables. VCCA is parameterized by DNNs and can be trained efficiently by backpropagation, and is therefore scalable. We have also shown that, by modeling the private variables that are specific to each view, the VCCA-private variant can disentangle shared/private variables and provide higher-quality reconstructions.
In the future, we will explore other prior distributions such as mixtures of Gaussians or discrete random variables, which may enforce clustering in the latent space and in turn work better for discriminative tasks. We will also explore other observation models, including replacing the autoencoder objective with that of adversarial networks (Goodfellow et al., 2014; Makhzani et al., 2016; Chen et al., 2016). Another direction is to explicitly incorporate the structure of the inputs, such as the sequence structure of speech and text and the spatial structure of images.
ACKOWLEDGEMENTS
This research was supported by NSF grant IIS-1321015. The opinions expressed in this work are those of the authors and do not necessarily reflect the views of the funding agency. This research used GPUs donated by NVIDIA Corporation.
3As in Sohn et al. (2014), we exclude about 250000 samples which contain fewer than two tags.
A ADDITIONAL T-SNE VISUALIZATION OF NOISY MNIST

This paper considers the case where multiple views of data are learned through a probabilistic deep neural network formulation. This makes the model non-linear (unlike e.g. CCA) but makes inference difficult. Therefore, the VAE framework is invoked for inference.
In [Ref 1] the authors show that maximum likelihood estimation based on their linear latent model leads to the canonical correlation directions. But in the non-linear case with DNNs it's not clear (at least with the present analysis) what the solution is wrt to the canonical directions. There's no such analysis in the paper, hence I find it a stretch to refer to this model as a CCA type of model. In contrast, e.g. DCCA / DCCAE are taking the canonical correlation between features into account inside the objective and provide interpretations.
[Ref 1] F. R. Bach and M. I. Jordan. A probabilistic interpretation of canonical correlation analysis. Technical Report 688, 2005.
There is also a significant body of very related work on non-linear multi-view models which is not discussed in this paper. For example, there's been probabilistic non-linear multi-view models [Ref 2, 3], also extended to the Bayesian case with common/private spaces [Ref 4] and the variational / deep learning case [Ref 5].
[Ref 2] Ek et al. Gaussian process latent variable models for human pose estimation. MLMI, 2007.
[Ref 3] Shon et al. Learning shared latent structure for image synthesis and robotic imitation. NIPS, 2006.
[Ref 4] Damianou et al. Manifold relevance determination. ICML, 2012.
[Ref 5] Damianou and Lawrence. Deep Gaussian processes. AISTATS, 2013.
I can see the utility of this model as bringing together two elements: multi-view modeling and VAEs. This seems like an obvious idea but to the best of my knowledge it hasn't been done before and is actually a potentially very useful model.
However, the question is, what is the proper way of extending VAE to multiple views? The paper didn't convince me that VAE can work well with multiple views using the shown straightforward construction. Specifically, VCCA doesn't seem to promote the state of the art in terms of results (it actually is overall below the SOA), while the VCCA-private seems a quite ill-posed model: the dimensionalities d have to be manually tuned with exhaustive search; further, the actual model does not provide a consinstent way of encouraging the private and common variables to avoid learning redundant information. Relying only on dropout for this seems a quite ad-hoc solution (in fact, from Fig. 4 (ver2) it seems that the dropout rate is quite crucial). Perhaps good performance might be achieved with a lot of tuning (which might be why the FLICKR results got better in ver2 without changing the model), but it seems quite difficult to optimize for the above reasons. From a purely experimental point of view, VCCA-private doesn't seem to promote the SOA either. Of course one wouldn't expect any new published paper to beat all previous baselines, but it seems that extension of VAE to multiple views is a very interesting idea which deserves some more investigation of how to do it efficiently.
Another issue is the approximate posterior being parameterized only from one of the views. This makes the model less useful as a generic multi-view model, since it will misbehave in tasks other than classification. But if classification is the main objective, then one should compare to a proper classification model, e.g. a feedforward neural network.
The plots of Fig. 8 are very nice. Overall, the paper convinced me that there is merit in attaching multiple views to VAE. However, it didn't convince me a) that the proposed way to achieve this is practical b) that there is a connection to CCA (other than being a method for multiple views). The bottom line is that, although the paper is interesting, it needs a little more work.


ROTATION PLANE DOUBLY ORTHOGONAL RECUR-
Recurrent Neural Networks (RNNs) applied to long sequences suffer from the well known vanishing and exploding gradients problem. The recently proposed Unitary Evolution Recurrent Neural Network (uRNN) alleviates the exploding gradient problem and can learn very long dependencies, but its nonlinearities make it still affected by the vanishing gradient problem and so learning can break down for extremely long dependencies. We propose a new RNN transition architecture where the hidden state is updated multiplicatively by a time invariant orthogonal transformation followed by an input modulated orthogonal transformation. There are no additive interactions and so our architecture exactly preserves forward hidden state activation norm and backwards gradient norm for all time steps, and is provably not affected by vanishing or exploding gradients. We propose using the rotation plane parameterization to represent the orthogonal matrices. We validate our model on a simplified memory copy task and see that our model can learn dependencies as long as 5,000 timesteps.
1 INTRODUCTION
Deep Neural Networks have shown increasingly impressive performance on a wide variety of practical tasks. Recurrent Neural Networks (RNNs) are powerful sequence modeling tools that have found successful applications in speech recognition, natural language processing, image captioning, and many more (Sutskever et al., 2014; Bahdanau et al., 2014; Wu et al., 2016; Donahue et al., 2015; Karpathy & Li, 2015; Luong et al., 2014). One fundamental problem with RNNs is the so called vanishing and exploding gradients problem (Hochreiter, 1991; Bengio, 1994; Hochreiter et al., 2001). These problems occur when training RNNs using gradient descent to model long sequences where the gradient magnitude either goes towards zero or infinity, respectively, as the length of the sequence increases.
Several heuristics have been proposed to alleviate these problems. The Long Short Term Memory (LSTM) in Hochreiter et al. (1997) and Gated Recurrent Units (GRU) in Cho et al. (2014) have been incredibly successful recurrent transition architectures to model complicated dependencies for sequences up to several hundred timesteps long and are the main RNN architectures in use today. The IRNN model modifies the standard RNN transition to initialize at the identity, which increases the timestep length modeling capability (Le et al., 2015). Stabilizing the forward hidden state norm can have a positive effect on hidden state gradient norm preservation (Krueger & Memisevic, 2015; Ba et al., 2016; Cooijmans et al., 2016). A simple gradient norm clipping during hidden state backpropagation can also help to alleviate the exploding gradient problem for these architectures (Graves, 2013).
Recently, there has been a surge of interest in orthogonal and unitary transition architectures. Orthogonal and unitary transitions exactly preserve forward and gradient norms passed through them. Theory developed for the linear neural networks suggests that training time can be vastly shorter when the weight matrices are orthogonal, and even independent of depth (Saxe et al., 2013). The Unitary Evolution Recurrent Neural Networks utilizes a unitary recurrent transition followed by a contractive nonlinearity to exactly solve the exploding gradient problem and greatly alleviate the
vanishing gradient problem (Arjovsky et al., 2015). A very recent extension expands on this work to increase the expressive power of these transitions and increases the validated architecture up to 2000 timestep dependencies in Wisdom et al. (2016). Analytic solutions to the most common long term dependency example tasks, the memory copy and addition problems, are provided with linear orthogonal recurrent neural networks in Henaff et al. (2016). A nonlinear activation function that is locally orthogonal is proposed in Chernodub & Nowicki and shown to have great potential. This particular activation scheme, however, is discontinuous (not just nondifferentiable), and could increase optimization difficulty in some cases.
An open question that we try to partially address in this work is whether a recurrent transition architecture can be fully orthogonal or unitary (and thus linear) and still learn expressive models to solve practical tasks. To address this problem, we propose the rotation plane doubly orthogonal RNN, a novel recurrent transition architecture that provably preserves forward hidden state activation norm and backpropagated gradient norm and thus does not suffer from exploding or vanishing gradients. The doubly orthogonal refers to the fact that the RNN transition architecture updates the hidden state multiplicatively by a time invariant orthogonal transformation followed by an input modulated orthogonal transformation. Rotation plane refers to the parameterization we use to represent the orthogonal matrices. We evaluate our approach on a simplified 1-bit version of the memory copying task, and find that our architecture can scale to 5,000 timesteps on this task, outperforming past approaches.
2 DOUBLY ORTHOGONAL RNN ARCHITECTURE
In this section we describe our proposed architecture, which we call the Doubly Orthogonal Recurrent Neural Net (DORNN). We also show that the DORNN provably preserves forward norm and gradient norm.
We briefly review the definition of an orthogonal or unitary matrix, since it is fundamental to the definition and properties of our transition architecture. Orthogonal matrices, or rotation matrices, are defined as matrices Q ∈ Rn×n such that: QTQ = I . We restrict our attention to the special orthogonal matrices SO(n) such that det(Q) = 1. The set of special orthogonal matrices forms a matrix lie group. The complex analogue of an orthogonal matrix is a unitary matrix and is defined similarly as matrices U ∈ Cn×n such that UHU = I .
2.1 RECURRENCE EQUATIONS
Our recurrence equation is defined below:
ht+1 = Rxh(xt)Rhhht
where Rxh(xt) is an input modulated orthogonal or unitary matrix, and Rhh is a time invariant orthogonal or unitary matrix that is applied at every timestep. We parameterize Rhh by θhh and Rxh by φxh(xt), where φ is a function of the input xt. Figure 1 shows a graphical depiction of this transition architecture.
Our transition is different from most recurrent neural network transitions since it is fully multiplicative and contains no addition terms. The input enters into the equation by modulating the rotation applied to the hidden state. This allows for more expressivity in the hidden transition than a constant linear transition, though not as much as a nonlinear hidden state dependent transition. By having an input dependent transition, the hidden dynamics are more complicated than a constant linear transition and are nonlinear with respect to the input xt. Linear time-varying models can express very complicated policies, such as in Levine et al. (2016). Our model can be viewed as a linear orthogonal time-varying model where the variation due to time is due to different input signals applied.
2.2 FORWARD ACTIVATION NORM AND GRADIENT NORM PRESERVATION
Here we prove that this recurrent transition architecture exactly preserves (up to numerical precision) the forward hidden state activation norm and the backwards gradient norm. All proofs carry forward
with unitary matrices instead of orthogonal matrices when transposes are replaced with Hermitian transposes.
||ht+1|| = ||Rxh(xt)Rhhht|| = ||Rcombined(xt)ht|| = ||ht||
where Rcombined = Rxh(xt)Rhh. The last equality follows since orthogonal matrices are a group and so Rcombined ∈ SO(n), and ||Qv|| = ||v|| for any Q ∈ SO(n) and any v ∈ Rn, since ||Qv|| = √ vTQTQv = √ vT Iv = √ vT v = ||v|| by the definition of Q (orthogonal matrices preserve norm).
Now let C(hT ) be a scalar cost function. The vanishing gradients problem occurs if || ∂C∂h1 || → 0 as T →∞ and the exploding gradient problem occurs if || ∂C∂h1 || → ∞ as T →∞.
∂C ∂ht = ∂C ∂hT ∂hT ∂ht = ∂C ∂hT T−1∏ i=t ∂hi+1 ∂hi = ∂C ∂hT T−1∏ i=t RThhRxh(xi) T
and so
∣∣∣∣∣∣∣∣ ∂C∂ht ∣∣∣∣∣∣∣∣ = ∣∣∣∣∣ ∣∣∣∣∣ ∂C∂hT T−1∏ i=t RThhRxh(xi) T ∣∣∣∣∣ ∣∣∣∣∣ = ∣∣∣∣∣∣∣∣ ∂C∂hT ∣∣∣∣∣∣∣∣
where the last equality follows from ( ∏T−1 i=t R T hhRxh(xi)
T ) ∈ SO(n) : an orthogonal matrix’s transpose is its inverse and the inverse of a group element is in the group. So the norm of the gradient of the cost C at hidden state ht is the same as the final norm of the gradient at hidden state hT , and the transition does not suffer from vanishing or exploding gradients.
3 ROTATION PLANE DOUBLY ORTHOGONAL RNN
Within the Doubly Orthogonal RNN, there is a choice in how the orthogonal (alternatively, unitary), transformations are parameterized. This choice determines the number of parameters, how the gradient propagates from the hidden state to the input, and much more. There are a wide variety of possible DORNN architectures, since there are a wide variety of different ways to parameterize orthogonal and unitary matrices, each with their pros and cons. We provide a particular instantiation of the Doubly Orthogonal RNN by parameterizing the orthogonal matrices in terms of the composition of many plane rotations. We call this RNN architecture the Rotation Plane Doubly Orthogonal RNN, or RP-DORNN.
We note that while we focus on this architecture within the context of a recurrent neural network, the rotation plane parameterization of orthogonal matrices could be equally useful for parameterizing very deep feedforward weight matrices.
3.1 ROTATION PLANE REPRESENTATION OF ORTHOGONAL MATRICES
First we show how we parameterize a single plane rotation. The full architecture is generated by a composition of a sequence of these plane rotations.
Well known in numerical linear algebra, any orthogonal matrix Q ∈ SO(n) can be generated as the product of n Householder reflection matrices H = I − 2 uu T
||u||22 for nonzero vectors u ∈ Rn.
Work in Geometric Algebra (for example see chapter 6 of Dorst et al. (2009)) gives us further insight into this parameterization. Two subsequent Householder reflections generate a plane rotation in the plane spanned by the reflection vectors. The generated rotation is twice the angle between the two reflection vectors. We can use this to parameterize rotation planes in term of the desired angle of rotation, by generating two reflection vectors that produce the rotation. By rotating in several planes in sequence, we can generate arbitrary orthogonal matrices. Thus we can view the rotation angle in a given plane as either a parameter to be optimized or as an input from below in the neural network, both of which we utilize in our proposed recurrent architecture.
Concretely, for a plane spanned by two orthonormal vectors, w0, and w1, we generate a rotation of angle θ from w0 towards w1 in the w0−w1 plane by generating the following two reflection vectors v0 and v1 and composing their reflections.
v0 = w0
v1 = cos(θ/2)w0 + sin(θ/2)w1
then Rθ = (I − 2v1vT1 )(I − 2v0vT0 ). We don’t need to divide by the magnitude of v0 or v1 since by construction they are unit vectors. When we apply Rθ to a vector or batch of vectors B, we don’t have to generate the matrix Rθ, since
RθB = (I − 2v1vT1 )(I − 2v0vT0 )B = (I − 2v1vT1 )(B − 2v0(vT0 B)) =
= B − 2v1(vT1 B)− 2v0(vT0 B) + 4v1(vT1 (v0(vT0 B)))
and so we never have to form the full Rθ matrix, we only need to perform matrix multiplies of a vector with a matrix and the intermediate dimensions are much smaller than forming the dense Rθ matrix.
In the next section we treatw0, w1 as random constant orthonormal vectors and treat θ as a parameter or a function of inputs.
3.2 RP-DORNN
We generateRhh andRxh as a sequence of plane rotations in the Rotation Plane Doubly Orthogonal Recurrent Neural Network (RP-DORNN).
Rhh = k∏ i=1 Rθi
and
Rxh(xt) = l∏ i=1 Rφi(xt)
for l, k ≤ bn2 c. Each Rθi and Rφi are plane rotations in randomized orthogonal planes (where the planes from Rhh are orthogonal to one another and the planes from Rxh are orthogonal to one another but the planes from Rhh intersect with the ones from Rxh randomly), parameterized by the angle of rotation. For Rφi , the angle of rotation is a function of xt, and for Rθi . In our exploratory experiments we investigated several choices of l and k but for our final experiments we used l = k = bn2 c, so Rxh and Rhh can each affect the entire hidden space. For our experiments we generated the planes to rotate in for each of Rhh and Rxh by initializing a random Gaussian matrix and projecting to an orthogonal matrix, and taking consecutive pairs of columns as the orthonormal vectors that span a rotation plane (w0 and w1 for each plane in the notation of Section 3.1).
There are several possible choices for the parameterization of the angle parameters. In our exploratory experiments we investigated directly generating the angle value as a parameter but this suffers from topology wrapping: θ and θ +m2π for any integer m generate the same transformation. We settled on generating θi = 2πσ(αi) for real parameters αi, where σ is the sigmoid function and φi(xt) = πσ(Wxt + b) for learned affine transform Wxt + b. This only allows positive angle rotations for the Rxh and it places negative and positive angle rotations on the opposite side of the parameter space for Rhh (negative αi produces positive angle rotations and positive αi produce negative angle rotations). In addition, with this parameterization, σ(αi) and σ(Wxt + b) logarithmically approach 0 for very negative αi and Wxt + b, which is useful since for long timescales the transformations may be applied exponentially many times. In our experiments, αi were drawn uniformly between −3 and 0, W was initialized with a Gaussian, and b was initialized as 0. The Rotation Plane representation allows a fixed transformation subspace via the unvarying planes of rotation, but with a small number of angle parameters that are actively modulated to produce vastly different transformations for different inputs. Other orthogonal matrix representations do not give as much utility to be modulated by a small number of parameters that can be then generated by the input.
4 BACKGROUND: ALTERNATIVE ORTHOGONAL MATRIX REPRESENTATIONS AND OPTIMIZATION
4.1 UNITARY REPRESENTATION USED IN URNN
The approach in Arjovsky et al. (2015) parameterizes their unitary matrices as a combination of unitary building blocks, such as complex reflections, complex unit diagonal scalings, permutations, and the FFT and iFFT.
4.2 OPTIMIZATION REPRESENTATION ON ORTHOGONAL MATRICES
Another way to parameterize an orthogonal matrix is to start with an arbitrary orthogonal matrix and then ensure that the optimization process produces an orthogonal matrix at each time step.
Several ways of optimizing orthogonal matrices. The defining equation can be used as a regularizer on a regular matrix transition, i.e. ||I − QTQ||. This approach is used in some experiments in the ORNN paper, but it does not produce an exactly orthogonal matrix, and as such still suffers from exploding and vanishing gradients. Another possibility is to perform gradient descent on a matrix and reproject to the orthogonal matrices by SVD and setting the singular values to 1. The Lie algebra representation is Qt+1 = Qtq when q = exp(A) if A is skew-symmetric: AT +A = 0.
Then the representation optimizes w in skew symmetric matrices. This approach is used in Hyland & Rätsch (2016). The main downside of this approach is that you need to calculate gradient through exp, the matrix exponential, which we are not aware of a closed-form solution, and in Hyland & Rätsch (2016) the authors used finite differences to calculate the derivative of the exponential, which is incompatible with backpropagation. Recently proposed full capacity uRNN uses the Cayley transform: Qt+1 = (I + λ2A)
−1(I − λ2A)Qt for A skew symmetric (skew-Hermitian for unitary Q) to stay on the manifold of orthogonal (unitary) matrices.
5 EXPERIMENTS
Our experiments seek to answer the following questions:
How do the theoretically guaranteed preservation properties of the DORNN architecture contribute to practical training to learn extremely long term dependencies, and how is training time to success influenced by the length of a long term dependency.
We partially address these questions on our RP-DORNN architecture on a simplified 1-bit version of the memory copy task that is commonly used to test long term dependencies in RNN architectures. The task is as follows:
The input is a sequence of four dimension one-hot vectors (categorical variables with four categories) of length T +2 for a given T > 1. The first input is either a 1 or 2, followed by a string of T 0’s, and finally ends with a 3. All of the outputs except for the last output are arbitrary and unpenalized. The last output should be the same category as the first input (1 or 2, whichever was given). The loss is the categorical cross entropy of the last output timestep.
The simplifications from the usual memory copy experiment are that the number of categories is four instead of the usual ten, that the sequence to be copied is of length one instead of the usual ten, and that the loss only takes into account the portion of the output that corresponds to the copied output (the final timestep in our case), instead of the entire output sequence. These simplifications were performed in order to increase the number of experiments that could be run by training smaller models, and to function as a minimal experiment on the effect of increasing the sequence length. The third simplification was performed in order to increase the signal to noise ratio in the loss for very long sequences, since the majority of the loss comes from the intermediate outputs for very large T . The model would otherwise learn to use most of its capacity to reduce error in the irrelevant intermediate output stage instead of the relevant copied sequence stage. A useful side effect is that the entire gradient signal comes from the relevant copy output error.
We successfully trained the architecture on the task with T = 600, 800, 1600, 2400, 3200, and 5000, with a minibatch size of 128. There are only two possible input and output sequences, so the minibatch size is somewhat redundant, but the different proportion of 1 and 2 inputs at each timestep helps inject extra randomness to the training procedure. Figure 2 shows the training loss curves for each of the timesteps, compared against the baseline of outputting 1 or 2 with equal probability on the final timestep, which achieves ln(2) ≈ 0.693 loss. Figure 3 shows the accuracy of the network over training applied on the input minibatches. None of the hyperparameters were changed for scaling the timestep from 500 to 5000.
Note that for longer T , the training became increasingly unstable (although it was still able to succeed). A common phenomenon is that the loss would jump far above the baseline for a brief period but then return to a lower loss than before the jump. A possible explanation is that the training process becomes increasingly sensitive to small transition parameter changes as T increases, since the transition is applied T times, and so the final hidden state would land in a completely different region after a small parameter change and the network would have to adjust to the new regions.
Training was terminated for each T after perfect accuracy had been achieved for several hundred training iterations, but the loss would continue to asymptotically approach 0 if training were to progress as the network increases confidence in its predictions.
Saxe et al. (2013) suggests that training time should be independent of sequence length for a fully linear orthogonal neural network. We observed a roughly linear training time to success with respect to T . The discrepancy might be due to the nonlinearities present elsewhere in the network or due to the parameterization of the orthogonal matrices.
This experiments section will be expanded upon as more experimental results and comparisons are available.
6 DISCUSSION
While we describe one particular orthogonal recurrent transition, the rotation plane parameterization describes a rich space of orthogonal transitions. Our input to hidden dependent transition is expressive enough for simplified memory copy task and can learn extremely long term dependencies up to 5,000 time steps long. The architecture still needs to be validated on more complicated tasks. Future work:
The architecture we described here is linear with the transition dependent on the input, but nonlinear locally orthogonal transitions are possible discontinuous. We experimented with different locally orthogonal transitions but found them harder to train successfully (likely they are more unstable due to the discontinuity, especially amplified over the hundreds or thousands of timesteps). It might be possible to find a nonlinear transition that still produces an orthogonal Jacobian that is continuous: the equations don’t outright prevent it, but it results from the interplay of several constraints and so finding such an architecture is harder. An open question is whether is it possible to construct something like an LSTM within the space of orthogonal or unitary matrices, to get the best of both worlds? It is not clear how much expressivity in the recurrent transition is lost by having linear input dependent model instead of a nonlinear hidden to hidden transition.
A possible future direction is the combinations of orthogonal or unitary transitions for long term dependencies and regular RNN or LSTM units for complicated short term dependencies.
In this work we randomly fixed the planes to rotate in, but using the Cayley transform as in the full capacity uRNN the rotation planes could be optimized jointly, and so we could combine with our angle based parameterization to get full capacity input dependent rotations instead of being sensitive to the randomly initialized planes.
ACKNOWLEDGMENTS
This research was funded in part by Darpa through the SIMPlEX program and the FunLOL program, by NSF through a Career award. Zoe McCarthy is also supported by an NSF Graduate Fellowship. Peter Chen is also supported by a Berkeley AI Research (BAIR) lab Fellowship.

My main objection with this work is that it operates under a hypothesis (that is becoming more and more popular in the literature) that all we need is to have gradients flow in order to solve long term dependency problems. The usual approach is then to enforce orthogonal matrices which (in absence of the nonlinearity) results in unitary jacobians, hence the gradients do not vanish and do not explode. However this hypothesis is taken for granted (and we don't know it is true yet) and instead of synthetic data, we do not have any empirical evidence that is strong enough to convince us the hypothesis is true.
My own issues with this way of thinking is: a) what about representational power; restricting to orthogonal matrices it means we can not represent the same family of functions as before (e.g. we can't have complex attractors and so forth if we run the model forward without any inputs). You can only get those if you have eigenvalues larger than 1. It also becomes really hard to deal with noise (since you attempt to preserve every detail of the input, or rather every part of the input affects the output). Ideally you would want to preserve only what you need for the task given limited capacity. But you can't learn to do that. My issue is that everyone is focused on solving this preserved issue without worrying of the side-effects.
I would like one of these papers going for jacobians having eigenvalues of 1 show this helps in realistic scenarios, on complex datasets.