MAKING STOCHASTIC NEURAL NETWORKS FROM DETERMINISTIC ONES
It has been believed that stochastic feedforward neural networks (SFNN) have several advantages beyond deterministic deep neural networks (DNN): they have more expressive power allowing multi-modal mappings and regularize better due to their stochastic nature. However, training SFNN is notoriously harder. In this paper, we aim at developing efficient training methods for large-scale SFNN, in particular using known architectures and pre-trained parameters of DNN. To this end, we propose a new intermediate stochastic model, called Simplified-SFNN, which can be built upon any baseline DNN and approximates certain SFNN by simplifying its upper latent units above stochastic ones. The main novelty of our approach is in establishing the connection between three models, i.e., DNN → Simplified-SFNN → SFNN, which naturally leads to an efficient training procedure of the stochastic models utilizing pre-trained parameters of DNN. Using several popular DNNs, we show how they can be effectively transferred to the corresponding stochastic models for both multi-modal and classification tasks on MNIST, TFD, CIFAR-10, CIFAR-100 and SVHN datasets. In particular, our stochastic model built from the wide residual network has 28 layers and 36 million parameters, where the former consistently outperforms the latter for the classification tasks on CIFAR-10 and CIFAR-100 due to its stochastic regularizing effect.
1 INTRODUCTION
Recently, deterministic deep neural networks (DNN) have demonstrated state-of-the-art performance on many supervised tasks, e.g., speech recognition (Hinton et al., 2012a) and object recognition (Krizhevsky et al., 2012). One of the main components underlying these successes is on the efficient training methods for deeper and wider DNNs, which include backpropagation (Rumelhart et al., 1988), stochastic gradient descent (Robbins & Monro, 1951), dropout/dropconnect (Hinton et al., 2012b; Wan et al., 2013), batch/weight normalization (Ioffe & Szegedy, 2015; Salimans & Kingma, 2016), and various activation functions (Nair & Hinton, 2010; Gulcehre et al., 2016). On the other hand, stochastic feedforward neural networks (SFNN) (Neal, 1990) having random latent units are often necessary in order to model complex stochastic natures in many real-world tasks, e.g., structured prediction (Tang & Salakhutdinov, 2013), image generation (Goodfellow et al., 2014) and memory networks (Zaremba & Sutskever, 2015). Furthermore, it has been believed that SFNN has several advantages beyond DNN (Raiko et al., 2014): it has more expressive power for multi-modal learning and regularizes better for large-scale learning.
Training large-scale SFNN is notoriously hard since backpropagation is not directly applicable. Certain stochastic neural networks using continuous random units are known to be trainable efficiently using backpropagation under the variational techniques and the reparameterization tricks (Kingma & Welling, 2013). On the other hand, training SFNN having discrete, i.e., binary or multi-modal, random units is more difficult since intractable probabilistic inference is involved requiring too many random samples. There have been several efforts developing efficient training methods for SFNN having binary random latent units (Neal, 1990; Saul et al., 1996; Tang & Salakhutdinov, 2013; Bengio et al., 2013; Raiko et al., 2014; Gu et al., 2015) (see Section 2.1 for more details). However, training SFNN is still significantly slower than doing DNN of the same architecture, e.g., most prior
works on this line have considered a small number (at most 5 or so) of layers in SFNN. We aim for the same goal, but our direction is orthogonal to them.
Instead of training SFNN directly, we study whether pre-trained parameters of DNN (or easier models) can be transferred to it, possibly with further fine-tuning of light cost. This approach can be attractive since one can utilize recent advances in DNN on its design and training. For example, one can design the network structure of SFNN following known specialized ones of DNN and use their pre-trained parameters. To this end, we first try transferring pre-trained parameters of DNN using sigmoid activation functions to those of the corresponding SFNN directly. In our experiments, the heuristic reasonably works well. For multi-modal learning, SFNN under such a simple transformation outperforms DNN. Even for the MNIST classification, the former performs similarly as the latter (see Section 2 for more details). However, it is questionable whether a similar strategy works in general, particularly for other unbounded activation functions like ReLU (Nair & Hinton, 2010) since SFNN has binary, i.e., bounded, random latent units. Moreover, it lost the regularization benefit of SFNN: it is rather believed that transferring parameters of stochastic models to DNN helps its regularization, but the opposite direction is unlikely possible.
To address the issues, we propose a special form of stochastic neural networks, named SimplifiedSFNN, which intermediates between SFNN and DNN, having the following properties. First, Simplified-SFNN can be built upon any baseline DNN, possibly having unbounded activation functions. The most significant part of our approach lies in providing rigorous network knowledge transferring (Chen et al., 2015) between Simplified-SFNN and DNN. In particular, we prove that parameters of DNN can be transformed to those of the corresponding Simplified-SFNN while preserving the performance, i.e., both represent the same mapping and features. Second, Simplified-SFNN approximates certain SFNN, better than DNN, by simplifying its upper latent units above stochastic ones using two different non-linear activation functions. Simplified-SFNN is much easier to train than SFNN while utilizing its stochastic nature for regularization.
The above connection DNN→ Simplified-SFNN→ SFNN naturally suggests the following training procedure for both SFNN and Simplified-SFNN: train a baseline DNN first and then fine-tune its corresponding Simplified-SFNN initialized by the transformed DNN parameters. The pre-training stage accelerates the training task since DNN is faster to train than Simplified-SFNN. In addition, one can also utilize known DNN training techniques such as dropout and batch normalization for fine-tuning Simplified-SFNN. In our experiments, we train SFNN and Simplified-SFNN under the proposed strategy. They consistently outperform the corresponding DNN for both multi-modal and classification tasks, where the former and the latter are for measuring the model expressive power and the regularization effect, respectively. To the best of our knowledge, we are the first to confirm that SFNN indeed regularizes better than DNN. We also construct the stochastic models following the same network structure of popular DNNs including Lenet-5 (LeCun et al., 1998), NIN (Lin et al., 2014) and WRN (Zagoruyko & Komodakis, 2016). In particular, WRN (wide residual network) of 28 layers and 36 million parameters has shown the state-of-art performances on CIFAR-10 and CIFAR-100 classification datasets, and our stochastic models built upon WRN outperform the deterministic WRN on the datasets.
Organization. In Section 2, we focus on DNNs having sigmoid and ReLU activation functions and study simple transformations of their parameters to those of SFNN. In Section 3, we consider DNNs having general activation functions and describe more advanced transformations via introducing a new model, named Simplified-SFNN.
2 SIMPLE TRANSFORMATION FROM DNN TO SFNN
2.1 PRELIMINARIES FOR SFNN
Stochastic feedforward neural network (SFNN) is a hybrid model, which has both stochastic binary and deterministic hidden units. We first introduce SFNN with one stochastic hidden layer (and without deterministic hidden layers) for simplicity. Throughout this paper, we commonly denote the bias for unit i and the weight matrix of the `-th hidden layer by b`i and W
`, respectively. Then, the stochastic hidden layer in SFNN is defined as a binary random vector with N1 units, i.e., h1 ∈
{0, 1}N1 , drawn under the following distribution:
P ( h1 | x ) = N1∏ i=1 P ( h1i | x ) , where P ( h1i = 1 | x ) = σ ( W1ix+ b 1 i ) . (1)
In the above, x is the input vector and σ (x) = 1/ (1 + e−x) is the sigmoid function. Our conditional distribution of the output y is defined as follows:
P (y | x) = EP (h1|x) [ P ( y | h1 )] = EP (h1|x) [ N ( y |W2h1 + b2, σ2y )] ,
where N (·) denotes the normal distribution with mean W2h1 + b2 and (fixed) variance σ2y . Therefore, P (y | x) can express a very complex, multi-modal distribution since it is a mixture of exponentially many normal distributions. The multi-layer extension is straightforward via a combination of stochastic and deterministic hidden layers, e.g., see Tang & Salakhutdinov (2013), Raiko et al. (2014). Furthermore, one can use any other output distributions as like DNN, e.g., softmax for classification tasks.
There are two computational issues for training SFNN: computing expectations with respect to stochastic units in forward pass and computing gradients in backward pass. One can notice that both are computationally intractable since they require summations over exponentially many configurations of all stochastic units. First, in order to handle the issue in forward pass, one can use the follow-
ing Monte Carlo approximation for estimating the expectation: P (y | x) w 1M M∑ m=1 P (y | h(m)),
where h(m) ∼ P ( h1 | x ) and M is the number of samples. This random estimator is unbiased and has relatively low variance (Tang & Salakhutdinov, 2013) since its accuracy does not depend on the dimensionality of h1 and one can draw samples from the exact distribution. Next, in order to handle the issue in backward pass, Neal (1990) proposed a Gibbs sampling, but it is known that it often mixes poorly. Saul et al. (1996) proposed a variational learning based on the mean-field approximation, but it has additional parameters making the variational lower bound looser. More recently, several other techniques have been proposed including unbiased estimators of the variational bound using importance sampling (Tang & Salakhutdinov, 2013; Raiko et al., 2014) and biased/unbiased estimators of the gradient for approximating backpropagation (Bengio et al., 2013; Raiko et al., 2014; Gu et al., 2015).
2.2 SIMPLE TRANSFORMATION FROM SIGMOID-DNN AND RELU-DNN TO SFNN
Despite the recent advances, training SFNN is still very slow compared to DNN due to the sampling procedures: in particular, it is notoriously hard to train SFNN when the network structure is deeper and wider. In order to handle these issues, we consider the following approximation:
P (y | x) = EP (h1|x) [ N ( y |W2h1 + b2, σ2y )] w N ( y | EP (h1|x) [ W2h1 ] + b2, σ2y ) = N ( y |W2σ ( W1x+ b1 ) + b2, σ2y ) . (2)
Note that the above approximation corresponds to replacing stochastic units by deterministic ones such that their hidden activation values are same as marginal distributions of stochastic units, i.e., SFNN can be approximated by DNN using sigmoid activation functions, say sigmoid-DNN. When there exist more latent layers above the stochastic one, one has to apply similar approximations to all of them, i.e., exchanging the orders of expectations and non-linear functions, for making DNN and SFNN are equivalent. Therefore, instead of training SFNN directly, one can try transferring pretrained parameters of sigmoid-DNN to those of the corresponding SFNN directly: train sigmoidDNN instead of SFNN, and replace deterministic units by stochastic ones for the inference purpose. Although such a strategy looks somewhat ‘rude’, it was often observed in the literature that it reasonably works well for SFNN (Raiko et al., 2014) and we also evaluate it as reported in Table 1. We also note that similar approximations appear in the context of dropout: it trains a stochastic model averaging exponentially many DNNs sharing parameters, but also approximates a single DNN well.
Now we investigate a similar transformation in the case when DNN uses the unbounded ReLU activation function, say ReLU-DNN. Many recent deep networks are of ReLU-DNN type due to the gradient vanishing problem, and their pre-trained parameters are often available. Although it is straightforward to build SFNN from sigmoid-DNN, it is less clear in this case since ReLU is
unbounded. To handle this issue, we redefine the stochastic latent units of SFNN:
P ( h1 | x ) = N1∏ i=1 P ( h1i | x ) , where P ( h1i = 1 | x ) = min { αf ( W1ix+ b 1 i ) , 1 } . (3)
In the above, f(x) = max{x, 0} is the ReLU activation function and α is some hyper-parameter. A simple transformation can be defined similarly as the case of sigmoid-DNN via replacing deterministic units by stochastic ones. However, to preserve the parameter information of ReLU-DNN, one has to choose α such that αf ( W1ix+ b 1 i ) ≤ 1 and rescale upper parameters W2 as follows:
α−1 ← max i,x ∣∣∣f (Ŵ1ix+ b̂1i)∣∣∣ , (W1, b1)← (Ŵ1, b̂1) , (W2, b2)← ( Ŵ2/α, b̂2) . (4) Then, applying similar approximations as in (2), i.e., exchanging the orders of expectations and non-linear functions, one can observe that ReLU-DNN and SFNN are equivalent.
We evaluate the performance of the simple transformations from DNN to SFNN on the MNIST dataset (LeCun et al., 1998) and the synthetic dataset (Bishop, 1994), where the former and the latter are popular datasets used for a classification task and a multi-modal (i.e., one-to-many mappings) prediction learning, respectively. In all experiments reported in this paper, we commonly use the softmax and Gaussian with standard deviation of σy = 0.05 are used for the output probability on classification and regression tasks, respectively. The only first hidden layer of DNN is replaced by stochastic one, and we use 500 samples for estimating the expectations in the SFNN inference. As reported in Table 1, we observe that the simple transformation often works well for both tasks: the SFNN and sigmoid-DNN inferences (using same parameters trained by sigmoid-DNN) perform similarly for the classification task and the former significantly outperforms for the latter for the
multi-modal task (also see Figure 1). It might suggest some possibilities that the expensive SFNN training might not be not necessary, depending on the targeted learning quality. However, in case of ReLU, SFNN performs much worse than ReLU-DNN for the MNIST classification task under the parameter transformation.
3 TRANSFORMATION FROM DNN TO SFNN VIA SIMPLIFIED-SFNN
In this section, we propose an advanced method to utilize the pre-trained parameters of DNN for training SFNN. As shown in the previous section, simple parameter transformations from DNN to SFNN are not clear to work in general, in particular for activation functions other than sigmoid. Moreover, training DNN does not utilize the stochastic regularizing effect, which is an important benefit of SFNN. To address the issues, we design an intermediate model, called Simplified-SFNN. The proposed model is a special form of stochastic neural networks, which approximates certain SFNN by simplifying its upper latent units above stochastic ones. Then, we establish more rigorous connections between three models: DNN → Simplified-SFNN → SFNN, which leads to an efficient training procedure of the stochastic models utilizing pre-trained parameters of DNN. In our experiments, we evaluate the strategy for various tasks and popular DNN architectures.
3.1 SIMPLIFIED-SFNN OF TWO HIDDEN LAYERS AND NON-NEGATIVE ACTIVATION FUNCTIONS
For clarity of presentation, we first introduce Simplified-SFNN with two hidden layers and nonnegative activation functions, where its extensions to multiple layers and general activation functions are presented in Appendix B. We also remark that we primarily describe fully-connected SimplifiedSFNNs, but their convolutional versions can also be naturally defined. In Simplified-SFNN of two hidden layers, we assume that the first and second hidden layers consist of stochastic binary hidden units and deterministic ones, respectively. As like (3), the first layer is defined as a binary random vector with N1 units, i.e., h1 ∈ {0, 1}N1 , drawn under the following distribution:
P ( h1 | x ) = N1∏ i=1 P ( h1i | x ) , where P ( h1i = 1 | x ) = min { α1f ( W1ix+ b 1 i ) , 1 } . (5)
where x is the input vector, α1 > 0 is a hyper-parameter for the first layer, and f : R→ R+ is some non-negative non-linear activation function with |f ′(x)| ≤ 1 for all x ∈ R, e.g., ReLU and sigmoid activation functions. Now the second layer is defined as the following deterministic vector with N2 units, i.e., h2(x) ∈ RN2 : h2 (x) = [ f ( α2 ( EP (h1|x) [ s ( W2jh 1 + b2j )] − s (0) )) : ∀j ∈ N2 ] , (6)
where α2 > 0 is a hyper-parameter for the second layer and s : R → R is a differentiable function with |s′′(x)| ≤ 1 for all x ∈ R, e.g., sigmoid and tanh functions. In our experiments, we use the sigmoid function for s(x). Here, one can note that the proposed model also has the same computational issues with SFNN in forward and backward passes due to the complex expectation. One can train Simplified-SFNN similarly as SFNN: we use Monte Carlo approximation for estimating the expectation and the (biased) estimator of the gradient for approximating backpropagation inspired by Raiko et al. (2014) (more detailed explanation is presented in Appendix A).
We are interested in transferring parameters of DNN to Simplified-SFNN to utilize the training benefits of DNN since the former is much faster to train than the latter. To this end, we consider the following DNN of which `-th hidden layer is deterministic and defined as follows:
ĥ` (x) = [ ĥ`i (x) = f ( Ŵ`i ĥ `−1 (x) + b̂`i ) : i ∈ N ` ] , (7)
where ĥ0(x) = x. As stated in the following theorem, we establish a rigorous way how to initialize parameters of Simplified-SFNN in order to transfer the knowledge stored in DNN.
Theorem 1 Assume that both DNN and Simplified-SFNN with two hidden layers have same network structure with non-negative activation function f . Given parameters {Ŵ`, b̂` : ` = 1, 2} of DNN and input dataset D, choose those of Simplified-SFNN as follows:(
α1,W 1, b1 ) ← ( 1
γ1 , Ŵ1, b̂1
) , ( α2,W 2, b2 ) ← ( γ2γ1 s′ (0) , 1 γ2 Ŵ2, 1 γ1γ2 b̂2 ) , (8)
where γ1 = max
i,x∈D
∣∣∣f (Ŵ1ix+ b̂1i)∣∣∣ and γ2 > 0 is any positive constant. Then, it follows that
∣∣∣h2j (x)− ĥ2j (x)∣∣∣ ≤ γ1 (∑ i ∣∣∣Ŵ 2ij∣∣∣+ b̂2jγ−11 )2 2s′ (0) γ2 , ∀j,x ∈ D.
The proof of the above theorem is presented in Appendix D.1. Our proof is built upon the first-order Taylor expansion of non-linear function s(x). Theorem 1 implies that one can make Simplified-SFNN represent the function values of DNN with bounded errors using a linear transformation. Furthermore, the errors can be made arbitrarily small by choosing large γ2, i.e., lim γ2→∞
∣∣∣h2j (x)− ĥ2j (x)∣∣∣ = 0, ∀j,x ∈ D. Figure 2(c) shows that knowledge transferring loss decreases as γ2 increases on MNIST classification. Based on this, we choose γ2 = 50 commonly for all experiments.
3.2 WHY SIMPLIFIED-SFNN ?
Given a Simplified-SFNN model, the corresponding SFNN can be naturally defined by taking out the expectation in (6). As illustrated in Figure 2(a), the main difference between SFNN and SimplifiedSFNN is that the randomness of the stochastic layer propagates only to its upper layer in the latter, i.e., the randomness of h1 is averaged out at its upper units h2 and does not propagate to h3 or output y. Hence, Simplified-SFNN is no longer a Bayesian network. This makes training Simplified-SFNN much easier than SFNN since random samples are not required at some layers1 and consequently the quality of gradient estimations can also be improved, in particular for unbounded activation functions. Furthermore, one can use the same approximation procedure (2) to see that SimplifiedSFNN approximates SFNN. However, since Simplified-SFNN still maintains binary random units, it uses approximation steps later, in comparison with DNN. In summary, Simplified-SFNN is an intermediate model between DNN and SFNN, i.e., DNN→ Simplified-SFNN→ SFNN. The above connection naturally suggests the following training procedure for both SFNN and Simplified-SFNN: train a baseline DNN first and then fine-tune its corresponding Simplified-SFNN initialized by the transformed DNN parameters. Finally, the fine-tuned parameters can be used for SFNN as well. We evaluate the strategy for the MNIST classification, which is reported in Table 2 (see Appendix C for more detailed experiment setups). We found that SFNN under the two-stage training always performs better than SFNN under a simple transformation (4) from ReLU-DNN.
1 For example, if one replaces the first feature maps in the fifth residual unit of Pre-ResNet having 164 layers (He et al., 2016) by stochastic ones, then the corresponding DNN, Simplified-SFNN and SFNN took 1 mins 35 secs, 2 mins 52 secs and 16 mins 26 secs per each training epoch, respectively, on our machine with one Intel CPU (Core i7-5820K 6-Core@3.3GHz) and one NVIDIA GPU (GTX Titan X, 3072 CUDA cores). Here, we trained both stochastic models using the biased estimator (Raiko et al., 2014) with 10 random samples on CIFAR-10 dataset.
More interestingly, Simplified-SFNN consistently outperforms its baseline DNN due to the stochastic regularizing effect, even when we train both models using dropout (Hinton et al., 2012b) and batch normalization (Ioffe & Szegedy, 2015). In order to confirm the regularization effects, one can again approximate a trained Simplified-SFNN by a new deterministic DNN which we call DNN∗ and is different from its baseline DNN under the following approximation at upper latent units above binary random units:
EP (h`|x) [ s ( W`+1j h ` )] w s ( EP (h`|x) [ W`+1j h ` ]) = s (∑ i W `+1ij P ( h`i = 1 | x )) . (9)
We found that DNN∗ using fined-tuned parameters of Simplified-SFNN also outperforms the baseline DNN as shown in Table 2 and Figure 2(b).
3.3 EXPERIMENTAL RESULTS ON MULTI-MODAL LEARNING AND CONVOLUTIONAL NETWORKS
We present several experimental results for both multi-modal and classification tasks on MNIST (LeCun et al., 1998), Toronto Face Database (TFD) (Susskind et al., 2010), CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009) and SVHN (Netzer et al., 2011). Here, we present some key results due to the space constraints and more detailed explanations for our experiment setups are presented in Appendix C.
We first verify that it is possible to learn one-to-many mapping via Simplified-SFNN on the TFD and MNIST datasets, where the former and the latter are used to predict multiple facial expressions from the mean of face images per individual and the lower half of the MNIST digit given the upper half, respectively. We remark that both tasks are commonly performed in recent other works to test the multi-modal learning using SFNN (Raiko et al., 2014; Gu et al., 2015). In all experiments, we first train a baseline DNN, and the trained parameters of DNN are used for further fine-tuning those of Simplified-SFNN. As shown in Table 3 and Figure 3, stochastic models outperform their baseline DNN, and generate different digits for the case of ambiguous inputs (while DNN does not). We also evaluate the regularization effect of Simplified-SFNN for the classification tasks on CIFAR-10, CIFAR-100 and SVHN. Table 4 reports the classification error rates using convolutional neural networks such as Lenet-5 (LeCun et al., 1998), NIN (Lin et al., 2014) and WRN (Zagoruyko & Komodakis, 2016). Due to the regularization effects, Simplified-SFNNs consistently outperform
their baseline DNNs. For example, WRN∗ outperforms WRN by 0.08% on CIFAR-10 and 0.58% on CIFAR-100. We expect that if one introduces more stochastic layers, the error would be decreased more (see Figure 4), but it increases the fine-tuning time-complexity of Simplified-SFNN.
4 CONCLUSION
In order to develop an efficient training method for large-scale SFNN, this paper proposes a new intermediate stochastic model, called Simplified-SFNN. We establish the connection between three models, i.e., DNN → Simplified-SFNN → SFNN, which naturally leads to an efficient training procedure of the stochastic models utilizing pre-trained parameters of DNN. This connection naturally leads an efficient training procedure of the stochastic models utilizing pre-trained parameters and architectures of DNN. We believe that our work brings a new important direction for training stochastic neural networks, which should be of broader interest in many related applications.
A TRAINING SIMPLIFIED-SFNN
The parameters of Simplified-SFNN can be learned using a variant of the backpropagation algorithm (Rumelhart et al., 1988) in a similar manner to DNN. However, in contrast to DNN, there are two computational issues for simplified-SFNN: computing expectations with respect to stochastic units in forward pass and computing gradients in back pass. One can notice that both are intractable since they require summations over all possible configurations of all stochastic units. First, in order to handle the issue in forward pass, we use the following Monte Carlo approximation for estimating the expectation:
EP (h1|x) [ s ( W2jh 1 + b2j )] w 1
M M∑ m=1 s ( W2jh (m) + b2j ) , h(m) ∼ P ( h1 | x ) ,
where M is the number of samples. This random estimator is unbiased and has relatively low variance (Tang & Salakhutdinov, 2013) since its accuracy does not depend on the dimensionality of h1 and one can draw samples from the exact distribution. Next, in order to handle the issue in back pass, we use the following approximation inspired by (Raiko et al., 2014):
∂
∂W2j EP (h1|x)
[ s ( W2jh 1 + b2j )] w 1
M ∑ m ∂ ∂W2j s ( W2jh (m) + b2j ) ,
∂
∂W1i EP (h1|x)
[ s ( W2jh 1 + b2j )] w W 2ij M ∑ m s′ ( W2jh (m) + b2j ) ∂ ∂W1i P ( h1i = 1 | x ) ,
where h(m) ∼ P ( h1 | x ) and M is the number of samples. In our experiments, we commonly choose M = 20.
B EXTENSIONS OF SIMPLIFIED-SFNN
In this section, we describe how the network knowledge transferring between Simplified-SFNN and DNN, i.e., Theorem 1, generalizes to multiple layers and general activation functions.
B.1 EXTENSION TO MULTIPLE LAYERS
A deeper Simplified-SFNN with L hidden layers can be defined similarly as the case of L = 2. We also establish network knowledge transferring between Simplified-SFNN and DNN with L hidden layers as stated in the following theorem. Here, we assume that stochastic layers are not consecutive for simpler presentation, but the theorem is generalizable for consecutive stochastic layers.
Theorem 2 Assume that both DNN and Simplified-SFNN with L hidden layers have same network structure with non-negative activation function f . Given parameters {Ŵ`, b̂` : ` = 1, . . . , L} of DNN and input dataset D, choose the same ones for Simplified-SFNN initially and modify them for each `-th stochastic layer and its upper layer as follows:
α` ← 1 γ` , ( α`+1,W `+1, b`+1 ) ← ( γ`γ`+1 s′ (0) , Ŵ`+1 γ`+1 , b̂`+1 γ`γ`+1 ) ,
where γ` = max i,x∈D ∣∣∣f (Ŵ`ih`−1(x) + b̂`i)∣∣∣ and γ`+1 is any positive constant. Then, it follows that lim
γ`+1→∞ ∀ stochastic hidden layer ` ∣∣∣hLj (x)− ĥLj (x)∣∣∣ = 0, ∀j,x ∈ D. The above theorem again implies that it is possible to transfer knowledge from DNN to SimplifiedSFNN by choosing large γl+1. The proof of Theorem 2 is similar to that of Theorem 1 and given in Appendix D.2.
B.2 EXTENSION TO GENERAL ACTIVATION FUNCTIONS
In this section, we describe an extended version of Simplified-SFNN which can utilize any activation function. To this end, we modify the definitions of stochastic layers and their upper layers by introducing certain additional terms. If the `-th hidden layer is stochastic, then we slightly modify the original definition (5) as follows:
P ( h` | x ) = N`∏ i=1 P ( h`i | x ) with P ( h`i = 1 | x ) = min { α`f ( W1ix+ b 1 i + 1 2 ) , 1 } ,
where f : R → R is a non-linear (possibly, negative) activation function with |f ′(x)| ≤ 1 for all x ∈ R. In addition, we re-define its upper layer as follows:
h`+1 (x) = [ f ( α`+1 ( EP (h`|x) [ s ( W`+1j h ` + b`+1j )] − s (0)−s ′ (0)
2 ∑ i W `+1ij
)) : ∀j ] ,
where h0(x) = x and s : R→ R is a differentiable function with |s′′(x)| ≤ 1 for all x ∈ R. Under this general Simplified-SFNN model, we also show that transferring network knowledge from DNN to Simplified-SFNN is possible as stated in the following theorem. Here, we again assume that stochastic layers are not consecutive for simpler presentation.
Theorem 3 Assume that both DNN and Simplified-SFNN with L hidden layers have same network structure with non-linear activation function f . Given parameters {Ŵ`, b̂` : ` = 1, . . . , L} of DNN and input dataset D, choose the same ones for Simplified-SFNN initially and modify them for each `-th stochastic layer and its upper layer as follows:
α` ← 1 2γ` , ( α`+1,W `+1, b`+1 ) ← ( 2γ`γ`+1 s′(0) , Ŵ`+1 γ`+1 , b̂`+1 2γ`γ`+1 ) ,
where γ` = max i,x∈D ∣∣∣f (Ŵ`ih`−1(x) + b̂`i)∣∣∣, and γ`+1 is any positive constant. Then, it follows that lim
γ`+1→∞ ∀ stochastic hidden layer ` ∣∣∣hLj (x)− ĥLj (x)∣∣∣ = 0, ∀j,x ∈ D. We omit the proof of the above theorem since it is somewhat direct adaptation of that of Theorem 2.
C EXPERIMENTAL SETUPS
In this section, we describe detailed explanation about all the experiments described in Section 3. In all experiments, the softmax and Gaussian with the standard deviation of 0.05 are used as the output probability for the classification task and the multi-modal prediction, respectively. The loss was minimized using ADAM learning rule (Kingma & Ba, 2014) with a mini-batch size of 128. We used an exponentially decaying learning rate.
C.1 CLASSIFICATION ON MNIST
The MNIST dataset consists of 28 × 28 pixel greyscale images, each containing a digit 0 to 9 with 60,000 training and 10,000 test images. For this experiment, we do not use any data augmentation or pre-processing. Hyper-parameters are tuned on the validation set consisting of the last 10,000 training images. All Simplified-SFNNs are constructed by replacing the first hidden layer of a baseline DNN with stochastic hidden layer. As described in Section 3.2, we train Simplified-SFNNs under the two-stage procedure: first train a baseline DNN for first 200 epochs, and the trained parameters of DNN are used for initializing those of Simplified-SFNN. For 50 epochs, we train simplified-SFNN. We choose the hyper-parameter γ2 = 50 in the parameter transformation. All Simplified-SFNNs are trained with M = 20 samples at each epoch, and in the test, we use 500 samples.
C.2 MULTI-MODAL REGRESSION ON TFD AND MNIST
The Toronto Face Database (TFD) (Susskind et al., 2010) dataset consists of 48×48 pixel greyscale images, each containing a face image of 900 individuals with 7 different expressions. Similar to (Raiko et al., 2014), we use 124 individuals with at least 10 facial expressions as data. We randomly choose 100 individuals with 1403 images for training and the remaining 24 individuals with 326 images for the test. We take the mean of face images per individual as the input and set the output as the different expressions of the same individual. The MNIST dataset consists of 28 × 28 pixel greyscale images, each containing a digit 0 to 9 with 60,000 training and 10,000 test images. For this experiments, each pixel of every digit images is binarized using its grey-scale value. We take the upper half of the MNIST digit as the input and set the output as the lower half of it. All SimplifiedSFNNs are constructed by replacing the first hidden layer of a baseline DNN with stochastic hidden layer. We train Simplified-SFNNs with M = 20 samples at each epoch, and in the test, we use 500 samples. We use 200 hidden units for each layer of neural networks in two experiments. Learning rate is chosen from {0.005 , 0.002, 0.001, ... , 0.0001} , and the best result is reported for both tasks.
C.3 CLASSIFICATION ON CIFAR-10, CIFAR-100 AND SVHN
The CIFAR-10 and CIFAR-100 datasets consist of 50,000 training and 10,000 test images. The SVHN dataset consists of 73,257 training and 26,032 test images.2 We pre-process the data using global contrast normalization and ZCA whitening. For these datasets, we design a convolutional version of Simplified-SFNN. In a similar manner to the case of fully-connected networks, one can define a stochastic convolution layer, which considers the input feature map as a binary random matrix and generates the output feature map as defined in (6). All Simplified-SFNNs are constructed by replacing a hidden feature map of a baseline models, i.e., Lenet-5, NIN and WRN, with stochastic one as shown in Figure 5(d). We use WRN with 16 and 28 layers for SVHN and CIFAR datasets, respectively, since they showed state-of-the-art performance as reported by Zagoruyko & Komodakis (2016). In case of WRN, we introduce up to two stochastic convolution layers.For 100 epochs, we first train baseline models, i.e., Lenet-5, NIN and WRN, and trained parameters are used for initializing those of Simplified-SFNNs. All Simplified-SFNNs are trained with M = 5 samples and the test error is only measured by the approximation (9). The test errors of baseline models are measured after training them for 200 epochs similar to Zagoruyko & Komodakis (2016).
D PROOFS OF THEOREMS
D.1 PROOF OF THEOREM 1
First consider the first hidden layer, i.e., stochastic layer. Let γ1 = max i,x∈D
f ( Ŵ1ix+ b̂ 1 i ) be
the maximum value of hidden units in DNN. If we initialize the parameters ( α1,W 1, b1 ) ←(
1 γ1 , Ŵ1, b̂1
) , then the marginal distribution of each hidden unit i becomes
P ( h1i = 1 | x,W1,b1 ) = min { α1f ( Ŵ1ix+ b̂ 1 i ) , 1 } = 1 γ1 f ( Ŵ1ix+ b̂ 1 i ) , ∀i,x ∈ D. (10)
2We do not use the extra SVHN dataset for training.
Next consider the second hidden layer. From Taylor’s theorem, there exists a value z between 0 and x such that s(x) = s(0) + s′(0)x + R(x), where R(x) = s ′′(z)x2
2! . Since we consider a binary random vector, i.e., h1 ∈ {0, 1}N1 , one can write EP (h1|x) [ s ( βj ( h1 ))]
= ∑ h1 ( s (0) + s′ (0)βj ( h1 ) +R ( βj ( h1 ))) P ( h1 | x ) = s (0) + s′ (0)
(∑ i W 2ijP (h 1 i = 1 | x) + b2j ) + EP (h1|x) [ R(βj(h 1)) ] ,
where βj ( h1 ) := W2jh
1 + b2j is the incoming signal. From (6) and (10), for every hidden unit j, it follows that
h2j ( x;W2,b2 ) = f ( α2 ( s′(0) ( 1
γ1 ∑ i W 2ij ĥ 1 i (x) + b 2 j
) + EP (h1|x) [ R ( βj ( h1 ))])) .
Since we assume that |f ′(x)| ≤ 1, the following inequality holds:∣∣∣∣∣h2j (x;W2,b2)− f ( α2s ′(0) ( 1
γ1 ∑ i W 2ij ĥ 1 i (x) + b 2 j ))∣∣∣∣∣ ≤ ∣∣α2EP (h1|x) [R(βj(h1))]∣∣ ≤ α2
2 EP (h1|x)
[( W2jh 1 + b2j )2] ,
where we use |s′′(z)| < 1 for the last inequality. Therefore, it follows that ∣∣∣h2j (x;W2,b2)− ĥ2j (x;Ŵ2, b̂2)∣∣∣ ≤ γ1 (∑ i ∣∣∣Ŵ 2ij∣∣∣+ b̂2jγ−11 )2 2s′(0)γ2 , ∀j,
since we set ( α2,W 2, b2 ) ← ( γ2γ1 s′(0) , Ŵ2 γ2 , γ−11 γ2 b̂2 ) . This completes the proof of Theorem 1.
D.2 PROOF OF THEOREM 2
For the proof of Theorem 2, we first state the two key lemmas on error propagation in SimplifiedSFNN.
Lemma 4 Assume that there exists some positive constant B such that∣∣∣h`−1i (x)− ĥ`−1i (x)∣∣∣ ≤ B, ∀i,x ∈ D, and the `-th hidden layer of NCSFNN is standard deterministic layer as defined in (7). Given parameters {Ŵ`, b̂`} of DNN, choose same ones for NCSFNN. Then, the following inequality holds:∣∣∣h`j (x)− ĥ`j (x)∣∣∣ ≤ BN `−1Ŵ `max, ∀j,x ∈ D. where Ŵ `max = max
ij ∣∣∣Ŵ `ij∣∣∣. Proof. See Appendix D.3.
Lemma 5 Assume that there exists some positive constant B such that∣∣∣h`−1i (x)− ĥ`−1i (x)∣∣∣ ≤ B, ∀i,x ∈ D, and the `-th hidden layer of simplified-SFNN is stochastic layer. Given parameters {Ŵ`,Ŵ`+1, b̂`, b̂`+1} of DNN, choose those of Simplified-SFNN as follows:
α` ← 1 γ` , ( α`+1,W `+1, b`+1 ) ← ( γ`γ`+1 s′ (0) , Ŵ`+1 γ`+1 , b̂`+1 γ`γ`+1 ) ,
where γ` = max j,x∈D ∣∣∣f (Ŵ`jh`−1(x) + b̂`j)∣∣∣ and γ`+1 is any positive constant. Then, it follows that ∣∣∣h`+1k (x)− ĥ`+1k (x)∣∣∣ ≤ BN `−1N `Ŵ `maxŴ `+1max + ∣∣∣∣∣∣∣ γ` ( N `Ŵ `+1max + b̂ `+1 maxγ −1 ` )2 2s′(0)γ`+1
∣∣∣∣∣∣∣ , ∀k,x ∈ D, where b̂`max = max
j ∣∣∣̂b`j∣∣∣ and Ŵ `max = max ij ∣∣∣Ŵ `ij∣∣∣. Proof. See Appendix D.4.
Assume that `-th layer is first stochastic hidden layer in Simplified-SFNN. Then, from Theorem 1, we have
∣∣∣h`+1j (x)− ĥ`+1j (x)∣∣∣ ≤ ∣∣∣∣∣∣∣ γ` ( N `Ŵ `+1max + b̂ `+1 maxγ −1 ` )2 2s′(0)γ`+1 ∣∣∣∣∣∣∣ , ∀j,x ∈ D. (11)
According to Lemma 4 and 5, the final error generated by the right hand side of (11) is bounded by
τ`γ` ( N `Ŵ `+1max + b̂ `+1 maxγ −1 ` )2 2s′ (0) γ`+1 , (12)
where τ` = L∏
`′=l+2
( N ` ′−1Ŵ ` ′
max
) . One can note that every error generated by each stochastic layer
is bounded by (12). Therefore, it follows that
∣∣∣hLj (x)− ĥLj (x)∣∣∣ ≤ ∑ `:stochastic hidden layer
τ`γ` ( N `Ŵ `+1max + b̂ `+1 maxγ −1 ` )2 2s′ (0) γ`+1  , ∀j,x ∈ D. From above inequality, we can conclude that
lim γ`+1→∞
∀ stochastic hidden layer ` ∣∣∣hLj (x)− ĥLj (x)∣∣∣ = 0, ∀j,x ∈ D. This completes the proof of Theorem 2.
D.3 PROOF OF LEMMA 4
From assumption, there exists some constant i such that | i| < B and
h`−1i (x) = ĥ `−1 i (x) + i, ∀i,x.
By definition of standard deterministic layer, it follows that
h`j (x) = f (∑ i Ŵ `ijh `−1 i (x) + b̂ `−1 j ) = f (∑ i Ŵ `ij ĥ `−1 i (x) + ∑ i Ŵ `ij i + b̂ ` j ) .
Since we assume that |f ′(x)| ≤ 1, one can conclude that∣∣∣∣∣h`j (x)− f (∑
i
Ŵ `ij ĥ `−1 i (x) + b̂ ` j )∣∣∣∣∣ ≤ ∣∣∣∣∣∑ i Ŵ `ij i ∣∣∣∣∣ ≤ B ∣∣∣∣∣∑ i Ŵ `ij ∣∣∣∣∣ ≤ BN `−1Ŵ `max. This completes the proof of Lemma 4.
D.4 PROOF OF LEMMA 5
From assumption, there exists some constant `−1i such that ∣∣ `−1i ∣∣ < B and
h`−1i (x) = ĥ `−1 i (x) + `−1 i , ∀i,x. (13)
Let γ` = max j,x∈D ∣∣∣f (Ŵ`jh`−1(x) + b̂`j)∣∣∣ be the maximum value of hidden units. If we initialize the parameters ( α`,W `, b` ) ← (
1 γ` , Ŵ`, b̂`
) , then the marginal distribution becomes
P ( h`j = 1 | x,W`,b` ) = min { α`f ( Ŵ`jh `−1 (x) + b̂`j ) , 1 } = 1 γ` f ( Ŵ`jh `−1 (x) + b̂`j ) , ∀j,x.
From (13), it follows that
P ( h`j = 1 | x,W`,b` ) = 1
γ` f
( Ŵ`jĥ `−1 (x) + ∑ i Ŵ `ij `−1 i + b̂ ` j ) , ∀j,x.
Similar to Lemma 4, there exists some constant `j such that ∣∣ `j∣∣ < BN `−1Ŵ `max and
P ( h`j = 1 | x,W`,b` ) = 1
γ`
( ĥ`j (x) + ` j ) , ∀j,x. (14)
Next, consider the upper hidden layer of stochastic layer. From Taylor’s theorem, there exists a value z between 0 and t such that s(x) = s(0) + s′(0)x+ R(x), where R(x) = s ′′(z)x2
2! . Since we consider a binary random vector, i.e., h` ∈ {0, 1}N` , one can write
EP (h`|x)[s(βk(h`))] = ∑ h` ( s(0) + s′(0)βk(h `) +R ( βk(h `) )) P (h` | x)
= s(0) + s′(0) ∑ j W `+1jk P (h ` j = 1 | x) + b`+1k +∑ h` R(βk(h `))P (h` | x),
where βk(h`) = W`+1k h ` + b`+1k is the incoming signal. From (14) and above equation, for every hidden unit k, we have
h`+1k (x;W `+1,b`+1)
= f α`+1 s′(0)  1 γ` ∑ j W `+1jk ĥ ` j(x) + ∑ j W `+1jk ` j + b`+1k + EP (h`|x) [R(βk(h`))]  . Since we assume that |f ′(x)| < 1, the following inequality holds:∣∣∣∣∣∣h`+1k (x;W`+1,b`+1)− f α`+1s′(0)  1 γ` ∑ j W `+1ij ĥ ` j(x) + b `+1 j
∣∣∣∣∣∣ ≤ ∣∣∣∣∣∣α`+1s ′(0) γ` ∑ j W `+1jk ` j + α`+1EP (h`|x) [ R(βk(h `)) ]∣∣∣∣∣∣
≤ ∣∣∣∣∣∣α`+1s ′(0) γ` ∑ j W `+1jk ` j ∣∣∣∣∣∣+ ∣∣∣α`+1 2 EP (h`|x) [( W`+1k h ` + b`+1k )2]∣∣∣ , (15)
where we use |s′′(z)| < 1 for the last inequality. Therefore, it follows that ∣∣∣h`+1k (x)− ĥ`+1k (x)∣∣∣ ≤ BN `−1N `Ŵ `maxŴ `+1max + ∣∣∣∣∣∣∣ γ` ( N `Ŵ `+1max + b̂ `+1 maxγ −1 ` )2 2s′(0)γ`+1 ∣∣∣∣∣∣∣ , since we set ( α`+1,W `+1, b`+1 ) ← ( γ`+1γ` s′(0) , Ŵ`+1 γ`+1 , γ−1` b̂ `+1 γ`+1 ) . This completes the proof of Lemma 5.

Update: Because no revision of the paper has been provided by the authors, I am reducing my rating to "marginally below acceptance".
----------
This paper addresses the problem of training stochastic feedforward neural networks.  It proposes to transfer weights from a deterministic deep neural network trained using standard procedures (including techniques such as dropout and batch normalization) to a stochastic network having the same topology.  The initial mechanism described for performing the transfer involves a rescaling of unit inputs and layer weights, and appropriate specification of the stochastic latent units if the DNN used for pretraining employs ReLU nonlinearities.  Initial experiments on MNIST classification and a toy generative task with a multimodal target distribution show that the simple transfer process works well if the DNN used for pretraining uses sigmoid nonlinearities, but not if the pretraining DNN uses ReLUs.  To tackle this problem, the paper introduces the "simplified stochastic feedforward neural network," in which every stochastic layer is followed by a layer that takes an expectation over samples from its input, thus limiting the propagation of stochasticity in the network.  A modified process for transferring weights from a pretraining DNN to the simplified SFNN is described and justified.  The training process then occurs in three steps:  (1) pretrain a DNN, (2) transfer weights from the DNN to a simplified SFNN and continue training, and (3) optionally transfer the weights to a full SFNN and continue training or transfer them to a deterministic model (called DNN*) and continue training.  The third step can be skipped and the simplified SFNN may also be used directly as an inference model.  Experimental results on MNIST classification show that the use of simplified SFNN training can improve a deterministic DNN* model over a DNN baseline trained with batch normalization and dropout.  Experiments on two generative tasks (MNIST-half and the Toronto Faces Database) show that the proposed pretraining process improves test set negative log-likelihoods.  Finally, experiments on CIFAR-10, CIFAR-100, and SVHN with the LeNet-5, network-in-network, and wide residual network architectures show that use of a stochastic training step can improve performance of a deterministic (DNN*) model.
It is a bit confusing to refer to "multi-modal" tasks, when what is meant is "generative tasks with a multimodal target distribution" because "multi-modal" task can also refer to a learning task that crosses sensory modalities such as audio-visual speech recognition, text-based image retrieval, or image captioning.  I recommend that you use the more precise term ("generative tasks with a multimodal target distribution") early in the introduction and then say that you will refer to such tasks as "multi-modal tasks" in the rest of the paper for the sake of brevity.
The paper would be easier to read if "SFNN" were not used to refer to both the singular ("stochastic feedforward neural network") and plural ("stochastic feedforward neural networks") cases.  When the plural is meant, write "SFNNs".
In Table 1, why does the 3 hidden layer SFNN initialized from a ReLU DNN have so much worse of a test NLL than the 2 hidden layer SFNN initialized from a ReLU DNN?
The notation that uses superscripts to indicate layer indexes is confusing.  The reader naturally parses N² as "N squared" and not as "the number of units in the second layer."
When you transfer weights back from the simplified SFNN to the DNN* model, do you need to perform some sort of rescaling that undoes the operations in Equation (8) in the paper?
What does NCSFNN stand for in the supplementary material?
Pros
+ The proposed model is easy to implement and apply to other tasks.
+ The MNIST results showing that the stochastic model training can produce a deterministic model (called DNN* in the paper) that generalizes better than a DNN trained with batch normalization and dropout is quite exciting.
Cons
- For the reasons outlined above, the paper is at times a bit hard to follow.
- The results CIFAR-10, CIFAR-100, and SVHN would be more convincing if the baselines used dropout and batch normalization.  While this is shown on MINST, demonstration of a similar result on a more challenging task would strengthen the paper.
Minor issues
It has been believed that stochastic → It is believed that stochastic
underlying these successes is on the efficient training methods → underlying these successes is efficient training methods
necessary in order to model complex stochastic natures in many real-world tasks → necessary in to model the complex stochastic nature of many real-world tasks
structured prediction, image generation and memory networks : memory networks are models, not tasks.
Furthermore, it has been believed that SFNN → Furthermore, it is believed that SFNN
using backpropagation under the variational techniques and the reparameterization tricks  → using backpropagation with variational techniques and reparameterization tricks
There have been several efforts developing efficient training methods → There have been several efforts toward developing efficient training methods
However, training SFNN is still significantly slower than doing DNN → However, training a SFNN is still significantly slower than training a DNN
e.g., most prior works on this line have considered a → consequently most prior works in this area have considered a
Instead of training SFNN directly → Instead of training a SFNN directly
whether pre-trained parameters of DNN → whether pre-trained parameters from a DNN
with further fine-tuning of light cost → with further low-cost fine-tuning
recent advances in DNN on its design and training → recent advances in DNN design and training
it is rather believed that transferring parameters →  it is believed that transferring parameters
but the opposite direction is unlikely possible → but the opposite is unlikely
To address the issues, we propose → To address these issues, we propose
which intermediates between SFNN and DNN, → which is intermediate between SFNN and DNN,
in forward pass and computing gradients in backward pass → in the forward pass and computing gradients in the backward pass
in order to handle the issue in forward pass →  in order to handle the issue in the forward pass
Neal (1990) proposed a Gibbs sampling → Neal (1990) proposed Gibbs sampling
for making DNN and SFNN are equivalent → for making the DNN and SFNN equivalent
in the case when DNN uses the unbounded ReLU → in the case when the DNN uses the unbounded ReLU
are of ReLU-DNN type due to the gradient vanishing problem → are of the ReLU-DNN type because they mitigate the gradient vanishing problem
multiple modes in outupt space y → multiple modes in output space y
The only first hidden layer of DNN → Only the first hidden layer of the DNN
is replaced by stochastic one, → is replaced by a stochastic layer,
the former significantly outperforms for the latter for the → the former significantly outperforms the latter for the
simple parameter transformations from DNN to SFNN are not clear to work in general, → simple parameter transformations from DNN to SFNN do not clearly work in general,
is a special form of stochastic neural networks → is a special form of stochastic neural network
As like (3), the first layer is → As in (3), the first layer is
This connection naturally leads an efficient training procedure → This connection naturally leads to an efficient training procedure


ING PLATFORM AND APPLICATION TO DISTRIBUTED TRAINING
Deep learning is increasingly attracting attention for processing big data. Existing frameworks for deep learning must be set up to specialized computer systems. Gaining sufficient computing resources therefore entails high costs of deployment and maintenance. In this work, we implement a matrix library and deep learning framework that uses JavaScript. It can run on web browsers operating on ordinary personal computers and smartphones. Using JavaScript, deep learning can be accomplished in widely diverse environments without the necessity for software installation. Using GPGPU from WebCL framework, our framework can train large scale convolutional neural networks such as VGGNet and ResNet. In the experiments, we demonstrate their practicality by training VGGNet in a distributed manner using web browsers as the client.
Deep learning is increasingly attracting attention for processing big data. Existing frameworks for deep learning must be set up to specialized computer systems. Gaining sufficient computing resources therefore entails high costs of deployment and maintenance. In this work, we implement a matrix library and deep learning framework that uses JavaScript. It can run on web browsers operating on ordinary personal computers and smartphones. Using JavaScript, deep learning can be accomplished in widely diverse environments without the necessity for software installation. Using GPGPU from WebCL framework, our framework can train large scale convolutional neural networks such as VGGNet and ResNet. In the experiments, we demonstrate their practicality by training VGGNet in a distributed manner using web browsers as the client.
1 INTRODUCTION
Recently, machine learning, which uses big data derived from user activity on websites, images and videos is increasingly getting attention. Deep learning is at the center of that attention. Conventional machine learning techniques have required hand-crafted features specialized to a particular domain such as image or voice. In contrast, deep learning has a hugely important benefit that can illustrate data flow from raw data to an objective value in a single neural network and can train thoroughly using those data. In the computer vision domain, a team of Hinton (Krizhevsky et al., 2012) achieved outstanding classification accuracy using deep learning in an object classification competition ILSVRC2012 (Russakovsky et al., 2015). In the subsequent years’ competitions, deep-learningbased methods evolved continually and exhibited superior performance (Simonyan & Zisserman, 2014a; Szegedy et al., 2014; He et al., 2016). Convolutional neural networks (CNNs) trained for ILSVRC object classification are helpful for improving classification accuracy for scene recognition and video recognition by functioning as a feature extractor or being fine-tuned (Zhou et al., 2014; Simonyan & Zisserman, 2014b). Moreover, application is beginning to emerge in other areas such as medical imaging (Tajbakhsh et al., 2016). Software platforms for deep learning are expected to play an important role in accelerating a wide range of research efforts and applications.
Although deep learning achieved significant recognition accuracy that cannot be achieved using conventional methods, the number of parameters that can be trained is greater, resulting in requests for huge amounts of training data. This shortcoming not only increases data collection costs but also increases computational costs of training larger parameters with larger data. Moreover, trialand-error must be undertaken to ascertain a good neural network structure; thereby higher costs become necessary. What resolved this computational cost difficulty and enabled deep learning to work on a practical scale problem is general purpose computing on GPU (GPGPU) technology, which offers rapid matrix calculation. However, a deep learning framework must be set up on a dedicated computer. If a user wants to train a huge network, then a cluster computing system that uses MPI or Hadoop must be used for collaboration of multiple computers to obtain larger working memory and computational speed. To set up and maintain these systems generally presents
an expensive task. For that reason, such systems are available only to expert IT companies or laboratories.
This work specifically examines JavaScript, the programming language that runs on web browsers installed on ordinary personal computers and smartphones. With the recent advancement of web technology, JavaScript became the standard programming language to implement rich applications on web browsers. Word processors provided by Google and Microsoft are the popular examples. Those applications are traditionally implemented as native applications. This is not only a change of programming language; it brings an advantage of install-free convenience. Moreover, the communication features of web browsers are used not only during the loading of the application, but are also used by the application on demand, using so-called Ajax technology. For example, using this technology with a Google service spreadsheet, modifications made by one user are shown in real time on other users’ displays. By making full use of this technology, collaboration of an application running on web browsers across the internet becomes possible. Moreover, web browsers such as Google Chrome run not only on Windows, but also on Mac OS X, Linux, Android, and iOS smartphones. They provide a compatible JavaScript executing environment. More recently, a small microcontroller board for prototyping Internet of Things (IoT) devices runs Linux. JavaScript can run on these devices. However, JavaScript is rarely used for scientific computation. This is mainly because JavaScript assumes single-threaded execution. It has no fast matrix computation library, which is crucially important for scientific computation. To resolve this difficulty, our previous work proposed the fast matrix computation library, which uses a parallel computing platform, WebCL, from JavaScript (Miura et al., 2015). In WebCL, GPGPU can be utilized from JavaScript code. Moreover, its application to deep learning is proposed (Miura & Harada, 2015). However, existing implementations cannot fully exploit the functionality of JavaScript and WebCL. For that reason, only a small six-layer CNN for classifying CIFAR-10 (Krizhevsky, 2009) dataset can be trained. In this work, our objective is to provide a deep learning platform that can train practical large-scale CNN as large as VGGNet. In the Experiment section, we present preliminary results on training VGGNet by distributed computation using web browsers as the computation client. In the following section, we restrict our description to CNN only, but our system is applicable to neural networks of other kinds by implementing the layers that they need.
Our contributions are the following:
• We implemented the fastest matrix library and deep learning library that can run on web browsers using GPGPU. The source code is provided as open-source software1.
• Even where GPGPU cannot be used, native JavaScript implementation is provided, which allows high-level multi-dimensional matrix operation.
• We describe the possibility of training large scale CNN in a distributed manner without installing software in computation nodes, except for a generic plugin.
2 RELATED WORK
In this section, we first describe the studies related to distributed computing using generic computers that are not designed for scientific computing. The SETI@home project searches for extraterrestrial life (Anderson et al., 2002). In that research effort, radio waves analyses were performed distributedly on computers of volunteers. Although dedicated software had to be installed, more than 3 million computers participated in the project and contributed vast amounts of computational resources. Merelo-Guervos et al. (2008); Klein & Spector (2007) distributedly computed genetic algorithm (GA) using web browsers as computing nodes. The main component of GA was calculation of the fitness of population, which could be computed completely in parallel, thereby achieving extremely effective distributed computing. In our work, the main task to be distributed is deep learning, for which a large amount of weight parameters must be communicated frequently. Therefore, the communication efficiency becomes important.
Secondly, we explain distributed computing of deep learning. Dean et al. (2012) proposed a mechanism called DistBelief, which divides a neural network into multiple blocks of neurons and trains each block in a different computer. Large amounts of data are transferred at the division borders.
1Download code from https://github.com/mil-tokyo
They require n-to-n communication, which is unsuitable for environment in which computing nodes are not in the same LAN. deeplearning4j 2 provides distributed computing of deep learning framework that runs on the distributed computing Hadoop. However, Hadoop must be installed in all computing nodes, thereby imposing high deployment and maintenance costs. Meeds et al. (2014) developed a distributed deep learning system using web browsers. However, it is implemented in native JavaScript. For that reason, training with a large-scale dataset is nearly impossible because of the computational speed. In this work, we inherit the good properties of a JavaScript (web browser) based computing environment, with the aim of making training of practical CNN possible.
3 MATRIX LIBRARY IMPLEMENTATION
In this section, we describe the fast and generic matrix library “Sushi2”, which is based on previous library “Sushi.” They are using WebCL technology, which is a parallel computing platform to be used from JavaScript. WebCL is a JavaScript wrapper for parallel computing platform OpenCL, standardized by Khronos Group, which provides a unified interface to multi-core CPU and GPGPU. In contrast to NVIDIA CUDA, GPUs from AMD and Intel can also be used as accelerators. Unfortunately, WebCL is not built-in feature of web browsers, but there is an add-on for Firefox and WebCL-integrated Chromium. Our library also works with node.js (server-side JavaScript execution environment), in which node-opencl3 library can be used to accelerate computation. Although Sushi2 performs best in a WebCL environment, most functions have equivalent native JavaScript implementation. Sushi2 currently uses WebCL for the acceleration of numerical calculation, but it is possible to use other solutions including WebGL or asm.js by substituting implementation of matrix manipulation. In WebCL, “kernel” is the function to run on GPGPU. Kernel, which is written in C language, must be compiled before use. Sushi2 wraps them to allow users to write simple codes. Details of low-level WebCL operations are available in the literature (Miura et al., 2015).
Though Sushi achieved efficient calculation on GPGPU, currently it lacks the availability for large scale neural networks that require matrices of large dimensions. Sushi2 is developed to overcome such problems that Sushi has been facing and achieved the following benefits:
• Use simple and efficient data structures to achieve good performance. • Allow users to understand how to use it easily. • Support CPU (native JavaScript) and GPGPUmatrix without burdening ordinary users with
learning WebCL programming.
Most general purpose matrix libraries for JavaScript represent a multi-dimensional matrix with a nested JavaScript array. In contrast, Sushi2 represents a matrix with TypedArray, which is used for transferring numeric data between the CPU and GPGPU. TypedArray is a one-dimensional numeric array with fixed size and bit width at construction, as in arrays of C language. The array accommodates efficient storing and manipulation of large data. TypedArray which stores 32-bit floating point numbers is named Float32Array and the one that stores 8-bit unsigned integer is named Uint8Array. The numeric type of JavaScript is a 64-bit floating point number, but some WebCL environments do not support it. Therefore, the basic numeric type of matrix is a 32-bit floating point number. However, the precision of a 32-bit floating number is only 23-bit, so it cannot be used as an index of a large matrix (which have more than 223 elements). This is a problem for functions such as argmax, so a 32-bit signed integer matrix is also implemented. Moreover, an 8-bit unsigned integer matrix for raw image data and a logical matrix for Boolean operations are implemented.
Functions for the operating matrix are designed to be similar to those of MATLAB, which allows new users to use Sushi2 quickly. Operations for matrices that have more than two dimensions are implemented. It is a simple matter to operate color images and sets of color images (fourdimensional matrix). Almost all patterns for indexing operation in MATLAB are implemented. For import or export of a matrix, efficient binary format of numpy4 is implemented as well as the native JavaScript nested Array.
2http://deeplearning4j.org 3https://github.com/mikeseven/node-opencl 4http://docs.scipy.org/doc/numpy/neps/npy-format.html
Function $M.gpuArray transfers a matrix to GPGPU. In functions that support WebCL, operations of matrices in GPGPU are accelerated. In JavaScript, unused memory is released by garbage collection, but this is not applied for memory allocated on the GPGPU by WebCL. It has to be released by explicitly calling the destruct method. To make programming convenient, an “autodestruct” helper function is supplied. When the closure passed to autodestruct finishes, the matrices allocated in it are released automatically. Figure 1 presents a sample implementation of a fully-connected layer of CNN. Whether input matrices are on GPGPU or not, they can be processed in the same code.
Most GPGPU kernels are implemented originally for Sushi2, but matrix multiplication kernel is ported from clBLAS’s5 “sgemm”, because it requires advanced optimization.
Table 1 presents a speed comparison between our library and existing JavaScript based matrix libraries; Sylvester6 and Math.js7. The hardware environment is on Table 2 (AMD). When GPGPU is used, the time includes data transfer between the CPU and GPGPU. Task 1 represents simple element-wise task. Task 2 represents relatively expensive element-wise task. Task 3 and 4 are matrix multiplication task; the complexity of operations is greater than the number of elements. Our matrix representation (TypedArray) seems to be better than native JavaScript Array used in other libraries, even without WebCL. We can see clear superiority of using GPGPUwhen the computational cost is high.
4 DEEP LEARNING LIBRARY IMPLEMENTATION
In this section, we describe deep learning library “Sukiyaki2”, which is based on matrix library Sushi2.
5https://github.com/clMathLibraries/clBLAS 6http://sylvester.jcoglan.com/ 7http://mathjs.org/
Sukiyaki2 implements modules that are necessary for deep learning: layers, network structure manager, and optimizers. Users can use a single layer separately, as well as training network by supplying configuration file to the executable. Figure 2 portrays a sample of a network definition file. For network analysis required for distributed computing in the future, we used the architecture with statically defined relations of layers. Improvements from our previous work include: enabling network graph branch (necessary for ResNet training), addition of some layers including dropout and batch normalization, efficient binary export of network parameters. Users can implement the original layers and optimizers to train new neural networks. It works automatically with CPU and GPGPU if it can be implemented by Sushi2’s matrix operations. For cases in which a performance bottleneck exists, a dedicated GPGPU kernel can also be implemented. Using GPGPU for training is recommended, but almost all functions have native JavaScript fallback.
Figure 3 portrays a sample application for recognizing digits captured using a camera. The network is trained using MNIST dataset (LeCun et al., 1998b). Although image data are given as a flat byte array, extensive functions of Sushi2 allow short implementation of image recognition only in 10 lines. Recent web browsers for smartphones follow the JavaScript standard, and it is possible to develop such applications in this sample.
5 EXPERIMENTS
5.1 SINGLE-GPGPU TRAINING
In this section, we evaluate the CNN training performance of the proposed system. The specifications of hardware used for experiments are shown in Table 2.
First, we compared our library and existing deep learning library ConvNetJS by Andrej Karpathy8, which is written in JavaScript. We evaluated them by training LeNet with MNIST dataset (LeCun et al., 1998b). The network structure is based on LeCun et al. (1998a), which contains two convolutional layers and two fully-connected layers. The batch size is 64. Firefox (version 32) and node.js (version 4.3.0) are used as the JavaScript execution environment. A tiny server application is implemented and used for supplying the dataset and saving the trained model to and from the web browser.
The measured calculation speed is presented in Table 3. In Firefox, the performance gain was relatively low because the control overhead of GPGPU is dominant in the small CNN. In node.js, this overhead is smaller, thus using GPGPU allowed faster computation by a large margin.
Next, we trained VGGNet (Simonyan & Zisserman, 2014a) and ResNet (He et al., 2016) as practical scale CNNs. VGGNet is proposed by Simonyan & Zisserman (2014a) at ILSVRC2014. 16-layer version, denoted as VGG16, includes 13 convolutional layers and 3 fully-connected layers. It is among the largest CNNs that are commonly used. ResNet is the winner of ILSVRC2015. 152-layer version, denoted as ResNet152, includes 151 convolutional layers and 1 fully-connected layer, but the bottleneck structure reduces the number of parameters.
We used Caffe (Jia et al., 2014), a popular deep learning library, for comparison. The mainstream version of Caffe employs NVIDIA CUDA as the interface to GPGPU. We designate this version as Caffe (CUDA). CUDA is not compatible with GPGPUs other than NVIDIA’s. Caffe uses cuBLAS for matrix operations such as multiplication. There are forks of Caffe which use OpenCL as an cross-platform GPGPU interface. One such fork is OpenCL-Caffe by AMD9, which uses clBLAS as the matrix operation. Another one is the opencl branch of Caffe by Fabian Tschopp10. It uses ViennaCL11 for matrix operations. In Caffe (CUDA), the cuDNN accelerator library from NVIDIA can also be attached. We used same batch size in the same CNN / GPU setting for fair comparison.
The training speed is presented in Table 4. By virtue of GPGPU, VGG16 and ResNet152 can be trained, which was difficult using existing JavaScript based libraries. In ResNet152, more than 1,000 GPGPU kernels are executed and its execution overhead seems to be problematic on Firefox environment. Currently, our library is not faster than Caffe, but it achieved the same order of speed. Especially, Caffe (CUDA) provides the best performance. This difference mainly comes from the speed of convolution. Implementation of convolution in Caffe is similar to ours. To perform con-
8http://cs.stanford.edu/people/karpathy/convnetjs/index.html 9https://github.com/amd/OpenCL-caffe
10https://github.com/BVLC/caffe/tree/opencl 11http://viennacl.sourceforge.net/
volution, elements of the input matrix are re-ordered (i.e. lowering). Then the output is gained by matrix multiplication with the weight. Table 4 presents the calculation speed in matrix multiplication used in computation of VGG16, performed by cuBLAS and clBLAS.
As the table shows, clBLAS gives inferior speed, especially on gradient computation of layers that are close to the input layer. In such layers, the matrix shape is far from square. For that reason, performance tuning for such input shape or implementation without matrix multiplication is needed. In the CUDA environment, Lavin (2015) showed that 96% of theoretical GPGPU performance is achieved in convolution by circumspect implementation.
5.2 DISTRIBUTED TRAINING
In this subsection, we describe a preliminary evaluation of distributed training of CNN.
The method of distributed training is simple data-parallelism. The system is depicted in Fig. 5. First the server distributes network weight Wt and images in a batch. A batch for the iteration (It) is divided into N splits, It1, It2, ..., ItN , where N is the number of computing clients. After the client K calculates gradient of weight ∆WtK using assigned batch split, they send the gradient to the server. The server takes the average of the gradients from all clients and then updates the weight using it (Wt+1 = Wt − η 1NΣ∆WtK). The optimization method is momentum SGD. The result is equivalent regardless of the number of clients.
First, we trained LeNet distributedly in Nexus 7 tablets (Android OS). Chrome browser is used as the client. The batch size is 120 and divided by the clients equally. Figure 6 (left) shows the speedup according to the increase in the number of clients. Naturally, the absolute speed is slow, but we can demonstrate that the computational power of mobile devices can be accumulated and nearly linear speedup is achieved.
Next, we train large scale CNN; VGG16. Its weight and gradient have 130 million elements. It therefore requires 500 MB if represented as 32-bit floating point numbers, which poses a large communication bottleneck. To suppress this issue, we implemented 8-bit representation of each element proposed by Dettmers (2016). We used p2.xlarge instance of Amazon Web Services for GPGPU environment. It contains NVIDIA K80 GPU. The batch size is 256 according to (Simonyan & Zisserman, 2014a). Single forward-backward procedure cannot process 256 images at the same time due to the memory limit, so we average the gradients from multiple forward-backward procedure.
We show the speed of calculation with respect to the number of computing clients in Fig. 6 (right). Although our main focus is using web browser as clients, the result on using node.js as clients is also shown for reference. Under current settings, use of four clients achieved 2.8 times faster computation than with one client setting. The speed is much faster than existing OpenCL-based Caffe. Due to the communication overhead, the speed saturates at 8 clients even when 8-bit representation is employed.
Although we used K80, a high-end GPU, for this experiment, our motivation is to use ordinary personal computers for distributed computing. We can assume that latest ordinary personal computers (not dedicated for 3D game) have 1/10 performance compared to K80. In K80, we could train VGG16 with 29 seconds per iteration using 8 computers. In 1/10 performance GPU, we can estimate that maximum speed is 100 seconds per iteration using 16 computers, considering both calculation and network time. We compressed the weight to 1/4 size by the method of Dettmers, if we can compress it to 1/10 further, the maximum speed will be 31 seconds per iteration using 64 computers. Thus, further improvements demand reduction of communications and a better strategy of parallelism. We leave those improvements as a subject for future work.
6 CONCLUSION
We implemented a JavaScript based matrix library and deep learning library, to perform deep learning and to develop applications that use a trained model without a dedicated computer system. Using GPGPU via WebCL, our library provides much better performance than existing JavaScript based libraries. It became possible to train VGG16 and ResNet152. However, the performance is not reaching Caffe running on NVIDIA CUDA environment. A salient difficulty is that matrix multiplication necessary for convolution is slower. Additionally, we used WebCL as GPGPU interface, but currently it is not included in web browsers. Further improvements in web technology must be undertaken to make full computing power available to scripts in web pages. In experiments of distributed training of VGG16 using web browsers as computing client, 2.8x speed improvement was gained from four clients. The speed is much faster than existing OpenCL-based Caffe using single computer. The parallelization method used in the experiment is naı̈ve, and further exploration of this area will be undertaken as a subject of future work.
ACKNOWLEDGMENTS
This work was supported by CREST, JST.

This paper presents a JavaScript framework including WebCL components for training and deploying deep neural networks. The authors show that it is possible to reach competitive speeds with this technology, even higher speed than a compiled application with ViennaCL on AMD GPUs. While remaining a little more than factor three slower than compiled high performance software on NVIDIA GPUs, it offers compelling possibilities for easily deployable training and application settings for deep learning.
My main points of criticism are:
1. In Tab. 4 different batch sizes are used. Even if this is due to technical limits for the Javascript library, it would only be fair to use the smaller batch sizes for the other frameworks as well (on the GPUs probably in favor of the presented framework).
2. In Fig. 6, why not include more information in the graphs? Especially, as stated in the question, why not include the node.js values? While I do see the possible application with one server and many "low performance" clients, the setting of having a few dedicated high performance servers is quite likely. Even if not, these are good values to compare with. For the sake of consistency, please include in both subfigures Firefox, Chrome, node.js.
Apart from these points, well-written, understandable and conclusive.