MULTIAGENT SYSTEM FOR LAYER FREE NETWORK
We propose a multiagent system that have feedforward networks as its subset while free from layer structure with matrix-vector scheme. Deep networks are often compared to the brain neocortex or visual perception system. One of the largest difference from human brain is the use of matrix-vector multiplication based on layer architecture. It would help understanding the way human brain works if we manage to develop good deep network model without the layer architecture while preserving their performance. The brain neocortex works as an aggregation of the local level interactions between neurons, which is rather similar to multiagent system consists of autonomous partially observing agents than units aligned in column vectors and manipulated by global level algorithm. Therefore we suppose that it is an effective approach for developing more biologically plausible model while preserving compatibility with deep networks to alternate units with multiple agents. Our method also has advantage in scalability and memory efficiency. We reimplemented Stacked Denoising Autoencoder(SDAE) as a concrete instance with our multiagent system and verified its equivalence with the standard SDAE from both theoritical and empirical perspectives. Additionary, we also proposed a variant of our multiagent SDAE named "Sparse Connect SDAE", and showed its computational advantage with the MNIST dataset.
We propose a multiagent system that have feedforward networks as its subset while free from layer structure with matrix-vector scheme. Deep networks are often compared to the brain neocortex or visual perception system. One of the largest difference from human brain is the use of matrix-vector multiplication based on layer architecture. It would help understanding the way human brain works if we manage to develop good deep network model without the layer architecture while preserving their performance. The brain neocortex works as an aggregation of the local level interactions between neurons, which is rather similar to multiagent system consists of autonomous partially observing agents than units aligned in column vectors and manipulated by global level algorithm. Therefore we suppose that it is an effective approach for developing more biologically plausible model while preserving compatibility with deep networks to alternate units with multiple agents. Our method also has advantage in scalability and memory efficiency. We reimplemented Stacked Denoising Autoencoder(SDAE) as a concrete instance with our multiagent system and verified its equivalence with the standard SDAE from both theoritical and empirical perspectives. Additionary, we also proposed a variant of our multiagent SDAE named "Sparse Connect SDAE", and showed its computational advantage with the MNIST dataset.
1 INTRODUCTION
Deep networks are used in many tasks, especially in image recognition, signal processing, NLP and robot manipulation. Almost all deep network models utilize structure with layers. One of the primary reasons to use those layer structure is to utilize parallel computation techniques and hardware level technologies such as GPU or dedicated tensor processors.
Layers can be viewed as a restriction on the connection pattern of the units that they must be aligned in a row to form a vector. Though layers are related to vector-matrix processing technologies and necessary to make use of them, it is not evident that this vector-matrix restriction is the most appropriate model to capture the latent representation of the data.
Vector-matrix approach requires dense matrices to store weight parameters even after sparse representations and weights are learned, which requires inefficient use of memory.
There are several methods to reduce the cost by downsize the model such as distillation(Hinton et al. (2014)), and limiting the model space with binarizing(Courbariaux et al. (2015)) and hashing Han et al. (2015), but they still needs dense matrix filled with many zeros.
In this paper we propose a framework of multiagent calculation network that has feedforward matrix-vector network as a subset but free from the notion of layer to solve these problems.
Our multiagent network consists of many agents that replaces each unit in standard deep networks. The agents act autonomously and calculations are executed as an accumulation of many local level communication among the agents. This local calculation scheme is contrary to the previous feedforward network implementation that all computations in the units in the same layer is done simultaneously (Figure 1).
Specifically, we reimplemented Stacked Denoising Autoencoder(SDAE) (Vincent et al. (2008)) as one of the variations in our framework. SDAE is one of the earliest successful deep network model and free from spatiality assumption in CNNs and RNNs. SDAE is related to the Ladder Network (Rasmus et al. (2015)) in that both employ reconstruction and denoising mechanism. We suppose starting from the model strictly derived from SDAE and gradually extending the model is one good approach to develop useful algorithms to construct deep multiagent networks. We show that the standard vector-matrix SDAE can be interpreted and reconstructed as a specific case of our multiagent network in the sense that both SDAE compute the exact same result in the end but with different implementations and processes.
Once we establish the basic multiagent SDAE, we aim to extend the SDAE and examine its behavior. One of the minimum and simplest modification to the proposed multiagent SDAE is to keep the node’s locations and possible edge connections as they are, but randomly truncate the edges. We call this testing model "Sparse connect SDAE" or "SCSDAE" in the latter section. This model is an example of the potential of our model to handle sparse weight parameters more efficient than the standard networks composed of dense weight matrices.
Our contributions are the following.
1. We propose new multiagent-based neural network system framework that free from restrictions of layer scheme with matrix-vector while getting more biologically plausible.
2. We prove that SDAE can be reinterpreted as a special case of our multiagent system.
3. We also propose Sparse Connect SDAE, a primitive extension of our multiagent system.
4. We demonstrate the performance of the proposed models on XOR toy dataset and the permutation-invariant MNIST task.
2 RELATED WORKS
Foerster et al. (2016) and Sukhbaatar et al. (2016) model multiagent environment as a deep network, but their unit of agent is a whole network which models an individual actor and not a feature inside the network. The motivation and architecture of our proposal multiagent framework is different from theirs and we suppose our method could be a complementary method between the different granularities of agents.
There are several approaches to design biologically plausible model (Sussillo & Abbott (2009); Risi & Stanley (2014); Mi et al. (2014)). Cao et al. (2014) convert regular CNN into spiking neural networks (SNN). Osogami & Otsuka (2015) build a variation of Boltzman Machine that follows the properties of Spike-timing dependent plasticity (STDP), which makes it more close to biological neural networks. Lee et al. (2014) proposes a modification of autoencoder that uses a novel credit assignment method called "target propagation" in place of back-propagation and achieved state
of the art performance. It can be a solution to remove inherent biologiacal implausibility of back propagation.
Our SCSDAE is a variation of SDAE and can be viewed as a technique for truncating edges and reduce the computational cost. There are several techniques to downsize networks and enables us to load them on mobile devices for realtime processing. One example is distillation (Hinton et al. (2014)). It needs huge networks as a superviser network and it cannot be used as a mean to full scratch to new domain. MADE (Mathieu et al. (2015)) uses binary masking matrix to represent hard zero and thus conditional distribution. Limiting weight connection to binary (Courbariaux et al. (2015)) is another approach and its applicability is actively studied. Han et al. (2015) combines pruning, quantization and Huffman coding together.
3 REIMPLEMENTATION OF DEEP NETWORK AS A MULTIAGENT SYSTEM
In this section we describe our algorithm. First we show how we can reinterpret some common properties of multiagent systems to form a deep network. Next we define our multiagent system in general form. Then we prove the standard SDAE is indeed a special case of our multiagent system in the sense that both calculate the exact same computational result in the same order. Finally we show an example of our SDAE’s variations.
3.1 THE PROPERTIES OUR MULTIAGENT SYSTEM SHOULD SUFFICE
We extracted some common aspects of those multiagent system from the view of deep network development as follows:
1. The system consists of several number of autonomous units (as agents) and the environment.
2. All units are autonomous and only act when stimulated by messages from the other units and the environment.
3. Units process calculation only with local information they hold.
3.2 DEFINITION OF MULTIAGENT NETWORK
In this section we define the multiagent system that has the properties we stated above.
• A system is consists of several units and one environment. • Units are consists of nodes and edges. • Edges connect between two nodes. • Nodes has some variables and can memorize actual computed values of them.
The variables includes not only the input data and feature variables themselves, but parameters of adjacent edge’s weights and intermediate variables that are needed to calculate feature variable’s activation values. These additional variables can also be updated.
Nodes can transfer informations with other nodes connected with edges by message passing. These information may involve the value of variable computed by the units, the errors accumulated through epoch in the units, and any other things. Sometimes nodes may receive data input from the environment as a message, and may send value back to environment to let them compute global cost function.
The environment can manipulate unit from outside (via message passing) and change their relationships. For example,
• The environment can generate new unit. • The environment can connect between nodes. • The environment can change the state of the unit. • The environment can send units input data.
These special manipulation may seems to break locality and autonomy, but its global manipulation is limited to changing the overall network structure. In the calculation process, the algorithm still run by agents local algorithm themselves rather than step-by-step instructions from the environment. The environment input some data to input units, and then, all it can do is to wait at the output units to get the result and it isn’t aware of the internal behavior of each units and the order of execution.
Algorithm 2 is the general form of our proposed multiagent system. The notation is matched with the multiagent MLP in later sections. Note that the order of calculation is not yet specialized to match to standard MLP, and thus free from the constants related to global structure such as Nx and Nz .
3.3 THE EQUIVALENCE BETWEEN THE MULTIAGENT AND STANDARD VERSIONS OF SDAE
We show how to reconstruct Stacked Denoising Autoencoder((Vincent et al. (2008))) with the proposed multiagent system.We concurrently prove that our multiagent version of SDAE and the standard SDAE have indeed the same algorithm and do the same calculation. In our case, we suppose that the two criteria below are sufficient for our objective
• the equivalence of the value computed • the equivalence of the order of computation
given the same input and the same random sampling. (e.g. initial parameters, order of input data, noise variables, etc.)
We begin our proof with a simple model and reuse it for prove of more complex models. Our first target model is the simple multi layer perceptron(MLP) with only one hidden layer and no pretraining. We then go on to the autoencoder and Denoising Autoencoder, which is a kind of extension of MLP. Finally we show the equivalence of the two SDAEs.
For bravity, we limit the choice of activation function for each unit to sigmoid, and we use unit-wise mean square error (MSE) for cost function. We also use simple SGD without minibatch. There are several methods that empirically known to perform well. (e.g. Adam (Kingma & Ba (2014)) for optimization, cross entropy for cost function, ReLU and softmax for activation function.) However we prioritize to establish the basic architecture of deep multiagent network, and choose these simpler methods. We can also extend the algorithm to apply minibatch updating easily.
3.3.1 EQUIVALENCE WITH MLP
Now we prove the equivalence of our proposed multiagent version of MLP and standard implementation.
Suppose there is a MLP consists of an input layer, a hidden layer and an output layer. Let Nx, Ny, Nz be the number of input, hidden and output layer’s dimension respectively. Simillary, let xi(i = 1 · · ·Nx), yj(j = 1 · · ·Ny), zk(k = 1 · · ·Nz) be the activated output value (hereafter simply "output value" ) at each unit in each layer respectively. The weight value between these variables are wij , wjk, thus the forward propagation from input to hidden is calculated by yj ← σ(w0j + ∑Nx i=1 wijxi) and from hidden to output is zk ← σ(w ′ 0k + ∑Ny j=1 wjkyj). Here Θ = {wij , w ′
jk | i = 0 · · ·Nx, j = 0 · · ·Ny, k = 1 · · ·Nz} represents the weight parameters and σ is the sigmoid function. We also define dataset D = {(x(1)data, t (1) data), · · · , (x (D) data, t (D) data)} as an array of pairs of input data x (d) data = {x(d)1data , · · · , x (d) Nxdata } and label data t(d)data = {t (d) 1data
, · · · , t(d)Nzdata}, where d = 1 · · ·D. Hereafter we might drop the data index (d) for readability.
We define our objective function as L = ∑D
d=1 L (d) , where L(d) = 1Nz ∑Nz k=1(tk − zk)2 and η be a
learning rate. Algorithm 1 is the standard version algorithm to optimize this objective.
Next we construct a multiagent network that is equivalent to Algorithm 1 by specializing the general Algorithm 2. We first generate units uxi , uyj , uzk(i = 1 · · ·Nx, j = 1 · · ·Ny, k = 1 · · ·Nz) from the environment. We also denote the set of units Ux = {uxi | i = 1 · · ·Nx}, and similary Uy, Uz for set of all uyj , uzk respectively. Each unit and the environment possess the unique varibles as listed in Table 1.
Algorithm 1 Standard MLP
Initialize wij , w ′
jk for all (i, j), (j, k) while criteria is not satisfied do
for d = 1 · · ·D do for k = 1 · · ·Nz do tk ← t(d)kdata
end for for i = 1 · · ·Nx do xi ← x(d)idata end for for j = 1 · · ·Ny do yj ← σ(w0j + ∑Nx i=1 wijxi) end for for k = 1 · · ·Nz do zk ← σ(w ′ 0k + ∑Ny j=1 wjkyj)
δk ← 2Nz (zk − tk)zk(1− zk) for j = 1 · · ·Ny do w ′
jk ← w ′
jk − ηδkyj end for
end for for j = 1 · · ·Ny do δj ← yj(1− yj) ∑Nz k=1 w ′
jkδk for i = 1 · · ·Nx do wij ← wij − ηδjxi
end for end for L(d) ← 1Nz ∑Nz k=1(tk − zk)2
end for L← ∑D d=1 L (d)
end while
Algorithm 2 General form of our multiagent algorithm (the environment)
Initialize uxi , uyj , uzk for all i, j, k while criteria is not satisfied do
for all d ∈ {1 · · ·D} do for all uzk ∈ Uz do
Input t(d)kdata to uzk end for for all uxi ∈ Ux do
Input x(d)idata to ux end for L(d) ← ∑ k(tk − zk)2
end for L← ∑ d L (d)
end while
Algorithm 3 Multiagent MLP (the environment) Initialize uxi , uyj , uzk for all i, j, k while criteria is not satisfied do
for d = 1 · · ·D do for k = 1 · · ·Nz do
Input t(d)kdata to uz end for for i = 1 · · ·Nx do
Input x(d)idata to ux end for L(d) ← 1Nz ∑Nz k=1(tk − zk)2
end for L← ∑D d=1 L (d)
end while
The unit uxi corresponds to xi and receive xidata from the environment. The unit uzk corresponds to both zk, tk and is input tkdata . Each uxi receive input data, assign the data into the variable owned by itself, then send that value as message to uyj (algorithm 6).
uyj corresponds to yj and stores wij(i = 0 · · ·Nx) to calculate yj . The unit must wait until receive message from all uxi(i = 1 · · ·M). Then the unit can calculate yj (algorithm 4). The calculated value of yj is sent again to uzk , so similary zk can also become able to be calculated.
We also need to introduce a state variable for SDAE. We will discuss it at the end of this section.
Finally, the environment inputs data in the following order: uz1 → · · · → uzNz → ux1 → · · · → uxNx (Algorithm 3). These inputs stimulate units and the units invoke their message handling algorithm individually. The information required to calculate the cost function L(d) is eventually acumulated in the units uyk . The environment would read out these and calculate the objective L (d).
Algorithm 4 The handler of uyj when receiving a message
if The sender is uxi then if Received messages from all uxi(i = 1 · · ·Nx) then yj ← σ(w0j + ∑Nx i=1 wijxi)
for k = 1 · · ·Nz do Send uzk the value of yj
end for end if
else if The sender is uzk then if Received messages from all uzk(z = 1 · · ·Nz) then δj ← yj(1− yj) ∑Nz k=1 w ′
jkδk for i = 1 · · ·Nx do wij ← wij − ηδjxi
end for end if
end if
Algorithm 5 The handler of uzk when receiving data from the environment tk ← tdkdata
Algorithm 6 The handler of uxi when receiving data from the environment xi ← xdidata for j = 1 · · ·Ny do
Send uyj the value of xi via message end for
Algorithm 7 The handler of uzkwhen receiving a message
if Received messages from all uyi(j = 1 · · ·Ny) and the environment then zk ← σ(w ′ 0k + ∑Ny j=1 w ′
jkyj δk ← 2Nz (zk − tk)zk(1− zk) for j = 1 · · ·Ny do w ′
jk ← w ′
jk − ηδkyj end for for j = 1 · · ·Ny do
Send uyj the value of w ′
jkδzk end for
end if
Comparing both algorithms, we can verify that the all update statement and order for all variables are strictly matched between the proposed multiagent SDAE and the standard version.
3.3.2 EQUIVALENCE WITH AUTOENCODER
Now we advance to the autoencoder algorithm which is a special case of MLP. In autoencoders, The input and output dimension is the same (Nx = Nz). The cost function and difference at output units are changed to L(d) ← 1Nz ∑Nz k=1(xk − zk)2 and δk ← 2 Nz
(zk − xk)zk(1− zk) respectively. In the corresponding multiagent version, the environment don’t need to send tidata to uzi . Instead uxi send the input value xi to uzi right before messaging to uy1 .
uzi don’t take any action in response to receiving message from uxi , just as same as from the environment. With the modifications above, the two versions become equivalent.
3.3.3 EQUIVALENCE WITH DENOISING AUTOENCODER
Denoising Autoencoder (hereafter "DAE") is an extension of autoencoder and used as a building block of SDAE. In DAE, when calculating y of the hidden layer, we don’t directly use raw value of x. Instead, we add noise to x to make x̃, then y uses x̃. Note that the reconstruction layer z still use the original x value.
To reinterpret this DAE by our multiagent system, we can assign x̃i to uxi along with xi. When the data is input, we can calculate x̃ right after the assignment to xi. Each xi(i = 1 · · ·Nx) send corresponding yj(j = 1 · · ·Ny) the noised value x̃i instead of xi. Then the two algorithm become equivalent again.
3.3.4 EQUIVALENCE WITH STANDARD SDAE
We finally reimplement the whole SDAE algorithm by our multiagent system and show its equivalence with the standard version again. The learning process of SDAE can be divided into pretraining and fine-tuning and we can explain the two phase separately.
pretraining phase In the pretraining phase of SDAE, let xi, yi, zi be the input, hidden, the associated reconstruction layer value of DA, respectively. To stack a learning DA, yj is considered as a new input units from the view of the new DA. Let z ′
j be the reconstruction unit’s value of yj itself, and y ′
k be the new stacked hidden layer units. We make a full connection between each layers. Thus the equations relating to these variables change to y ′ k ← σ(w0 + ∑Ny j=1 wjkyj) and z ′ j ←
σ(w0 + ∑N y ′ k=1 wkjy ′ k). The new objective function after the stacking is Lj = 1 Ny ∑Ny j=1(z ′
j − yj)2. Note that by stacking a new DAE layer, we are making a new task of objective function from the previous objective.
Now we describe the change needed on our multiagent system. We need to introduce the notion of state for the units associated with the hidden layers in the standard version. When the objective function changed, the network structure and the state of the units must be changed in response too.
The hidden units can take the the two states, "state A (On learning)" and "state B (Learned, stable)". The units are initiated to "state A" and the behavior in this state doesn’t change from the previous sections. But the units in "state B" doesn’t send messages to the units for reconstruction zi and instead send the same messages to the "state A" units in the stacked layer.
Additionary, we need to append a reconstruction unit for each hidden unit changed to "state B". These pairs of the "state B" unit and appended reconstruction unit act as same as the pairs of input and reconstruction units we stated in DAE section except that they send message not after the data is input, but after the feature yj is calculated.
Once the pretraining for one block of DAE ends, all the hidden units in that DAE change to "state B". Then they form a new DAE starting from "state B" hidden units through the new stacked "state A" units then the associated reconstruction units appended. This DAE can be optimized in the same way of the single multiagent DAE we have described with only one exception that the data is not sent from the environment, but instaed the feature value yj in the previous DAE is calculated in the same order (j = 1 · · ·Ny)
fine-tuning phase In order to use SDAE for supervised learning task, we pretrain some predefined number of hidden layers by layerwise pretraining, and at the end stack the layer for supervised purpose (typically a softmax layer for classification). This is another form of the change of the objective function and the multiagent system can deal with it too by changing network and unit’s state.
Specifically, when the all pretraining phase ends, connect the output units for supervised learning onto the topmost hidden units, and all feature unit changes its state to the newly introduced "state C (On fine-tuning)". The "state C" is almost the same as "state A", but the only differences are the two:
• They don’t send message to reconstruction units. • They always send backpropagation message to the previous units.
This backpropagation message must be sent in ascending order as well as the previous message passing.
Here we reinterpreted the standard pretraining and fine-tuning algorithm with our proposed multiagent system. These two cover the all learning phase of SDAE, therefore we have successfully reinterpreted the whole SDAE and showed its equivalence of the standard SDAE.
3.4 SPARSE CONNECT SDAE
Now we obtain new multiagent SDAE, and theoritically showed its equivalence to standard SDAE. Next we want to extend this model and experiment its behavior. One of the minimal and simplest
modification to this model is to truncate the edges randomly. We call this model "Sparse connect SDAE" or "SCSDAE" for the latter section.
Since our goal here is to check the basic behavior when we truncate edges, we take one of the simplest rule that we basically connect all edges between nodes (if the distance meets criteria) but drop them for certain threshold probability. Once the connection is established, we fix them and will not modify online. For example, connection rate 1.0 (100%) means it is the same network as SDAE and 0.3 means 30% of edges in SDAE remain intact and the others are dropped.
We can expect the time spent in learning is proportional to the connection rate. This is an obvious strong point against the standard matrix-vector based network. In matrix-vector based networks, we must fix the size of weight matrices and keep them at their original size even if most weights are learned to be soft zero. In our multiagent model however, we only need to calculate for existing edges. We will see whether this expectation holds in the next experiments section.
4 EXPERIMENTS
We implemented the proposed multiagent network and measure its performance. We verified empirical equivalence between the multiagent SDAE and standard SDAE. We also measure performance of Sparse connect SDAE, which we described in 3.4. We change its connection rate and see how the performance and learning time changes.
4.1 DATASET
We use two datasets for experiment, one is an artificial XOR function toy dataset made by us for this experiment, and the other is the MNIST handwritten digit dataset. Since the result and conclusion is almost same, we only report for the MNIST dataset. The details of the datasets are described in Appendix A.
4.2 MODEL SETTINGS
We use the sigmoid function as activation functions of all units. We use the unit-wise MSRE as mentioned in the section 3 for both reconstruction errors in pretraining phase and classification in finetune phase. In pretraining, we simply take the mean through the all reconstruction units to check if the training works. For finetuning, we again take the sum of MSRE as cost function, and as classification error, we take the label associated with the unit that has the most activated value and discretely compare that value to measure the error rate. The SDAE algorithm includes random noise addition and is not fully deterministic. So we run each experiment settings for 3 times and take the average through the runs for all metrics we show. We set learning rate for pretraining and finetuning to 0.001 and 0.1, respectively. Corruption rate of SDAE is fixed to 0.3.
4.3 COMPARISON BETWEEN MULTIAGENT AND NORMAL SDAE
We compared the test error rate through epochs between multiagnt and normal SDAE. Figure 2 show the mean classification error rate through 3 simulations with the MNIST dataset. We can see the two graphs form almost the same shape, which suggests the multiagent implementation of SDAE is empirically equivalent to normal SDAE. This is a verification of the theoritical proof we showed in section 3.
In order to check this equivalence more precisely, we evaluate the difference of maximum weight update between multiagent and normal SDAE (Table 2). We gave an input data to both networks and measure the largest change of weight change update for each edge group shown in Table 2. To make the problem simple and clear, for this experiment we used only one hidden layer and no pretraining, so the model is similiar to a naive MLP. From the Table 2, we can see that the decimal order of largest difference is 3 digits small as that of information error with 32bit floating point numbers. We suppose this difference is small enough to be well ignored.
4.4 SPARSE CONNENCT SDA
We gradually changed the connectivity rate of our proposed SCSDAE and compare its performance. In this experiment, we measure the performance by the two aspect, the error rate and the calculation time, because we expect that the less number of edge mean the less calculation in time in our model as we described in chapter 3.
Figure 3 shows the meausred calculation time and Figure 4 for the error rate. In both figures the horizontal coordinate indicates the number of iterated epochs and the vertical is for time and error rate, respectively. From Figure 3, we can verify that the calculation time decreases linearly as we expected.
Figure 4 shows the error rate on test set after the given number of epochs passed. Contrary to our intuition, it can be said that in some cases despite of the decrease in connection probability, error rate do not deteriorate. The reason is not obvious, but we suppose that the more connection established, the more it became difficult and time requiring to the model to converge enough to show the potential of the model class. It can be also possible that the forced sparsity gave the model unexpected advantage to obtain sparse coding efficiently.
5 CONCLUSION
We proposed a fundamental framework to reinterpret deep networks as multiagent systems. Specifically, we reimplemented a model equivalent to Stacked Denoising Autoencoder(SDAE) but the inside implementation is given with the multiagent system. We verified this equivalence from both theoritical and empirical aspects. We also tested the behaviour when we actually let the model drop
some edges permanently and demostrated the reduction of computation time as the connection rate decreases.
Our contribution is to propose new multiagent based neural network system that free existing deep network from restriction of layer scheme. Our system involves the standard SDAE as its subset and has large potential of extension. Our model could be extended to more biologically plausible variation. Our experiment with the proposed Sparse Connect SDAE demostrate the advantage of non-matrix calculation and permanent drop of edges.
The next step is to sophisticate the model so that the agents are more strictly independent from the environment. Then we will be able to use this system as a framework for extending more flexible deep network. For example, we can make arbitrary connection of units. We can also consider mixing of units with different activation function, dropout strategy, learning rate.
A. DATASETS
XOR function toy dataset consists of random ordering of four points representing the four region of XOR function on 2D coordinate, which are set of tuple (x1, x2, y) = [(1, 1, 1), (1,−1,−1), (−1, 1,−1), (−1,−1,−1)]. The value ’1’ represents boolean value ’true’ and ’-1’ for ’false’. x1 and x2 mean boolean value input to the XOR function and y is the output.
MNIST handwritten digit dataset is a famous image recognision task. The input is a 28x28 image patch of digit classified to 10 classes(0-9). It contains 60000 train images and 10000 test images. Since our goal is not to pursue the state of the art performance of this dataset, but verify the common property of our proposed model with more realistic problem rather than the XOR toy sample, so we did no preprocessing on the dataset. We directly input image pixel value as a 784 row vector and didn’t use the prior that the image is 2D data with spatiality which is importatnt in convolutional network. This setting is called "permutation invariant" version of the MNIST task.
B. EXPERIMENTS DETAILS
For the XOR task, we set the number of hidden units to [3, 4, 5]. For MNIST, we used [100, 100, 100] network for all experiments. The learning rate was fixed to 0.001 at pretraining time and 0.1 at finetuning. We trained 30 epochs of pretrainings for each MLP layer and 50 epochs of finetuning. We used a single core of Intel(R) E5-2630 @ 2.40GHz to measure the performance invariant to paralization method. The effect of paralization is a question for future research.

The paper reframes feed forward neural networks as a multi-agent system.
It seems to start from the wrong premise that multi-layer neural networks were created expressed as full matrix multiplications. This ignores the decades-long history of development of artificial neural networks, inspired by biological neurons, which thus started from units with arbitrarily sparse connectivity envisioned as computing in parallel. The matrix formulation is primarily a notational convenience; note also that when working with sparse matrix operations (or convolutions) zeros are neither stored not multiplied by.
Besides the change in terminology, essentially renaming neurons agents, I find the paper brings nothing new and interesting to the table.
Pulling in useful insights from a different communitiy such as multi-agent systems would be most welcome. But for this to be compelling, it would have to be largely unheard-of elements in neural net research, with clear supporting empirical evidence that they significantly improve accuracy or efficiency. This is not achieved in the present paper.


In this paper we present a new ensemble method, called Boosted Residual Networks, which builds an ensemble of Residual Networks by growing the member network at each round of boosting. The proposed approach combines recent developements in Residual Networks a method for creating very deep networks by including a shortcut layer between different groups of layers with the Deep Incremental Boosting, which has been proposed as a methodology to train fast ensembles of networks of increasing depth through the use of boosting. We demonstrate that the synergy of Residual Networks and Deep Incremental Boosting has better potential than simply boosting a Residual Network of fixed structure or using the equivalent Deep Incremental Boosting without the shortcut layers.
1 INTRODUCTION
Residual Networks, a type of deep network recently introduced in He et al. (2015a), are characterized by the use of shortcut connections (sometimes also called skip connections), which connect the input of a layer of a deep network to the output of another layer positioned a number of levels “above” it. The result is that each one of these shortcuts shows that networks can be build in blocks, which rely on both the output of the previous layer and the previous block.
Residual Networks have been developed with many more layers than traditional Deep Networks, in some cases with over 1000 blocks, such as the networks in He et al. (2016). A recent study in Veit et al. (2016) compares Residual Networks to an ensemble of smaller networks. This is done by unfolding the shortcut connections into the equivalent tree structure, which closely resembles an ensemble. An example of this can be shown in Figure 1.
Dense Convolutional Neural Networks Huang et al. (2016) are another type of network that makes use of shortcuts, with the difference that each layer is connected to all its ancestor layers directly by a shortcut. Similarly, these could be also unfolded into an equivalent ensemble.
True ensemble methods are often left as an afterthought in Deep Learning models: it is generally considered sufficient to treat the Deep Learning method as a “black-box” and use a well-known generic Ensemble method to obtain marginal improvements on the original results. Whilst this is an effective way of improving on existing results without much additional effort, we find that it can amount to a waste of computations. Instead, it would be much better to apply an Ensemble method that is aware, and makes us of, the underlying Deep Learning algorithm’s architecture.
We define such methods as “white-box” Ensembles, which allow us to improve on the generalisation and training speed compared to traditional Ensembles, by making use of particular properties of the
base classifier’s learning algorithm and architecture. We propose a new such method, which we call Boosted Residual Networks, which makes use of developments in Deep Learning, previous other white-box Ensembles and combines several ideas to achieve improved results on benchmark datasets.
Using a white-box ensemble allows us to improve on the generalisation and training speed by making use of the knowledge of the base classifier’s structure and architecture. Experimental results show that Boosted Residual Networks achieves improved results on benchmark datasets.
The next section presents the background on Deep Incremental Boosting. Then the proposed Boosted Residual Networks method is described. Experiments and results are discussed next, and the paper ends with conlusions.
2 BACKGROUND
Deep Incremental Boosting, introduced in Mosca & Magoulas (2016a), is an example of such whitebox ensemble method developed for building ensembles Convolutional Networks. The method makes use of principles from transfer of learning, like for example those used in Yosinski et al. (2014), applying them to conventional AdaBoost (Schapire (1990)). Deep Incremental Boosting increases the size of the network at each round by adding new layers at the end of the network, allowing subsequent rounds of boosting to run much faster. In the original paper on Deep Incremental Boosting Mosca & Magoulas (2016a), this has been shown to be an effective way to learn the corrections introduced by the emphatisation of learning mistakes of the boosting process. The argument as to why this works effectively is based on the fact that the datasets at rounds s and t+ 1 will be mostly similar, and therefore a classifier ht that performs better than randomly on the resampled dataset Xt will also perform better than randomly on the resampled dataset Xt+1. This is under the assumption that both datasets are sampled from a common ancestor set Xa. It is subsequently shown that such a classifier can be re-trained on the differences between Xt and Xt+1.
This practically enables the ensemble algorithm to train the subsequent rounds for a considerably smaller number of epochs, consequently reducing the overall training time by a large factor. The original paper also provides a conjecture-based justification for why it makes sense to extend the previously trained network to learn the “corrections” taught by the boosting algorithm. A high level description of the method is shown in Algorithm 1, and the structure of the network at each round is illustrated in Figure 3.
Algorithm 1 Deep Incremental Boosting
D0(i) = 1/M for all i t = 0 W0 ← randomly initialised weights for first classifier while t < T do
Xt ← pick from original training set with distribution Dt ut ← create untrained classifier with additional layer of shape Lnew copy weights from Wt into the bottom layers of ut ht ← train ut classifier on current subset Wt+1 ← all weights from ht ǫt = 1 2 ∑ (i,y)∈B Dt(i)(1− ht(xi, yi) + ht(xi, y)) βt = ǫt/(1− ǫt) Dt+1(i) = Dt(i) Zt · β(1/2)(1+ht(xi,yi)−ht(xi,y)) where Zt is a normalisation factor such that Dt+1 is a distribution αt =
1 βt
t = t+ 1 end while H(x) = argmaxy∈Y ∑T t=1 logαtht(x, y)
3 CREATING THE BOOSTED RESIDUAL NETWORK
In this section we propose a method for generating Boosted Residual Networks. This works by increasing the size of an original residual network by one residual block at each round of boosting. The method achieves this by selecting an injection point index pi at which the new block is to be added, which is not necessarily the last block in the network, and by transferring the weights from the layers below pi in the network trained at the previous round of boosting.
Because the boosting method performs iterative re-weighting of the training set to skew the resample at each round to emphasize the training examples that are harder to train, it becomes necessary to utilise the entire ensemble at test time, rather than just use the network trained in the last round. This has the effect that the Boosted Residual Networks cannot be used as a way to train a single Residual Network incrementally. However, as we will discuss later, it is possible to alleviate this situation by deriving an approach that uses bagging instead of boosting; therefore removing the necessity to use the entire ensemble at test time. It is also possible to delete individual blocks from a Residual Network at training and/or testing time, as presented in He et al. (2015a), however this issue is considered out of the scope of this paper.
The iterative algorithm used in the paper is shown in Algorithm 2. At the first round, the entire training set is used to train a network of the original base architecture, for a number of epochs n0. After the first round, the following steps are taken at each subsequent round t:
• The ensemble constructed so far is evaluated on the training set to obtain the set errors ǫ, so that a new training set can be sampled from the original training set. This is a step common to all boosting algorithms.
• A new network is created, with the addition of a new block of layers Bnew immediately after position pt, which is determined as an initial pre-determined position p0 plus an offset i ∗ δp for all the blocks added at previous layers. This puts the new block of layers immediately after the block of layers added at the previous round, so that all new blocks are effectively added sequentially.
• The weights from the layers below pt are copied from the network trained at round t − 1 to the new network. This step allows to considerably shorten the training thanks to the transfer of learning shown in Yosinski et al. (2014).
• The newly created network is subsequently trained for a reduced number of epochs nt>0.
• The new network is added to the ensemble following the traditional rules and weight αt used in AdaBoost.
Figure 3 shows a diagram of how the Ensemble is constructed by deriving the next network at each round of boosting from the network used in the previous round.
Algorithm 2 Boosted Residual Networks
D0(i) = 1/M for all i t = 0 W0 ← randomly initialised weights for first classifier set initial injection position p0 while t < T do
Xt ← pick from original training set with distribution Dt ut ← create untrained classifier with an additional block Bnew of pre-determined shapeNnew determine block injection position pt = pt−1 + |Bnew| connect the input of Bnew to the output of layer pt − 1 connect the output of Bnew and of layer pt − 1 to a merge layer mi connect the merge layer to the remainder of the network copy weights from Wt into the bottom layers l < pt of ut ht ← train ut classifier on current subset Wt+1 ← all weights from ht ǫt = 1 2 ∑ (i,y)∈B Dt(i)(1− ht(xi, yi) + ht(xi, y)) βt = ǫt/(1− ǫt) Dt+1(i) = Dt(i) Zt · β(1/2)(1+ht(xi,yi)−ht(xi,y)) where Zt is a normalisation factor such that Dt+1 is a distribution αt =
1 βt
t = t+ 1 end while H(x) = argmaxy∈Y ∑T t=1 logαtht(x, y)
We identified a number of optional variations to the algorithm that may be implemented in practice, which we have empirically established as not having an impact on the overall performance of the network. We report them here for completeness.
• Freezing the layers that have been copied from the previous round.
• Only utilising the weights distribution for the examples in the training set instead of resampling, as an input to the training algorithm.
• Inserting the new block always at the same position, rather than after the previouslyinserted block (we found this to affect performance negatively).
3.1 COMPARISON TO APPROXIMATE ENSEMBLES
While both Residual Networks and Densely Connected Convolutional Networks may be unfolded into an equivalent ensemble, we note that there is a differentiation between an actual ensemble method and an ensemble “approximation”. During the creation of an ensemble, one of the principal factors is the creation of diversity: each base learner is trained independently, on variations (resamples in the case of boosting algorithms) of the training set, so that each classifier is guaranteed to learn a different function that represents an approximation of the training data. This is the enabling factor for the ensemble to perform better in aggregate.
In the case of Densely Connected Convolutional Networks (DCCN) specifically, one may argue that a partial unfolding of the network could be, from a schematic point of view, very similar to an ensemble of incrementally constructed Residual Networks. We make the observation that, although this would be correct, on top of the benefit of diversity, our method also provides a much faster training methodology: the only network that is trained for a full schedule is the network created at the first round, which is also the smallest one. All subsequent networks are trained for a much shorter schedule, saving a considerable amount of time. Additionally, while the schematic may seem identical, there is a subtle difference: each member network outputs a classification of its own, which is then aggregated by weighted averaging, whilst in a DCCN the input of the final aggregation layer is the output of each underlying set of layers. We conjecture that this aggressive dimensionality reduction before the aggregation will have a regularising effect on the ensemble.
4 EXPERIMENTS AND DISCUSSION
In the experiments we used the MNIST, CIFAR-10 and CIFAR-100 datasets, and compared Boosted Residual Networks (BRN) with an equivalent Deep Incremental Boosting (DIB) without the skipconnections, AdaBoost with the equivalent Residual Network as its base classifier (AdaBoost), and the single Residual Network (Single Net) In order to reduce noise, we aligned the random initialisation of all networks across experiments, by fixing the seeds for the random number generators, and no dataset augmentation was used, both online and offline. Results are reported in Table 1, while Figure 4 shows a side-by-side comparison of accuracy levels at each round of boosting for both DIB and BRN on the MNIST and CIFAR-100 test sets. This figure illustrates how BRNs are able to consistently outperform DIB, regardless of ensemble size, and although such differences still fall within a Bernoulli confidence interval of 95%, we make the note that this does not take account of the fact that all the random initialisations were aligned, so both methods started with the exact same network.
Table 2 shows that this is achieved without significant changes in the training time1. The main speed increase is due to the fact that the only network being trained with a full schedule is the first network, which is also the smallest, whilst all other derived networks are trained for a much shorter schedule (in this case only 10% of the original training schedule).
The initial network architectures for the first round of boosting are shown in Table 3a for MNIST, and Table 3b for CIFAR-10 and CIFAR-100. It is worth mentioning that we used relatively simple network architectures that were fast to train, which still perform well on the datasets at hand, with accuracy close to, but not comparable to, the state-of-the-art. This enabled us to test larger Ensembles within an acceptable training time.
Training used the WAME method (Mosca & Magoulas (2016b)), which has been shown to be faster than Adam and RMSprop, whilst still achieving comparable generalisation. This is thanks to a
1In some cases BRN is actually faster than DIB, but we believe this to be just noise due to external factors such as system load.
specific weight-wise learning rate acceleration factor that is determined based only on the sign of the current and previous partial derivative ∂E(x) ∂wij . For the single Residual Network, and for the networks in AdaBoost, we trained each member for 100 epochs. For Deep Incremental Boosting and Boosted Residual Networks, we trained the first round for 50 epochs, and every subsequent round for 10 epochs, and ran all the algorithms for 10 rounds of boosting, except for the single network. The structure of each incremental block added to Deep Incremental Boosting and Boosted Residual Networks at each round is shown in Table 4a for MNIST, and in Table 4b for CIFAR-10 and CIFAR-100. All layers were initialised following the reccommendations in He et al. (2015b).
Distilled Boosted Residual Network: DBRN In another set of experiments we tested the performance of a Distilled Boosted Residual Network (DBRN). Distillation has been shown to be an effective process for regularising large Ensembles of Convolutional Networks in Mosca & Magoulas (2016c), and we have applied the same methodology to the proposed Boosted Residual Network. For the distilled network structure we used the same architecture as that of the Residual Network from the final round of boosting. Accuracy results in testing are presented in Table 5, and for completeness of comparison we also report the results for the distillation of DIB, following the same procedure, as DDIB.
Bagged Residual Networks: BARN We experimented with substituting the boosting algorithm with a simpler bagging algorithm (Breiman (1996)) to evaluate whether it would be possible to only use the network from the final round of bagging as an approximation of the Ensemble. We called this the Bagged Approximate Residual Networks (BARN) method. We then also tested the performance of the Distilled version of the whole Bagging Ensemble for comparison. The results are reported as “DBARN”. The results are reported in Table 6. It is clear that trying to use the last round of bagging is not comparable to using the entire Bagging ensemble at test time, or deriving a new distilled network from it.
5 CONCLUSIONS AND FUTURE WORK
In this paper we have derived a new ensemble algorithm specifically tailored to Convolutional Networks to generate Boosted Residual Networks. We have shown that this surpasses the performance of a single Residual Network equivalent to the one trained at the last round of boosting, of an ensemble of such networks trained with AdaBoost, and Deep Incremental Boosting on the MNIST and CIFAR datasets, without using augmentation techniques.
We then derived and looked at a distilled version of the method, and how this can serve as an effective way to reduce the test-time cost of running the Ensemble. We used Bagging as a proxy to test generating the approximate Residual Network, which, with the parameters tested, does not perform as well as the original Residual Network, BRN or DBRN.
Further experimentation of the Distilled methods presented in the paper, namely DBRN and DBARN, is necessary to fully investigate their behaviour. This is indeed part of our work in the near future. Additionally, the Residual Networks built in our experiments were comparatively smaller than those that achieve state-of-the-art performance. Reaching state-of-the-art on specific benchmark datasets was not our goal, instead we intended to show that we developed a methodology that makes it feasible to created ensembles of Residual Networks following a “white-box” approach to
significantly improve the training times and accuracy levels. Nevertheless, it might be appealing in the future to evaluate the performance improvements obtained when creating ensembles of larger, state-of-the-art, networks. Additional further investigation could also be conducted on the creation of Boosted Densely Connected Convolutional Networks, by applying the same principle to DCCN instead of Residual Networks.

This paper proposes a boosting based ensemble procedure for residual networks by adopting the Deep Incremental Boosting method that was used for CNN's(Mosca & Magoulas, 2016a). At each step t, a new block of layers are added to the network at a position p_t and the weights of all layers are copied to the current network to speed up training.
The method is not sufficiently novel since the steps of Deep Incremental Boosting are slightly adopted. Instead of adding a layer to the end of the network, this version adds a block of layers to a position p_t (starts at a selected position p_0) and merges layer accordingly hence slightly adopts DIB.
The empirical analysis does not use any data-augmentation. It is not clear whether the improvements (if there is) of the ensemble disappear after data-augmentation.  Also, one of the main baselines, DIB has no-skip connections therefore this can negatively affect the fair comparison. The authors argue that they did not involve state of art Res Nets since their analysis focuses on the ensemble approach, however any potential improvement of the ensemble can be compensated with an inherent feature of Res Net variant. The boosting procedure can be computationally restrictive in case of ImageNet training and Res Net variants may perform much better in that case too. Therefore the baselines should include the state of art Res Nets and Dense Convolutional networks hence current results are preliminary.
In addition, it is not clear how sensitive the boosting to the selection of injection point.
This paper adopts DIB to Res Nets and provides some empirical analysis however the contribution is not sufficiently novel and the empirical results are not satisfactory for demonstrating that the method is significant.
Pros
-provides some preliminary results for boosting of Res Nets
Cons
-not sufficiently novel: an incremental approach
-empirical analysis is not satisfactory


Restricted Boltzmann Machine (RBM) is a bipartite graphical model that is used as the building block in energy-based deep generative models. Due to its numerical stability and quantifiability of its likelihood, RBM is commonly used with Bernoulli units. Here, we consider an alternative member of the exponential family RBM with leaky rectified linear units – called leaky RBM. We first study the joint and marginal distributions of the leaky RBM under different leakiness, which leads to interesting interpretation of the leaky RBM model as truncated Gaussian distribution. We then propose a simple yet efficient method for sampling from this model, where the basic idea is to anneal the leakiness rather than the energy; – i.e., start from a fully Gaussian/Linear unit and gradually decrease the leakiness over iterations. This serves as an alternative to the annealing of the temperature parameter and enables numerical estimation of the likelihood that are more efficient and far more accurate than the commonly used annealed importance sampling (AIS). We further demonstrate that the proposed sampling algorithm enjoys relatively faster mixing than contrastive divergence algorithm, which improves the training procedure without any additional computational cost.
1 INTRODUCTION
In this paper, we are interested in deep generative models. One may naively classify these models into a family of directed deep generative models trainable by back-propagation (e.g., Kingma & Welling, 2013; Goodfellow et al., 2014), and deep energy-based models, such as deep belief network (Hinton et al., 2006) and deep Boltzmann machine (Salakhutdinov & Hinton, 2009). The building block of deep energy-based models is a bipartite graphical model called restricted Boltzmann machine (RBM). The RBM model consists of two layers, visible and hidden. The resulting graphical model which can account for higher-order interactions of the visible units (visible layer) using the hidden units (hidden layer). It also makes the inference easier that there are no interactions between the variables in each layer.
The conventional RBM uses Bernoulli units for both the hidden and visible units (Smolensky, 1986). One extension is using Gaussian visible units to model general natural images (Freund & Haussler, 1994). For hidden units, we can also generalize Bernoulli units to the exponential family (Welling et al., 2004; Ravanbakhsh et al., 2016).
Nair & Hinton (2010) propose a variation using Rectified Linear Unit (ReLU) for the hidden layer with a heuristic sampling procedure, which has promising performance in terms of reconstruction error and classification accuracy. Unfortunately, due to its lack of strict monotonicity, ReLU RBM does not fit within the framework of exponential family RBMs (Ravanbakhsh et al., 2016). Instead we study leaky-ReLU RBM (leaky RBM) in this work and address two important issues i) a better training (sampling) algorithm for ReLU RBM and; ii) a better quantification of leaky RBM –i.e., evaluation of its performance in terms of likelihood.
We study some of the fundamental properties of leaky RBM, including its joint and marginal distributions (Section 2). By analyzing these distributions, we show that the leaky RBM is a union of
truncated Gaussian distributions. In this paper, we show that training leaky RBM involves underlying positive definite constraints. Because of this, the training can diverge if these constrains are not satisfied. This is an issue that was previously ignored in ReLU RBM, as it was mainly used for pre-training rather than generative modeling.
Our contribution in this paper is three-fold: I) we systematically identify and address model constraints in leaky RBM (Section 3); II) for the training of leaky RBM, we propose a meta algorithm for sampling, which anneals leakiness during the Gibbs sampling procedure (Section 3) and empirically show that it can boost contrastive divergence with faster mixing (Section 5); III) We demonstrate the power of the proposed sampling algorithm on estimating the partition function. In particular, comparison on several benchmark datasets shows that the proposed method outperforms the conventional AIS (Salakhutdinov & Murray, 2008) in terms of efficiency and accuracy (Section 4). Moreover, we provide an incentive for using leaky RBM by showing that the leaky ReLU hidden units perform better than the Bernoulli units in terms of the model log-likelihood (Section 4).
2 RESTRICTED BOLTZMANN MACHINE AND RELU
The Boltzmann distribution is defined as p(x) = e−E(x)/Z where Z = ∑ x e −E(x) is the partition function. Restricted Boltzmann Machine (RBM) is a Boltzmann distribution with a bipartite structure It is also the building block for many deep models (e.g., Hinton et al., 2006; Salakhutdinov & Hinton, 2009; Lee et al., 2009), which are widely used in numerous applications (Bengio, 2009). The conventional Bernoulli RBM, models the joint probability p(v, h) for the visible units v ∈ [0, 1]I and the hidden units h ∈ [0, 1]J as p(v, h) ∝ exp(−E(v, h)), where E(v, h) = a>v − v>Wh + b>h. The parameters are a ∈ RI , b ∈ RJ and W ∈ RI×J . We can derive the conditional probabilities as
p(vi = 1|h) = σ  J∑ j=1 Wijhj + ai  and p(hj = 1|v) = σ( I∑ i=1 Wijvi + bj ) , (1)
where σ(x) = (1 + e−x)−1 is the sigmoid function.
One extension of Bernoulli RBM is replacing the binary visible units by linear units v ∈ RI with independent Gaussian noise. The energy function in this case is given by
E(v, h) = I∑ i=1 (vi − ai)2 2σ2i − I∑ i=1 J∑ j=1 vi σi Wijhj + b >h.
To simplify the notation, we assume a normalized data so that ai and σi is no longer required. The energy function is accordingly simplified to E(v, h) = ‖v‖ 2
2 − v >Wh + b>h (Note that the
elimination does not influence the discussion and one can easily extend all the results in this paper to the model that includes ai and σi.).
The conditional distributions are as follows:
p(vi|h) = N  J∑ j=1 Wijhj , 1  and p(hj = 1|v) = σ( I∑ i=1 Wijvi + bj ) , (2)
where N (µ, V ) is a Gaussian distribution with mean µ and variance V . To simplify the notation, in the following we define ηj = ∑I i=1Wijvi + bj – that is ηj is the input to the j
th hidden layer neuron and similarly define νi = ∑J j=1Wijhj + ai. Using this notation the conditionals in the (2) are p(vi|νi) = N (νi, 1) and p(hj = 1|ηj) = σ(ηj).
2.1 RELU RBM WITH CONTINUOUS VISIBLE UNITS
From (1) and (2), we can see that the mean of the p(hj |v) is the nonlinearity of the hidden unit at ηj = ∑I i=1Wijvi + bj – e.g., mean of the Bernoulli unit is the sigmoid function. From this perspective, we can extend the sigmoid function to other functions and thus allow RBM to have more expressive power (Ravanbakhsh et al., 2016). In particular, it would be interesting to use rectified linear unit (ReLU) nonlinearity, f(ηj) = max(0, ηj), for generative modeling.
Nair & Hinton (2010) use an RBM with visible Gaussian unit and ReLU hidden activation functions for pretraining. They suggest sampling from max(0, ηj+N (0, σ(ηj)) for conditional sampling from the hidden units (compare to (2)). However, this sampling heuristic does not suggest the parametric form of the joint ReLU-Gaussian distribution. This also means we cannot evaluate it using methods such as Annealed Importance Sampling that require access to this parametric form. In fact, only strictly monotonic activation functions can derive feasible joint and conditional distributions in the exponential familly RBM and ReLU is not strictly monotonic Ravanbakhsh et al. (2016). Similar activation functions that are monotonic are Softplus, f(ηj) = log(1 + eηj ) and leaky ReLU (Maas et al., 2013), defined as f(ηj) = max(cηj , ηj), where c ∈ (0, 1) is the leakiness parameter. In contrast to the ReLU RBM the joint parametric form of these two distributions are available. However, the energy (logarithm of the joint probability) in the case of Softplus activation function contains a polylogarithmic term that requires evaluation of an infinite series; see Table 1 in Ravanbakhsh et al. (2016). For this reason, here we focus on Leaky-ReLU activation function.
By Ravanbakhsh et al. (2016), the conditional probability of the activation, assuming the nonlinearity f(ηj), is generally defined as p(hj |v) = exp (−Df (ηj‖hj) + g(hj)), where Df (ηj‖hj) is the Bregman Divergence associated with f, and g(hj) is the base (or carrier) measure in the exponential family which ensures the distribution is well-defined. The Bergman divergence, for strictly monotonic function f , is Df (ηj‖hj) = −ηjhj + F (ηj) + F ∗(hj), where F with ddηj F (ηj) = f(ηj) is the anti-derivative (integral) of f and F ∗ is the anti-derivative of f−1 (i.e., f−1(f(η)) = η); Note that due to the strict monotonicity of f , f−1 is well-defined, and F and F ∗ are commonly referred to as conjugate duals.
Considering the leaky ReLU activation function f(η) = max(cη, η), using this formalism, the conditional distributions of hidden units in the leaky RBM simplifies to (see Appendix A.1 for details)
p(hj |v) = { N (ηj , 1), if ηj > 0 N (cηj , c), if ηj ≤ 0.
(3)
Since the visible units uses the identity function, the corresponding conditional distribution is a Gaussian1
p(vi|h) = N  J∑ j=1 Wijhj , 1  , (4) Having these two conditional distributions is enough for training a leaky RBM model using contrastive divergence (Hinton, 2002) or some other alternatives (e.g., Tieleman, 2008; Tieleman & Hinton, 2009).
3 TRAINING AND SAMPLING FROM LEAKY RBM
Given the conditional distributions p(v|h) and p(h|v), the joint distribution p(v, h) from the general treatment for MRF model is (Yang et al., 2012; Ravanbakhsh et al., 2016)
p(v, h) ∝ exp v>Wh− I∑ i=1 (F̃ ∗(vi) + g(vi))− J∑ j=1 (F ∗(hj) + g(hj))  , (5) where F̃ ∗(vi) and F ∗(hj) are anti-derivatives of the inverses of the activation functions f̃(vi) and f(hj) for visible units vi and hidden units hj , respectively (see Section 2.1). Assuming f(ηj) = max(cηj , c) and f̃(νi) = νi in leaky-ReLU RBM, the joint distribution above becomes (see Appendix A.2 for details)
p(v, h) ∝ exp v>Wh− ‖v‖2 2 − ∑ ηj>0 ( h2j 2 + log √ 2π ) − ∑ ηj≤0 ( h2j 2c + log √ 2cπ ) + b>h  , 1which can also be written as p(vi|h) = exp ( −Df̃ (νi‖vi) + g(vi) ) , where νi = ∑ j=1Wijhj and
f̃(νi) = νi and Df̃ (νi‖vi) = (νi − vi) 2 and g(vi) = − log
√ 2π.
and the corresponding visible marginal distribution is
p(v) ∝ exp −1 2 v> I −∑ ηj>0 WjW > j − c ∑ ηj≤0 WjW > j  v + ∑ ηj>0 bjW > j v + c ∑ ηj≤0 bjW > j v  . (6)
where Wj is the j-th column of W .
3.1 LEAKY RBM AS UNION OF TRUNCATED GAUSSIAN DISTRIBUTIONS
From (6) we see that the marginal probability is determined by the affine constraints ηj > 0 or ηj ≤ 0 for all hidden units j. By combinatorics, these constraints divide RI (the visible domain) into at most M = ∑I i=1 ( J i ) convex regions R1, · · ·RM . An example with I = 2 and J = 3 is shown in Figure 1. If I > J , then we have at most 2J regions.
We discuss the two types of these regions. For bounded regions, such as R1 in Figure 1, the integration of (6) is also bounded, which results in a valid distribution. Before we discuss the unbounded cases, we define Ω = I − ∑J j=1 αjWjW > j , where αj = 1ηj>0 + c1ηj≤0. For the unbounded region, if Ω ∈ RI×I is a positive definite (PD) matrix, then the probability density is proportional to a multivariate Gaussian distribution with mean µ = Ω−1 (∑J j=1 αjbjWj ) and precision matrix Ω (covariance matrix Ω−1) but over an affine-constrained region. Therefore, the distribution of each unbounded region can be treated as a truncated Gaussian distribution. The marginal distrubution can be treated as a union of truncated Gaussain distribution. Note that leaky RBM is different from Su et al. (2017), which use single truncated Gaussian distribution to model joint (conditional) distributions and require approximated and more complicated sampling algorithms for truncated Gaussian distribution, while leaky RBM only requires to sample from Gaussian distributions.
On the other hand, if Ω is not PD, and the region Ri contains the eigenvectors with negative eigenvalues of Ω, the integration of (6) over Ri is divergent (infinite), which can not result in a valid probability distribution. In practice, with this type of parameter, when we do Gibbs sampling on the conditional distributions, the sampling will diverge. However, it is unfeasible to check exponentially many regions for each gradient update. Theorem 1. If I −WW> is positive definite, then I − ∑ j αjWjW > j is also positive definite, for all αj ∈ [0, 1].
The proof is shown in Appendix 1. From Theorem 1 we can see that if the constraint I −WW> is PD, then one can guarantee that the distribution of every region is a valid truncated Gaussian distribution. Therefore, we introduce the following projection step for each W after the gradient update.
argmin W̃
‖W − W̃‖2F
s.t. I − W̃W̃> 0 (7)
Theorem 2. The above projection step (7) can be done by shrinking the singular values to be less than 1.
The proof is shown in Appendix C. The training algorithm of the leaky RBM is shown in Algorithm 1. By using the projection step (7), we could treat the leaky RBM as the union of truncated Gaussian distributions, which uses weight vectors to divide the space of visible units into several regions and use a truncated Gaussian distribution to model each region. Note that the leaky RBM model is different from Su et al. (2016), which uses a truncated Gaussian distribution to model the conditional distribution p(h|v) instead of the marginal distribution. The empirical study about the divergent values and the necessity of the projection step is shown in Appendix D. Without the projection step, when we run Gibbs sampling for several iterations from the model, the sampled values will diverge because the model does not have a valid marginal distribution p(v). It also implies that we cannot train leaky RBM with larger CD steps without projection, which would result in divergent gradients. The detailed discussion is shown in Appendix D.
Algorithm 1 Training Leaky RBM for t = 1, . . . , T do
Estimate gradient gθ by CD or other algorithms with (13) and (4), where θ = {W,a, b}. θ(t) ← θ(t−1) + ηgθ. Project W (t) by (7).
end for
3.2 SAMPLING FROM LEAKY-RELU RBM
Gibbs sampling is the core procedure for RBM, including training, inference, and estimating the partition function (Fischer & Igel, 2012; Tieleman, 2008; Salakhutdinov & Murray, 2008). For every task, we start from randomly initializing v by an arbitrary distribution q, and iteratively sample from the conditional distributions. Gibbs sampling guarantees the procedure result in the stationary distribution in the long run for any initialized distribution q. However, if q is close to the target distribution p, it can significantly shorten the number of iterations to achieve the stationary distribution. If we set the leakiness c to be 1, then (6) becomes a simple multivariate Gaussian distribution N ( (I −WW>)−1Wb, (I −WW>)−1 ) , which can be easily sampled without Gibbs sampling. Also, the projection step (7) guarantees it is a valid Gaussian distribution. Then we decrease the leakiness with a small , and use samples from the multivariate Gaussian distribution when c = 1 as the initialization to do Gibbs sampling. Note that the distribution of each region is a truncated Gaussian distribution. When we only decrease the leakiness with a small amount, the resulted distribution is a “similar” truncated Gaussian distribution with more concentrated density. From this observation, we could expect the original multivariate Gaussian distribution serves as a good initialization. The one-dimensional example is shown in Figure 2. We then repeat this procedure until we reach the target leakiness. The algorithm can be seen as annealing the leakiness during the Gibbs sampling procedure. The meta algorithm is shown in Algorithm 2. Next, we show the proposed sampling algorithm can help both the partition function estimation and the training of leaky RBM.
Algorithm 2 Meta Algorithm for Sampling from Leaky RBM. Sample v from N ( (I −WW>)−1Wb, (I −WW>)−1 ) = (1− c)/T and c′ = 1 for t = 1, . . . , T do
Decrease c′ = c′ − and perform Gibbs sampling by using (13) and (4) with leakiness c′ end for
4 PARTITION FUNCTION ESTIMATION
It is known that estimating the partition function of RBM is intractable (Salakhutdinov & Murray, 2008). Existing approaches, including Salakhutdinov & Murray (2008); Grosse et al. (2013); Liu et al. (2015); Carlson et al. (2016) focus on using sampling to approximate the partition function of the conventional Bernoulli RBM instead of the RBM with Gaussian visible units and non-Bernoulli hidden units. In this paper, we focus on extending the classic annealed importance sampling (AIS) algorithm (Salakhutdinov & Murray, 2008) to leaky RBM.
Assuming that we want to estimate the partition function Z of p(v) with p(v) = p∗(v)/Z and p∗(v) ∝ ∑ h exp(−E(v, h)), Salakhutdinov & Murray (2008) start from a initial distribution
p0(v) ∝ ∑ h exp(−E0(v, h)), where computing the partition Z0 of p0(v) is tractable and we can draw samples from p0(v). They then use the “geometric path” to anneal the intermediate distribution as pk(v) ∝ p∗k(v) = ∑ h exp (−βkE0(v, h)− (1− βk)E(v, h)), where they grid βk from 1 to 0. If we let β0 = 1, we can draw samples vk from pk(v) by using samples vk−1 from pk−1(v) for k ≥ 1 via Gibbs sampling. The partition function is then estimated via Z = Z0M ∑M i=1 ω (i), where
ω(i) = p∗1(v (i) 0 )
p∗0(v (i) 0 )
p∗2(v (i) 1 ) p∗1(v (i) 1 ) · · · p∗K−1(v (i) K−2) p∗K−2(v (i) K−2) p∗K(v (i) K−1) p∗K−1(v (i) K−1) , and βK = 0.
Salakhutdinov & Murray (2008) use the initial distribution with independent visible units and without hidden units. We consider application of AIS to the leaky-ReLU case with E0(v, h) = ‖v‖2 2 , which results in a multivariate Gaussian distribution p0(v). Compared with the meta algorithm shown in Algorithm 2 which anneals between leakiness, AIS anneals between energy functions.
4.1 STUDY ON TOY EXAMPLES
As we discussed in Section 3.1, leaky RBM with J hidden units is a union of 2J truncated Gaussian distributions. Here we perform a study on the leaky RBM with a small number hidden units. Since in this example the number of hidden units is small, we can integrate out all possible configurations of h. However, integrating a truncated Gaussian distribution with general affine constraints does not have analytical solutions, and several approximations have been developed (e.g., Pakman & Paninski, 2014). To compare our results with the exact partition function, we consider a special case that has the following form:
p(v) ∝ exp −1 2 v> I −∑ ηj>0 WjW > j − c ∑ ηj≤0 WjW > j  v  . (8)
Compared to (6), it is equivalent to the setting where b = 0. Geometrically, everyWj passes through the origin. We further put the additional constraint Wi ⊥ Wj ,∀i 6= j. Therefore. we divide the whole space into 2J equally-sized regions. A three dimensional example is shown in Figure 3. Then the partition function of this special case has the analytical form
Z = 1
2J ∑ αj∈{1,c},∀j (2π)− I 2 ∣∣∣∣∣∣∣ I − J∑ j=1 αjWjW > j − 12 ∣∣∣∣∣∣∣ .
We randomly initialize W and use SVD to make columns orthogonal. Also, we scale ‖Wj‖ to satisfy I −WW> 0. The leakiness parameter is set to be 0.01. For Salakhutdinov & Murray (2008) (AIS-Energy), we use 105 particles with 105 intermediate distributions. For the proposed method (AIS-Leaky), we use only 104 particles with 103 intermediate distributions. In this small problem we study the cases when the model has 5, 10, 20 and 30 hidden units and 3072 visible units. The true log partition function logZ is shown in Table 1 and the difference between logZ and the estimates given by the two algorithms are shown in Table 2.
From Table 1, we observe that AIS-Leaky has significantly better and more stable estimations than AIS-Energy especially and this gap increases as we increase the number of hidden units. AIS-Leaky achieves this with orders magnitude reduced computation –e.g., here it uses ∼.1% of resources used by conventional AIS. For example, when we increase J from 5 to 30, the bias (difference) of AIS-Leaky only increases from 0.02 to 0.13; however, the bias of AIS-Energy increases from 1.76 to 9.6. We further study the implicit connection between the proposed AIS-Leaky and AIS-Energy in Appendix E, which shows AIS-Leaky is a special case of AIS-Energy under certain conditions.
4.2 COMPARISON BETWEEN LEAKY-RELU RBM AND BERNOULLI-GAUSSIAN RBM
It is known that the reconstruction error is not a proper approximation of the likelihood (Hinton, 2012). One commonly adopted way to compare generative models is to sample from the model, and visualize the images to check the quality. However, Theis et al. (2016) show the better visualization does not imply better likelihood. Also, the single layer model cannot adequately model the complicated natural images (the result for Bernoulli-Gaussian RBM has been shown in Ranzato & Hinton (2010)), which makes the visualization comparison difficult (Appendix F has few visualization results).
Fortunately, our accurate estimate of the partition function for leaky RBM can produce a reliable quantitative estimate of the representation power of leaky RBM. We compare the BernoulliGaussian RBM2, which has Bernoulli hidden units and Gaussian visible units. We trained both models with CD-203 and momentum. For both model, we all used 500 hidden units. We initialized W by sampling from Unif(0, 0.01), a = 0, b = 0 and σ = 1. The momentum parameter was 0.9 and the batch size was set to 100. We tuned the learning rate between 10−1 and 10−6. We studied two benchmark data sets, including CIFAR10 and SVHN. The data was normalized to have zero mean and standard deviation of 1 for each pixel. The results of the log-likelihood are reported in Table 3.
From Table 3, leaky RBM outperforms Bernoulli-Gaussian RBM significantly. The unsatisfactory performance of Bernoulli-Gaussian RBM may be in part due to the optimization procedure. If we tune the decay schedule of the learning-rate for each dataset in an ad-hoc way, we observe the performance of Bernoulli-Gaussian RBM can be improved by ∼ 300 nats for both datasets. Also, increasing CD-steps brings slight improvement. The other possibility is the bad mixing during the CD iterations. The advanced algorithms Tieleman (2008); Tieleman & Hinton (2009) may help. Although Nair & Hinton (2010) demonstrate the power of ReLU in terms of reconstruction error and classification accuracy, it does not imply its superior generative capability. Our study confirms leaky RBM could have much better generative performance compared to Bernoulli-Gaussian RBM.
5 BETTER MIXING BY ANNEALING LEAKINESS
In this section, we show the idea of annealing between leakiness benefit the mixing in Gibbs sampling in other settings. A common procedure for comparison of sampling methods for RBM is through visualization. Here, we are interested in more quantitative metrics and the practical benefits of improved sampling. For this, we consider optimization performance as the evaluation metric.
The gradient of the log-likelihood function L(θ|vdata) of general RBM models is
∂L(θ|vdata) ∂θ = Eh|vdata
[ ∂E(v, h)
∂θ
] − Ev,h [ ∂E(v, h)
∂θ
] . (9)
Since the second expectation in (9) is usually intractable, different approximation algorithms are used (Fischer & Igel, 2012).
2Our GPU implementation with gnumpy and cudamat can reproduce the results of http://www.cs.toronto.edu/ tang/code/GaussianRBM.m
3CD-n means that contrastive divergence was run for n steps
In this section, we compare two gradient approximation procedures. The baselines are the conventional contrastive divergence (CD) (Hinton, 2002) and persistent contrastive divergence (Tieleman, 2008) (PCD). The second method is using Algorithm 2 (Leaky) with the same number of mixing steps as CD. The experiment setup is the same as that of Section 4.
The results are shown in Figure 4. The proposed sampling procedure is slightly better than typical CD steps. The reason is we only anneals the leakiness for 20 steps. To get accurate estimation requires thousands of steps as shown in Section 4 when we estimate the partition function. Therefore, the estimated gradient is still inaccurate. However, it still outperforms the conventional CD algorithm. On the other hand, unlike the binary RBM case shown in Tieleman (2008), PCD does not outperform CD with 20 mixing steps for leaky RBM.
The drawback of Algorithm 2 is that sampling v from N ( (I −WW>)−1Wb, (I −WW>)−1 ) requires computing mean, covariance and the Cholesky decomposition of the covariance matrix in every iteration, which are computationally expensive. We study a mixture algorithm by combining CD and the idea of annealing leakiness. The mixture algorithm replaces the sampling from N ( (I −WW>)−1Wb, (I −WW>)−1 ) with sampling from the empirical data distribution. The resulted mix algorithm is almost the same as CD algorithm while it anneals the leakiness over the iterations as Algorithm 2. The results of the mix algorithm is also shown in Figure 4.
The mix algorithm is slightly worse than the original leaky algorithm, but it also outperforms the conventional CD algorithm without additional computation cost. The comparison in terms of CPU time is shown in Appendix F. Annealing the leakiness helps the mix algorithm explore different modes of the distribution, thereby improves the training. The idea could also be combined with more advanced algorithms (Tieleman, 2008; Tieleman & Hinton, 2009)4.
6 CONCLUSION
In this paper, we study the properties of the exponential family distribution produced by leaky RBM. This study relates the leaky RBM model and truncated Gaussian distribution and reveals an underlying positive definite constraint of training leaky RBM. We further proposed a meta sampling algorithm, which anneals between leakiness during the Gibbs sampling procedure. We first demonstrate the proposed sampling algorithm is significantly more effective and efficient in estimating the partition function than the conventional AIS algorithm. Second, we show that the proposed sampling algorithm has comparatively better mixing properties (compared to CD). A few direction are worth further study; in particular we are investigating on speeding up the naive projection step; either using the barrier function as shown in Hsieh et al. (2011) or by eliminating the need for projection by artificially bounding the domain via additional constraints.
4We studied the PCD extension of the proposed sampling algorithm. However, the performance is not as stable as CD.
A DERIVATION OF LEAKY (RELU) RBM
A.1 CONDITIONAL DISTRIBUTIONS
For leaky RBM, the activation function of hidden units is defined as f(ηj) = max(cηj , ηj), where c ∈ (0, 1) and ηj = ∑I i=1Wijvi + bj . The inverse function of f is f
−1(hj) = min(hj , hj/c). Therefore, the anti-derivatives are
F (ηj) =
{ 1 2η 2 j , if ηj > 0
c 2η 2 j , else,
(10)
and
F ∗(hj) =
{ 1 2h 2 j , if ηj > 0
1 2ch 2 j , else.
(11)
The activation function of Gaussian visible units can be treated as the linear unit f̃(νi) = νi, where νi = ∑J j=1Wijhj . Following the similar steps for deriving F and F
∗, we get the anti-derivatives F̃ (νi) = 1 2ν 2 i and F̃ ∗(vi) = 1 2v 2 i .
From Ravanbakhsh et al. (2016), the conditional distribution is defined as
p(hj |ηj) = exp (−ηjhj + F (ηj) + F ∗(hj)) (12)
By plugging F and F ∗ into (12), we get the conditional distribution for leaky RBM
p(hj |v) = { N (ηj , 1)with g(hj) = − log( √ 2π), if ηj > 0
N (cηj , c)with g(hj) = − log( √ 2cπ), if ηj ≤ 0. (13)
Similarly, we have p(vi|νi) = N (νi, 1) with g(vi) = − log( √ 2π).
A.2 JOINT AND MARGINAL DISTRIBUTIONS
Given the conditional distributions p(v|h) and p(h|v), the joint distribution p(v, h) from the general treatment for MRF model given by Yang et al. (2012) is
p(v, h) ∝ exp v>Wh− I∑ i=1 (F̃ ∗(vi) + g(vi))− J∑ j=1 (F ∗(hj) + g(hj))  , (14) By plugging F ∗, F̃ ∗ and g from Section A.1 into (14), we have
p(v, h) ∝ exp v>Wh− ‖v‖2 2 − ∑ ηj>0 ( h2j 2 + log √ 2π ) − ∑ ηj≤0 ( h2j 2c + log √ 2cπ ) + b>h  , Then the marginal distribution is
p(v) ∝ ∫ h p(v, h)dh
∝ ∫ h exp ( −‖v‖ 2 2 ) ∏ ηj>0 exp ( − h2j 2 + ηjhj − log √ 2π ) ∏ ηj≤0 ( − h2j 2c + hjηj − log √ 2cπ ) dh
∝ exp ( −‖v‖ 2
2 ) ∏ ηj>0 exp ( η2j 2 ) ∏ ηj≤0 ( cη2j 2 )
∝ exp −1 2 v> I −∑ ηj>0 WjW > j − c ∑ ηj≤0 WjW > j  v + ∑ ηj>0 bjW > j v + c ∑ ηj≤0 bjW > j v  .
B PROOF OF THEOREM 1
Proof. Since WW>− ∑ j αjWjWj = ∑ j(1−αj)WjW>j 0, we have WW> ∑ j αjWjWj .
Therefore, I − ∑ j αjWjW > j I −WW> 0.
C PROOF OF THEOREM 2
Proof. Let the SVD decomposition of W and W̃ as W = USV > and W̃ = Ũ S̃Ṽ >. Then we have
‖W − W̃‖2F = ‖USV > − Ũ S̃Ṽ >‖2F ≥ I∑ i=1 (Sii − S̃ii)2, (15)
and the constraint I − W̃W̃> 0 can be rewritten as 0 ≤ S̃ii ≤ 1,∀i. The transformed problem has a Lasso-like formulation and we can solve it by S̃ii = min(Sii, 1) (Parikh & Boyd, 2014). Also, the lower bound ∑I i=1(Sii − S̃ii)2 in (15) becomes a tight bound when we set Ũ = U and Ṽ = V , which completes the proof.
D NECESSITY OF THE PROJECTION STEP
We conduct a short comparison to demonstrate the projection step is necessary for the leaky RBM on generative tasks. We train two leaky RBM as follows. The first model is trained by the same setting in Section 4. We use the convergence of log likelihood as the stopping criteria. The second model is trained by CD-1 with weight decay and without the projection step. We stop the training when the reconstruction error is less then 10−2. After we train these two models, we run Gibbs sampling with 1000 independent chains for several steps and output the average value of the visible units. Note that the visible units are normalized to zero mean. The results on SVHN and CIFAR10 are shown in Figure 5.
From Figure 5, the model trained by weight decay without projection step is suffered by the problem of the diverged values. It confirms the study shown in Section 3.1. It also implies that we cannot
train leaky RBM with larger CD steps when we do not do projection; otherwise, we would have the diverged gradients. Therefore, the projection is necessary for training leaky RBM for the generative purpose. However, we also oberseve that the projection step is not necessary for the classification and reconstruction tasks. he reason may be the independency of different evaluation criteria (Hinton, 2012; Theis et al., 2016) or other implicit reasons to be studied.
E EQUIVALENCE BETWEEN ANNEALING THE ENERGY AND LEAKINESS
We analyze the performance gap between AIS-Leaky and AIS-Energy. One major difference is the initial distribution. The intermediate marginal distribution of AIS-Energy has the following form:
pk(v) ∝ exp −1 2 v> I − (1− βk) ∑ ηj>0 WjW > j − (1− βk)c ∑ ηj≤0 WjW > j  v  . (16)
Here we eliminated the bias terms b for simplicity. Compared with Algorithm 2, (16) not only anneals the leakiness (1 − βk)c ∑ ηj≤0WjW > j when ηj ≤ 0, but also in the case (1 −
βk) ∑ ηj>0 WjW > j when ηj > 0, which brings more bias to the estimation. In other words, AIS-Leaky is a one-sided leakiness annealing while AIS-Energy is a two-sided leakiness annealing method.
To address the higher bias problem of AIS-Energy, we replace the initial distribution with the one used in Algorithm 2. By elementary calculation, the marginal distribution becomes
pk(v) ∝ exp −1 2 v> I −∑ ηj>0 WjW > j − (βk + (1− βk)c) ∑ ηj≤0 WjW > j  v  , (17)
which recovers the proposed Algorithm 2. From this analysis, we understand AIS-Leaky is a special case of conventional AIS-Energy with better initialization inspired by the study in Section 3. Also, by this connection between AIS-Energy and AIS-Leaky, we note that AIS-Leaky can be combined with other extensions of AIS (Grosse et al., 2013; Burda et al., 2015) as well.
F MORE EXPERIMENTAL RESULTS FOR SAMPLING
F.1 SAMPLED IMAGES
We show the sampled images from leaky RBM train on CIFAR10 and SVHN datasets. We randomly initialize 20 chains and run Gibbs sampling for 1000 iterations. The sampled results are shown in Figure 6 The results shows that single layer RBM does not adequately model CIFAR10 and SVHN
when compared to multilayer models. The similar results for single layer Bernoulli-Gaussian RBM from Ranzato & Hinton (2010) (in gray scale) is shown in Figure 7. Therefore, we instead focused on quantitative evaluation of the log-likelihood in Table 3.
F.2 COMPUTATIONAL TIME BETWEEN DIFFERENT SAMPLING STRATEGIES
The comparison in terms of CPU time of different sampling algorithms discussed in Section 5 is shown in Figure 8. Please note that the complexity of CD and Mix are the almost the same. Mix only need a few more constant time steps which can be ignored compared with sampling steps. Leaky is more time-consuming because of computing and decomposing the covariance matrix as we discussed in Section 5. We also report the execution time of each step of algorithms in Table 4.
F.3 STUDY ON RELU-BERNOULLI RBM
We study the idea of annealing leakiness on the RBM model with leaky ReLU hidden units and Bernoulli visible units. We create the toy dataset with 20, 25 and 30 visible units as shown in Figure 9. The small datasets allow exact computation of the partition function. For each dataset, we sample 60,000 images for training and 10,000 images for testing. We use 100 hidden units and PCD to train the model. The log likelihood results are shown in Table 5.
Compared to the Gaussian visible units case we study in Section 3, where p(v) is a multi-variate Gaussian distribution when c = 1, the partition function of p(v) in ReLU-Bernoulli when c = 1 does not have the analytical form. Therefore, we do the following two-stage alternative. We first run the standard AIS algorithm, which anneals the energy, to the distribution with leakiness c = 1. We then change to anneals the leakiness from 1 to the target value. For the typical AIS algorithm (AIS-Energy), we use 104 chains with 2 × 104 intermediate distributions. For the proposed twostaged algorithm (AIS-Leaky), we use 104 chains with 104 intermediate distributions for annealing to c = 1 and the other 104 distributions for annealing the leakiness. The results are shown in Table 6.
In Table 6, the standard AIS algorithm (AIS-Energy) has unsatisfactory performance. We show the performance of AIS for estimating the partition function of models with different leakiness on Toy20. We use the 104 independent chains and 2 × 104 intermediate distributions. The results are shown in Table 7. From Table 7, we observe that the AIS performances worse when the leakiness is closer to 0. Although we observed that increasing chains and intermediate distributions could improve the performance, but the improvements are limited. The study demonstrates when the
(a) I = 20
(b) I = 25
non-linearity of the distribution increases (the leakiness value c decreases), the standard AIS cannot effectively estimate the partition function within feasible computational time. On the other hand, it also confirm the proposed idea, annealing the leakiness, can serve as an effective building block for algorithms without enhancing the algorithm complexity. Note that the unsatisfactory performance of AIS may be addressed by Grosse et al. (2013). From Appendix E, the two-stage algorithm used here can also be improved by applying Grosse et al. (2013).
F.3.1 MNIST AND CALTECH DATASETS
We study MNIST and Caltech 101 Silhouettes datasets with 500 hidden units and train the model with CD-25. The results are shown in Table 8 and Table 9. The leaky RBM is better than conventional Bernoulli RBM and some deep models on MNIST data. Although leaky RBM deos not outperform Su et al. (2017), but it enjoys the advantage of the simpler sampling procedure (Gaussian distribution vs truncated Gaussian distribution) in the binary visible unit case.

The authors propose a novel energy-function for RBMs, using the leaky relu max(cx, x) activation function for the hidden-units. Analogous to ReLU units in feed-forward networks, these leaky relu RBMs split the input space into a combinatorial number of regions, where each region defines p(v) as a truncated Gaussian. A further contribution of the paper is in proposing a novel sampling scheme for the leaky RBM: one can run a much shorter Markov chain by initializing it from a sample of the leaky RBM with c=1 (which yields a standard multi-variate normal over the visibles) and then slowly annealing c. In low-dimension a similar scheme is shown to outperform AIS for estimating the partition function. Experiments are performed on both CIFAR-10 and SVHN.
This is an interesting paper which I believe would be of interest to the ICLR community. The theoretical contributions are strong: the authors not only introduce a proper energy formulation of ReLU RBMs, but also a novel sampling mechanism and an improvement on AIS for estimating their partition function.
Unfortunately, the experimental results are somewhat limited. The PCD baseline is notably absent. Including (bernoulli visible, leaky-relu hidden) would have allowed the authors to evaluate likelihoods on standard binary RBM datasets. As it stands, performance on CIFAR-10 and SVHN, while improved with leaky-relu, is a far cry from more recent generative models (VAE-based, or auto-regressive models). While this comparison may be unfair, it will certainly limit the wider appeal of the paper to the community. Furthermore, there is the issue of the costly projection method which is required to guarantee that the energy-function remain bounded (covariance matrix over each region be PSD). Again, while it may be fair to leave that for future work given the other contributions, this will further limit the appeal of the paper.
PROS:
Introduces an energy function having the leaky-relu as an activation function
Introduces a novel sampling procedure based on annealing the leakiness parameter
Similar sampling scheme shown to outperform AIS
CONS:
Results are somewhat out of date
Missing experiments on binary datasets (more comparable to prior RBM work)
Missing PCD baseline
Cost of projection method