In this work, we investigate a novel training procedure to learn a generative model as the transition operator of a Markov chain, such that, when applied repeatedly on an unstructured random noise sample, it will denoise it into a sample that matches the target distribution from the training set. The novel training procedure to learn this progressive denoising operation involves sampling from a slightly different chain than the model chain used for generation in the absence of a denoising target. In the training chain we infuse information from the training target example that we would like the chains to reach with a high probability. The thus learned transition operator is able to produce quality and varied samples in a small number of steps. Experiments show competitive results compared to the samples generated with a basic Generative Adversarial Net.
1 INTRODUCTION AND MOTIVATION
To go beyond the relatively simpler tasks of classification and regression, advancing our ability to learn good generative models of high-dimensional data appears essential. There are many scenarios where one needs to efficiently produce good high-dimensional outputs where output dimensions have unknown intricate statistical dependencies: from generating realistic images, segmentations, text, speech, keypoint or joint positions, etc..., possibly as an answer to the same, other, or multiple input modalities. These are typically cases where there is not just one right answer but a variety of equally valid ones following a non-trivial and unknown distribution. A fundamental ingredient for such scenarios is thus the ability to learn a good generative model from data, one from which we can subsequently efficiently generate varied samples of high quality.
Many approaches for learning to generate high dimensional samples have been and are still actively being investigated. These approaches can be roughly classified under the following broad categories:
• Ordered visible dimension sampling (van den Oord et al., 2016; Larochelle & Murray, 2011). In this type of auto-regressive approach, output dimensions (or groups of conditionally independent dimensions) are given an arbitrary fixed ordering, and each is sampled conditionally on the previous sampled ones. This strategy is often implemented using a recurrent network (LSTM or GRU). Desirable properties of this type of strategy are that the exact log likelihood can usually be computed tractably, and sampling is exact. Undesirable properties follow from the forced ordering, whose arbitrariness feels unsatisfactory especially for domains that do not have a natural ordering (e.g. images), and imposes for high-dimensional output a long sequential generation that can be slow. • Undirected graphical models with multiple layers of latent variables. These make inference, and thus learning, particularly hard and tend to be costly to sample from (Salakhutdinov & Hinton, 2009). • Directed graphical models trained as variational autoencoders (VAE) (Kingma & Welling, 2014; Rezende et al., 2014)
∗Associate Fellow, Canadian Institute For Advanced Research (CIFAR)
• Adversarially-trained generative networks. (GAN)(Goodfellow et al., 2014) • Stochastic neural networks, i.e. networks with stochastic neurons, trained by an adapted
form of stochastic backpropagation • Generative uses of denoising autoencoders (Vincent et al., 2010) and their generalization
as Generative Stochastic Networks (Alain et al., 2016) • Inverting a non-equilibrium thermodynamic slow diffusion process (Sohl-Dickstein et al.,
2015) • Continuous transformation of a distribution by invertible functions (Dinh et al. (2014), also
used for variational inference in Rezende & Mohamed (2015))
Several of these approaches are based on maximizing an explicit or implicit model log-likelihood or a lower bound of its log-likelihood, but some successful ones are not e.g. GANs. The approach we propose here is based on the notion of “denoising” and thus takes its root in denoising autoencoders and the GSN type of approaches. It is also highly related to the non-equilibrium thermodynamics inverse diffusion approach of Sohl-Dickstein et al. (2015). One key aspect that distinguishes these types of methods from others listed above is that sample generation is achieved thanks to a learned stochastic mapping from input space to input space, rather than from a latent-space to input-space.
Specifically, in the present work, we propose to learn to generate high quality samples through a process of progressive, stochastic, denoising, starting from a simple initial “noise” sample generated in input space from a simple factorial distribution i.e. one that does not take into account any dependency or structure between dimensions. This, in effect, amounts to learning the transition operator of a Markov chain operating on input space. Starting from such an initial “noise” input, and repeatedly applying the operator for a small fixed number T of steps, we aim to obtain a high quality resulting sample, effectively modeling the training data distribution. Our training procedure uses a novel “target-infusion” technique, designed to slightly bias model sampling to move towards a specific data point during training, and thus provide inputs to denoise which are likely under the model’s sample generation paths. By contrast with Sohl-Dickstein et al. (2015) which consists in inverting a slow and fixed diffusion process, our infusion chains make a few large jumps and follow the model distribution as the learning progresses.
The rest of this paper is structured as follows: Section 2 formally defines the model and training procedure. Section 3 discusses and contrasts our approach with the most related methods from the literature. Section 4 presents experiments that validate the approach. Section 5 concludes and proposes future work directions.
2 PROPOSED APPROACH
2.1 SETUP
We are given a finite data set D containing n points in Rd, supposed drawn i.i.d from an unknown distribution q∗. The data set D is supposed split into training, validation and test subsets Dtrain, Dvalid, Dtest. We will denote q∗train the empirical distribution associated to the training set, and use x to denote observed samples from the data set. We are interested in learning the parameters of a generative model p conceived as a Markov Chain from which we can efficiently sample. Note that we are interested in learning an operator that will display fast “burn-in” from the initial factorial “noise” distribution, but beyond the initial T steps we are not concerned about potential slow mixing or being stuck. We will first describe the sampling procedure used to sample from a trained model, before explaining our training procedure.
2.2 GENERATIVE MODEL SAMPLING PROCEDURE
The generative model p is defined as the following sampling procedure:
• Using a simple factorial distribution p(0)(z(0)), draw an initial sample z(0) ∼ p(0), where z(0) ∈ Rd. Since p(0) is factorial, the d components of z(0) are independent: p0 cannot model any dependency structure. z(0) can be pictured as essentially unstructured random noise. • Repeatedly apply T times a stochastic transition operator p(t)(z(t)|z(t−1)), yielding a more “denoised” sample z(t) ∼ p(t)(z(t)|z(t−1)), where all z(t) ∈ Rd.
• Output z(T ) as the final generated sample. Our generative model distribution is thus p(z(T )), the marginal associated to joint p(z(0), . . . , z(T )) = p(0)(z(0)) (∏T t=1 p (t)(z(t)|z(t−1)) ) .
In summary, samples from model p are generated, starting with an initial sample from a simple distribution p(0), by taking the T thsample along Markov chain z(0) → z(1) → z(2) → . . . → z(T ) whose transition operator is p(t)(z(t)|z(t−1)). We will call this chain the model sampling chain. Figure 1 illustrates this sampling procedure using a model (i.e. transition operator) that was trained on MNIST. Note that we impose no formal requirement that the chain converges to a stationary distribution, as we simply read-out z(T ) as the samples from our model p. The chain also needs not be time-homogeneous, as highlighted by notation p(t) for the transitions.
The set of parameters θ of model p comprise the parameters of p(0) and the parameters of transition operator p(t)(z(t)|z(t−1)). For tractability, learnability, and efficient sampling, these distributions will be chosen factorial, i.e. p(0)(z(0)) = ∏d i=1 p (0) i (z (0) i ) and p
(t)(z(t)|z(t−1)) =∏d i=1 p (t) i (z (t) i |z(t−1)). Note that the conditional distribution of an individual component i, p (t) i (z (t) i |z(t−1)) may however be multimodal, e.g. a mixture in which case p(t)(z(t)|z(t−1)) would be a product of independent mixtures (conditioned on z(t−1)), one per dimension. In our experiments, we will take the p(t)(z(t)|z(t−1)) to be simple diagonal Gaussian yielding a Deep Latent Gaussian Model (DLGM) as in Rezende et al. (2014).
2.3 INFUSION TRAINING PROCEDURE
We want to train the parameters of model p such that samples from Dtrain are likely of being generated under the model sampling chain. Let θ(0) be the parameters of p(0) and let θ(t) be the parameters of p(t)(z(t)|z(t−1)). Note that parameters θ(t) for t > 0 can straightforwardly be shared across time steps, which we will be doing in practice. Having committed to using (conditionally) factorial distributions for our p(0)(z(0)) and p(t)(z(t)|z(t−1)), that are both easy to learn and cheap to sample from, let us first consider the following greedy stagewise procedure. We can easily learn p(0)i (z
(0)) to model the marginal distribution of each component xi of the input, by training it by gradient descent on a maximum likelihood objective, i.e.
θ(0) = argmax θ Ex∼q∗train [ log p(0)(x; θ) ] (1)
This gives us a first, very crude unstructured (factorial) model of q∗.
Having learned this p(0), we might be tempted to then greedily learn the next stage p(1) of the chain in a similar fashion, after drawing samples z(0) ∼ p(0) in an attempt to learn to “denoise” the sampled z(0) into x. Yet the corresponding following training objective θ(1) = argmaxθ Ex∼q∗train,z(0)∼p(0) [ log p(1)(x|z(0); θ) ] makes no sense: x and z(0) are sampled independently of each other so z(0) contains no information about x, hence p(1)(x|z(0)) = p(1)(x). So maximizing this second objective becomes essentially the same as what we did when learning p(0). We would learn nothing more. It is essential, if we hope to learn a useful conditional distribution p(1)(x|z(0)) that it be trained on particular z(0) containing some information about x. In other words, we should not take our training inputs to be samples from p(0) but from a slightly different distribution, biased towards containing some information about x. Let us call it q(0)(z(0)|x). A natural choice for it, if it were possible, would be to take q(0)(z(0)|x) = p(z(0)|z(T ) = x) but this is an intractable inference, as all intermediate z(t) between z(0) and z(T ) are effectively latent states that we would need to marginalize over. Using a workaround such as a variational or MCMC approach would be a usual fallback. Instead, let us focus on our initial intent of guiding a progressive stochastic denoising, and think if we can come up with a different way to construct q(0)(z(0)|x) and similarly for the next steps q(t)i (z̃ (t) i |z̃(t−1),x).
Eventually, we expect a sequence of samples from Markov chain p to move from initial “noise” towards a specific example x from the training set rather than another one, primarily if a sample along the chain “resembles” x to some degree. This means that the transition operator should learn to pick up a minor resemblance with an x in order to transition to something likely to be even more similar to x. In other words, we expect samples along a chain leading to x to both have high probability under the transition operator of the chain p(t)(z(t)|z(t−1)), and to have some form of at least partial “resemblance” with x likely to increase as we progress along the chain. One highly inefficient way to emulate such a chain of samples would be, for teach step t, to sample many candidate samples from the transition operator (a conditionally factorial distribution) until we generate one that has some minimal “resemblance” to x (e.g. for a discrete space, this resemblance measure could be based on their Hamming distance). A qualitatively similar result can be obtained at a negligible cost by sampling from a factorial distribution that is very close to the one given by the transition operator, but very slightly biased towards producing something closer to x. Specifically, we can “infuse” a little of x into our sample by choosing for each input dimension, whether we sample it from the distribution given for that dimension by the transition operator, or whether, with a small probability, we take the value of that dimension from x. Samples from this biased chain, in which we slightly “infuse” x, will provide us with the inputs of our input-target training pairs for the transition operator. The target part of the training pairs is simply x.
2.3.1 THE INFUSION CHAIN
Formally we define an infusion chain z̃(0) → z̃(1) → . . . → z̃(T−1) whose distribution q(z̃(0), . . . , z̃(T−1)|x) will be “close” to the sampling chain z(0) → z(1) → z(2) → . . . → z(T−1) of model p in the sense that q(t)(z̃(t)|z̃(t−1),x) will be close to p(t)(z(t)|z(t−1)), but will at every step be slightly biased towards generating samples closer to target x, i.e. x gets progressively “infused” into the chain. This is achieved by defining q(0)i (z̃ (0) i |x) as a mixture between p (0) i (with a large mixture weight) and δxi , a concentrated unimodal distribution around xi, such as a Gaussian with small variance (with a small mixture weight)1. Formally q(0)i (z̃ (0) i |x) = (1 − α(t))p(0)i (z̃ (0) i ) + α (t)δxi(z̃ (0) i ), where 1 − α(t) and α(t) are the mixture weights 2. In other words, when sampling a value for z̃(0)i from q (0) i there will be a small probability α (0) to pick value close to xi (as sampled from δxi ) rather than sampling the value from p (0) i . We call α(t) the infusion rate. We define the transition operator of the infusion chain similarly as: q (t) i (z̃ (t) i |z̃(t−1),x) = (1− α(t))p (t) i (z̃ (t) i |z̃(t−1)) + α(t)δxi(z̃ (t) i ).
1Note that δxi does not denote a Dirac-Delta but a Gaussian with small sigma. 2In all experiments, we use an increasing schedule α(t) = α (t−1) +ω with α (0)
and ω constant. This allows to build our chain such that in the first steps, we give little information about the target and in the last steps we give more informations about the target. This forces the network to have less confidence (greater incertitude) at the beginning of the chain and more confidence on the convergence point at the end of the chain.
2.3.2 DENOISING-BASED INFUSION TRAINING PROCEDURE
For all x ∈ Dtrain: • Sample from the infusion chain z̃ = (z̃(0), . . . , z̃(T−1)) ∼ q(z̃(0), . . . , z̃(T−1)|x).
precisely so: z̃0 ∼ q(0)(z̃(0)|x) . . . z̃(t) ∼ q(t)(z̃(t)|z̃(t−1),x) . . . • Perform a gradient step so that p learns to “denoise” every z̃(t) into x.
θ(t) ← θ(t) + η(t) ∂ log p (t)(x|z̃(t−1); θ(t)) ∂θ(t)
where η(t) is a scalar learning rate. 3
As illustrated in Figure 2, the distribution of samples from the infusion chain evolves as training progresses, since this chain remains close to the model sampling chain.
2.4 STOCHASTIC LOG LIKELIHOOD ESTIMATION
The exact log-likelihood of the generative model implied by our model p is intractable. The logprobability of an example x can however be expressed using proposal distribution q as:
log p(x) = logEq(z̃|x) [ p(z̃,x)
q(z̃|x)
] (2)
Using Jensen’s inequality we can thus derive the following lower bound:
log p(x) ≥ Eq(z̃|x) [log p(z̃,x)− log q(z̃|x)] (3)
where log p(z̃,x) = log p(0)(z̃(0)) + (∑T−1
t=1 log p (t)(z̃(t)|z̃(t−1))
) + log p(T )(x|z̃(T−1)) and
log q(z̃|x) = log q(0)(z̃(0)|x) + ∑T−1 t=1 log q
(t)(z̃(t)|z̃(t−1),x). 3Since we will be sharing parameters between the p(t), in order for the expected larger error gradients on the earlier transitions not to dominate the parameter updates over the later transitions we used an increasing schedule η(t) = η0 tT for t ∈ {1, . . . , T}.
A stochastic estimation can easily be obtained by replacing the expectation by an average using a few samples from q(z̃|x). We can thus compute a lower bound estimate of the average log likelihood over training, validation and test data.
Similarly in addition to the lower-bound based on Eq.3 we can use the same few samples from q(z̃|x) to get an importance-sampling estimate of the likelihood based on Eq. 24.
2.4.1 LOWER-BOUND-BASED INFUSION TRAINING PROCEDURE
Since we have derived a lower bound on the likelihood, we can alternatively choose to optimize this stochastic lower-bound directly during training. This alternative lower-bound based infusion training procedure differs only slightly from the denoising-based infusion training procedure by using z̃(t) as a training target at step t (performing a gradient step to increase log p(t)(z̃(t)|z̃(t−1); θ(t))) whereas denoising training always uses x as its target (performing a gradient step to increase log p(t)(x|z̃(t−1); θ(t))). Note that the same reparametrization trick as used in Variational Autoencoders (Kingma & Welling, 2014) can be used here to backpropagate through the chain’s Gaussian sampling.
3 RELATIONSHIP TO PREVIOUSLY PROPOSED APPROACHES
3.1 MARKOV CHAIN MONTE CARLO FOR ENERGY-BASED MODELS
Generating samples as a repeated application of a Markov transition operator that operates on input space is at the heart of Markov Chain Monte Carlo (MCMC) methods. They allow sampling from an energy-model, where one can efficiently compute the energy or unnormalized negated log probability (or density) at any point. The transition operator is then derived from an explicit energy function such that the Markov chain prescribed by a specific MCMC method is guaranteed to converge to the distribution defined by that energy function, as the equilibrium distribution of the chain. MCMC techniques have thus been used to obtain samples from the energy model, in the process of learning to adjust its parameters.
By contrast here we do not learn an explicit energy function, but rather learn directly a parameterized transition operator, and define an implicit model distribution based on the result of running the Markov chain.
3.2 VARIATIONAL AUTO-ENCODERS
Variational auto-encoders (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) also start from an unstructured (independent) noise sample and non-linearly transform this into a distribution that matches the training data. One difference with our approach is that the VAE typically maps from a lower-dimensional space to the observation space. By contrast we learn a stochastic transition operator from input space to input space that we repeat for T steps. Another key difference, is that the VAE learns a complex heavily parameterized approximate posterior proposal q whereas our infusion based q can be understood as a simple heuristic proposal distribution based on p. Importantly the specific heuristic we use to infuse x into q makes sense precisely because our operator is a map from input space to input space, and couldn’t be readily applied otherwise. The generative network in Rezende et al. (2014) is a Deep Latent Gaussian Model (DLGM) just as ours. But their approximate posterior q is taken to be factorial, including across all layers of the DLGM, whereas our infusion based q involves an ordered sampling of the layers, as we sample from q(t)(z̃(t)|z̃(t−1),x). More recent proposals involve sophisticated approaches to sample from better approximate posteriors, as the work of Salimans et al. (2015) in which Hamiltonian Monte Carlo is combined with variational inference, which looks very promising, though computationally expensive, and Rezende & Mohamed (2015) that generalizes the use of normalizing flows to obtain a better approximate posterior.
4Specifically, the two estimates (lower-bound and IS) start by collecting k samples from q(z̃|x) and computing for each the corresponding ` = log p(z̃,x) − log q(z̃|x). The lower-bound estimate is then obtained by averaging the resulting `1, . . . `k, whereas the IS estimate is obtained by taking the log of the averaged e`1 , . . . , e`k (in a numerical stable manner as logsumexp(`1, . . . , `k)− log k).
3.3 SAMPLING FROM AUTOENCODERS AND GENERATIVE STOCHASTIC NETWORKS
Earlier works that propose to directly learn a transition operator resulted from research to turn autoencoder variants that have a stochastic component, in particular denoising autoencoders (Vincent et al., 2010), into generative models that one can sample from. This development is natural, since a stochastic auto-encoder is a stochastic transition operator form input space to input space. Generative Stochastic Networks (GSN) (Alain et al., 2016) generalized insights from earlier stochastic autoencoder sampling heuristics (Rifai et al., 2012) into a more formal and general framework. These previous works on generative uses of autoencoders and GSNs attempt to learn a chain whose equilibrium distribution will fit the training data. Because autoencoders and the chain are typically started from or very close to training data points, they are concerned with the chain mixing quickly between modes. By contrast our model chain is always restarted from unstructured noise, and is not required to reach or even have an equilibrium distribution. Our concern is only what happens during the T “burn-in” initial steps, and to make sure that it transforms the initial factorial noise distribution into something that best fits the training data distribution. There are no mixing concerns beyond those T initial steps.
A related aspect and limitation of previous denoising autoencoder and GSN approaches is that these were mainly “local” around training samples: the stochastic operator explored space starting from and primarily centered around training examples, and learned based on inputs in these parts of space only. Spurious modes in the generated samples might result from large unexplored parts of space that one might encounter while running a long chain.
3.4 REVERSING A DIFFUSION PROCESS IN NON-EQUILIBRIUM THERMODYNAMICS
The approach of Sohl-Dickstein et al. (2015) is probably the closest to the approach we develop here. Both share a similar model sampling chain that starts from unstructured factorial noise. Neither are concerned about an equilibrium distribution. They are however quite different in several key aspects: Sohl-Dickstein et al. (2015) proceed to invert an explicit diffusion process that starts from a training set example and very slowly destroys its structure to become this random noise, they then learn to reverse this process i.e. an inverse diffusion. To maintain the theoretical argument that the exact reverse process has the same distributional form (e.g. p(x(t−1)|x(t)) and p(x(t)|x(t−1)) both factorial Gaussians), the diffusion has to be infinitesimal by construction, hence the proposed approaches uses chains with thousands of tiny steps. Instead, our aim is to learn an operator that can yield a high quality sample efficiently using only a small number T of larger steps. Also our infusion training does not posit a fixed a priori diffusion process that we would learn to reverse. And while the distribution of diffusion chain samples of Sohl-Dickstein et al. (2015) is fixed and remains the same all along the training, the distribution of our infusion chain samples closely follow the model chain as our model learns. Our proposed infusion sampling technique thus adapts to the changing generative model distribution as the learning progresses.
Drawing on both Sohl-Dickstein et al. (2015) and the walkback procedure introduced for GSN in Alain et al. (2016), a variational variant of the walkback algorithm was investigated by Goyal et al. (2017) at the same time as our work. It can be understood as a different approach to learning a Markov transition operator, in which a “heating” diffusion operator is seen as a variational approximate posterior to the forward “cooling” sampling operator with the exact same form and parameters, except for a different temperature.
4 EXPERIMENTS
We trained models on several datasets with real-valued examples. We used as prior distribution p(0) a factorial Gaussian whose parameters were set to be the mean and variance for each pixel through the training set. Similarly, our models for the transition operators are factorial Gaussians. Their mean and elementwise variance is produced as the output of a neural network that receives the previous z(t−1) as its input, i.e. p(t)(z(t)i |z(t−1)) = N (µi(z(t−1)), σ2i (z(t−1))) where µ and σ2 are computed as output vectors of a neural network. We trained such a model using our infusion training procedure on MNIST (LeCun & Cortes, 1998), Toronto Face Database (Susskind et al., 2010), CIFAR-10 (Krizhevsky & Hinton, 2009), and CelebA (Liu et al., 2015). For all datasets, the only preprocessing we did was to scale the integer pixel values down to range [0,1]. The network
trained on MNIST and TFD is a MLP composed of two fully connected layers with 1200 units using batch-normalization (Ioffe & Szegedy, 2015) 5. The network trained on CIFAR-10 is based on the same generator as the GANs of Salimans et al. (2016), i.e. one fully connected layer followed by three transposed convolutions. CelebA was trained with the previous network where we added another transposed convolution. We use rectifier linear units (Glorot et al., 2011) on each layer inside the networks. Each of those networks have two distinct final layers with a number of units corresponding to the image size. They use sigmoid outputs, one that predict the mean and the second that predict a variance scaled by a scalar β (In our case we chose β = 0.1) and we add an epsilon = 1e − 4 to avoid an excessively small variance. For each experiment, we trained the network on 15 steps of denoising with an increasing infusion rate of 1% (ω = 0.01, α (0)
= 0), except on CIFAR-10 where we use an increasing infusion rate of 2% (ω = 0.02, α (0) = 0) on 20 steps.
4.1 NUMERICAL RESULTS
Since we can’t compute the exact log-likelihood, the evaluation of our model is not straightforward. However we use the lower bound estimator derived in Section 2.4 to evaluate our model during training and prevent overfitting (see Figure 3). Since most previous published results on non-likelihood based models (such as GANs) used a Parzen-window-based estimator (Breuleux et al., 2011), we use it as our first comparison tool, even if it can be misleading (Lucas Theis & Bethge, 2016). Results are shown in Table 1, we use 10 000 generated samples and σ = 0.17 . To get a better estimate of the log-likelihood, we then computed both the stochastic lower bound and the importance sampling estimate (IS) given in Section 2.4. For the IS estimate in our MNIST-trained model, we used 20 000 intermediates samples. In Table 2 we compare our model with the recent Annealed Importance Sampling results (Wu et al., 2016). Note that following their procedure we add an uniform noise of 1/256 to the (scaled) test point before evaluation to avoid overevaluating models that might have overfitted on the 8 bit quantization of pixel values. Another comparison tool that we used is the Inception score as in Salimans et al. (2016) which was developed for natural images and is thus most relevant for CIFAR-10. Since Salimans et al. (2016) used a GAN trained in a semi-supervised way with some tricks, the comparison with our unsupervised trained model isn’t straightforward. However, we can see in Table 3 that our model outperforms the traditional GAN trained without labeled data.
4.2 SAMPLE GENERATION
Another common qualitative way to evaluate generative models is to look at the quality of the samples generated by the model. In Figure 4 we show various samples on each of the datasets we used. In order to get sharper images, we use at sampling time more denoising steps than in the training time (In the MNIST case we use 30 denoising steps for sampling with a model trained on 15 denoising steps). To make sure that our network didn’t learn to copy the training set, we show in the last column the nearest training-set neighbor to the samples in the next-to last column. We can see that our training method allow to generate very sharp and accurate samples on various dataset.
5We don’t share batch norm parameters across the network, i.e for each time step we have different parameters and independent batch statistics.
4.3 INPAINTING
Another method to evaluate a generative model is inpainting. It consists of providing only a partial image from the test set and letting the model generate the missing part. In one experiment, we provide only the top half of CelebA test set images and clamp that top half throughout the sampling chain. We restart sampling from our model several times, to see the variety in the distribution of the bottom part it generates. Figure 5 shows that the model is able to generate a varied set of bottom halves, all consistent with the same top half, displaying different type of smiles and expression. We also see that the generated bottom halves transfer some information about the provided top half of the images (such as pose and more or less coherent hair cut).
5 CONCLUSION AND FUTURE WORK
We presented a new training procedure that allows a neural network to learn a transition operator of a Markov chain. Compared to the previously proposed method of Sohl-Dickstein et al. (2015) based on inverting a slow diffusion process, we showed empirically that infusion training requires far fewer denoising steps, and appears to provide more accurate models. Currently, many successful generative models, judged on sample quality, are based on GAN architectures. However these require to use two different networks, a generator and a discriminator, whose balance is reputed delicate to adjust, which can be source of instability during training. Our method avoids this problem by using only a single network and a simpler training objective.
Denoising-based infusion training optimizes a heuristic surrogate loss for which we cannot (yet) provide theoretical guarantees, but we empirically verified that it results in increasing log-likelihood estimates. On the other hand the lower-bound-based infusion training procedure does maximize an explicit variational lower-bound on the log-likelihood. While we have run most of our experiments with the former, we obtained similar results on the few problems we tried with lower-bound-based infusion training.
Future work shall further investigate the relationship and quantify the compromises achieved with respect to other Markov Chain methods including Sohl-Dickstein et al. (2015); Salimans et al. (2015)
and also to powerful inference methods such as Rezende & Mohamed (2015). As future work, we also plan to investigate the use of more sophisticated neural net generators, similar to DCGAN’s (Radford et al., 2016) and to extend the approach to a conditional generator applicable to structured output problems.
ACKNOWLEDGMENTS
We would like to thank the developers of Theano (Theano Development Team, 2016) for making this library available to build on, Compute Canada and Nvidia for their computation resources, NSERC and Ubisoft for their financial support, and three ICLR anonymous reviewers for helping us improve our paper.
A DETAILS ON THE EXPERIMENTS
A.1 MNIST EXPERIMENTS
We show the impact of the infusion rate α(t) = α (t−1)
+ ω for different numbers of training steps on the lower bound estimate of log-likelihood on the Validation set of MNIST in Figure 6. We also show the quality of generated samples and the lower bound evaluated on the test set in Table 4. Each experiment in Table 4 uses the corresponding models of Figure 6 that obtained the best lower bound value on the validation set. We use the same network architecture as described in Section 4, i.e two fully connected layers with Relu activations composed of 1200 units followed by two distinct fully connected layers composed of 784 units, one that predicts the means, the other one that predicts the variances. Each mean and variance is associated with one pixel. All of the the parameters of the model are shared across different steps except for the batch norm parameters. During training, we use the batch statistics of the current mini-batch in order to evaluate our model on the train and validation sets. At test time (Table 4), we first compute the batch statistics over the entire train set for each step and then use the computed statistics to evaluate our model on the test test.
We did some experiments to evaluate the impact of α or ω in α(t) = α (t−1)
+ ω. Figure 6 shows that as the number of steps increases, the optimal value for infusion rate decreases. Therefore, if we want to use many steps, we should have a small infusion rate. These conclusions are valid for both increasing and constant infusion rate. For example, the optimal α for a constant infusion rate, in Figure 6e with 10 steps is 0.08 and in Figure 6f with 15 steps is 0.06. If the number of steps is not enough or the infusion rate is too small, the network will not be able to learn the target distribution as shown in the first rows of all subsection in Table 4.
In order to show the impact of having a constant versus an increasing infusion rate, we show in Figure 7 the samples created by infused and sampling chains. We observe that having a small infusion rate over many steps ensures a slow blending of the model distribution into the target distribution.
In Table 4, we can see high lower bound values on the test set with few steps even if the model can’t generate samples that are qualitatively satisfying. These results indicate that we can’t rely on the lower bound as the only evaluation metric and this metric alone does not necessarily indicate the suitability of our model to generated good samples. However, it is still a useful tool to prevent overfitting (the networks in Figure 6e and 6f overfit when the infusion rate becomes too high). Concerning the samples quality, we observe that having a small infusion rate over an adequate number of steps leads to better samples.
A.2 INFUSION AND MODEL SAMPLING CHAINS ON NATURAL IMAGES DATASETS
In order to show the behavior of our model trained by Infusion on more complex datasets, we show in Figure 8 chains on CIFAR-10 dataset and in Figure 9 chains on CelebA dataset. In each Figure, the first sub-figure shows the chains infused by some test examples and the second subfigure shows the model sampling chains. In the experiment on CIFAR-10, we use an increasing schedule α(t) = α (t−1) + 0.02 with α(0) = 0 and 20 infusion steps (this corresponds to the training parameters). In the experiment on CelebA, we use an increasing schedule α(t) = α (t−1)
+0.01 with α(0) = 0 and 15 infusion steps.

The paper presents a method for training a generative model via an iterative denoising procedure. The denoising process is initialized with a random sample from a crude approximation to the data distribution and produces a high quality sample via multiple denoising steps. Training is performed by setting-up a Markov chain that slowly blends propositions from the current denoising model with a real example from the data distribution; using this chain the current denoising model is updated towards reproducing the changed, "better", samples from the blending process.
This is a clearly written paper that considers an interesting approach for training generative models. I was intrigued by the simplicity of the presented approach and really enjoyed reading the paper.
The proposed method is novel although it has clear ties to other recent work aiming to use denoising models for sampling from distributions such as the work by Sohl-Dickstein and the recent work on using DAEs as generative models.
I think this general direction of research is important. The proposed procedure takes inspiration from the perspective of generating samples by minimizing an energy function via transitions along a Markov chain and, if successful, it can potentially sidestep many problems of current procedures for training directed generative models such as:
- convergence and mode coverage problems as in generative adversarial networks
- problems with modeling multi-modal distributions which can arise when a too restrictive approximate inference model is paired with a powerful generative model
That being said, another method that seems promising for addressing these issues that also has some superficially similarity to the presented work is the idea of combining Hamiltonian Monte Carlo inference with variational inference as in [1]. As such I am not entirely convinced that the method presented here will be able to perform better than the mentioned paper; although it might be simpler to train. Similarly, although I agree that using a MCMC chain to generate samples via a MC-EM like procedure is likely very costly I am not convinced such a procedure won't at least also work reasonably well for the simple MNIST example. In general a more direct comparison between different inference methods using an MCMC chain like procedure would be nice to have but I understand that this is perhaps out of the scope of this paper. One thing that I would have expected, however, is a direct comparison to the procedure from Sohl-Dickstein in terms of sampling steps and generation quality as it is so directly related.
Other major points (good and bad):
- Although in general the method is explained well some training details are missing. Most importantly it is never mentioned how alpha or omega are set (I am assuming omega is 0.01 as that is the increase mentioned in the experimental setup). It is also unclear how alpha affects the capabilities of the generator. While it intuitively seems reasonable to use a small alpha over many steps to ensure slow blending of the two distributions it is not clear how necessary this is or at what point the procedure would break (I assume alpha = 1 won't work as the generator then would have to magically denoise a sample from the relatively uninformative draw from p0 ?). The authors do mention in one of the figure captions that the denoising model does not produce good samples in only 1-2 steps but that might also be an artifact of training the model with small alpha (at least I see no a priori reason for this). More experiments should be carried out here.
- No infusion chains or generating chains are shown for any of the more complicated data distributions, this is unfortunate as I feel these would be interesting to look at.
- The paper does a good job at evaluating the model with respect to several different metrics. The bound on the log-likelihood is nice to have as well!
- Unfortunately the current approach does not come with any theoretical guarantees. It is unclear for what choices of alpha the procedure will work and whether there is some deeper connection to MCMC sampling or energy based models. In my eyes this does not subtract from the value of the paper but would perhaps be worth a short sentence in the conclusion.
Minor points:
- The second reference seems broken
- Figure 3 starts at 100 epochs and, as a result, contains little information. Perhaps it would be more useful to show the complete training procedure and put the x-axis on a log-scale ?
- The explanation regarding the convolutional networks you use makes no sense to me. You write that you use the same structure as in the "Improved GANs" paper which, unlike your model, generates samples from a fixed length random input. I thus suppose you don't really use a generator with 1 fully connected network followed by up-convolutions but rather have several stages of convolutions followed by a fully connected layer and then up-convolutions ?
- The choice of parametrizing the variance via a sigmoid output unit is somewhat unusual, was there a specific reason for this choice ?
- footnote 1 contains errors: "This allow to" -> "allows to",  "few informations" -> "little information". "This force the network" -> "forces"
- Page 1 error: etc...
- Page 4 error: "operator should to learn"
[1] Markov Chain Monte Carlo and Variational Inference: Bridging the Gap, Tim Salimans and Diedrik P. Kingma and Max Welling, ICML 2015
>>> Update <<<<
Copied here from my response below:
I believe the response of the authors clarifies all open issues. I strongly believe the paper should be accepted to the conference. The only remaining issue I have with the paper is that, as the authors acknowledge the architecture of the generator is likely highly sub-optimal and might hamper the performance of the method in the evaluation. This however does not at all subtract from any of the main points of the paper.
I am thus keeping my score as a clear accept. I want to emphasize that I believe the paper should be published (just in case the review process results in some form of cut-off threshold that is high due to overall "inflated" review scores).


The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.
1 INTRODUCTION
The ImageNet challenge has led to significant advancements in exploring various architectural choices in CNNs (Russakovsky et al. (2015); Krizhevsky et al. (2012); Simonyan & Zisserman (2015); Szegedy et al. (2015a); He et al. (2016)). The general trend since the past few years has been that the networks have grown deeper, with an overall increase in the number of parameters and convolution operations. These high capacity networks have significant inference costs especially when used with embedded sensors or mobile devices where computational and power resources may be limited. For these applications, in addition to accuracy, computational efficiency and small network sizes are crucial enabling factors (Szegedy et al. (2015b)). In addition, for web services that provide image search and image classification APIs that operate on a time budget often serving hundreds of thousands of images per second, benefit significantly from lower inference times.
There has been a significant amount of work on reducing the storage and computation costs by model compression (Le Cun et al. (1989); Hassibi & Stork (1993); Srinivas & Babu (2015); Han et al. (2015); Mariet & Sra (2016)). Recently Han et al. (2015; 2016b) report impressive compression rates on AlexNet (Krizhevsky et al. (2012)) and VGGNet (Simonyan & Zisserman (2015)) by pruning weights with small magnitudes and then retraining without hurting the overall accuracy. However, pruning parameters does not necessarily reduce the computation time since the majority of the parameters removed are from the fully connected layers where the computation cost is low, e.g., the fully connected layers of VGG-16 occupy 90% of the total parameters but only contribute less than 1% of the overall floating point operations (FLOP). They also demonstrate that the convolutional layers can be compressed and accelerated (Iandola et al. (2016)), but additionally require sparse
∗Work done at NEC Labs †Supported in part by the NSF under Grant IIS-13-2079
BLAS libraries or even specialized hardware (Han et al. (2016a)). Modern libraries that provide speedup using sparse operations over CNNs are often limited (Szegedy et al. (2015a); Liu et al. (2015)) and maintaining sparse data structures also creates an additional storage overhead which can be significant for low-precision weights.
Recent work on CNNs have yielded deep architectures with more efficient design (Szegedy et al. (2015a;b); He & Sun (2015); He et al. (2016)), in which the fully connected layers are replaced with average pooling layers (Lin et al. (2013); He et al. (2016)), which reduces the number of parameters significantly. The computation cost is also reduced by downsampling the image at an early stage to reduce the size of feature maps (He & Sun (2015)). Nevertheless, as the networks continue to become deeper, the computation costs of convolutional layers continue to dominate.
CNNs with large capacity usually have significant redundancy among different filters and feature channels. In this work, we focus on reducing the computation cost of well-trained CNNs by pruning filters. Compared to pruning weights across the network, filter pruning is a naturally structured way of pruning without introducing sparsity and therefore does not require using sparse libraries or any specialized hardware. The number of pruned filters correlates directly with acceleration by reducing the number of matrix multiplications, which is easy to tune for a target speedup. In addition, instead of layer-wise iterative fine-tuning (retraining), we adopt a one-shot pruning and retraining strategy to save retraining time for pruning filters across multiple layers, which is critical for pruning very deep networks. Finally, we observe that even for ResNets, which have significantly fewer parameters and inference costs than AlexNet or VGGNet, still have about 30% of FLOP reduction without sacrificing too much accuracy. We conduct sensitivity analysis for convolutional layers in ResNets that improves the understanding of ResNets.
2 RELATED WORK
The early work by Le Cun et al. (1989) introduces Optimal Brain Damage, which prunes weights with a theoretically justified saliency measure. Later, Hassibi & Stork (1993) propose Optimal Brain Surgeon to remove unimportant weights determined by the second-order derivative information. Mariet & Sra (2016) reduce the network redundancy by identifying a subset of diverse neurons that does not require retraining. However, this method only operates on the fully-connected layers and introduce sparse connections.
To reduce the computation costs of the convolutional layers, past work have proposed to approximate convolutional operations by representing the weight matrix as a low rank product of two smaller matrices without changing the original number of filters (Denil et al. (2013); Jaderberg et al. (2014); Zhang et al. (2015b;a); Tai et al. (2016); Ioannou et al. (2016)). Other approaches to reduce the convolutional overheads include using FFT based convolutions (Mathieu et al. (2013)) and fast convolution using the Winograd algorithm (Lavin & Gray (2016)). Additionally, quantization (Han et al. (2016b)) and binarization (Rastegari et al. (2016); Courbariaux & Bengio (2016)) can be used to reduce the model size and lower the computation overheads. Our method can be used in addition to these techniques to reduce computation costs without incurring additional overheads.
Several work have studied removing redundant feature maps from a well trained network (Anwar et al. (2015); Polyak & Wolf (2015)). Anwar et al. (2015) introduce a three-level pruning of the weights and locate the pruning candidates using particle filtering, which selects the best combination from a number of random generated masks. Polyak & Wolf (2015) detect the less frequently activated feature maps with sample input data for face detection applications. We choose to analyze the filter weights and prune filters with their corresponding feature maps using a simple magnitude based measure, without examining possible combinations. We also introduce network-wide holistic approaches to prune filters for simple and complex convolutional network architectures.
Concurrently with our work, there is a growing interest in training compact CNNs with sparse constraints (Lebedev & Lempitsky (2016); Zhou et al. (2016); Wen et al. (2016)). Lebedev & Lempitsky (2016) leverage group-sparsity on the convolutional filters to achieve structured brain damage, i.e., prune the entries of the convolution kernel in a group-wise fashion. Zhou et al. (2016) add group-sparse regularization on neurons during training to learn compact CNNs with reduced filters. Wen et al. (2016) add structured sparsity regularizer on each layer to reduce trivial filters, channels or even layers. In the filter-level pruning, all above work use `2,1-norm as a regularizer.
Similar to the above work, we use `1-norm to select unimportant filters and physically prune them. Our fine-tuning process is the same as the conventional training procedure, without introducing additional regularization. Our approach does not introduce extra layer-wise meta-parameters for the regularizer except for the percentage of filters to be pruned, which is directly related to the desired speedup. By employing stage-wise pruning, we can set a single pruning rate for all layers in one stage.
3 PRUNING FILTERS AND FEATURE MAPS
Let ni denote the number of input channels for the ith convolutional layer and hi/wi be the height/width of the input feature maps. The convolutional layer transforms the input feature maps xi ∈ Rni×hi×wi into the output feature maps xi+1 ∈ Rni+1×hi+1×wi+1 , which are used as input feature maps for the next convolutional layer. This is achieved by applying ni+1 3D filters Fi,j ∈ Rni×k×k on the ni input channels, in which one filter generates one feature map. Each filter is composed by ni 2D kernels K ∈ Rk×k (e.g., 3 × 3). All the filters, together, constitute the kernel matrix Fi ∈ Rni×ni+1×k×k. The number of operations of the convolutional layer is ni+1nik
2hi+1wi+1. As shown in Figure 1, when a filter Fi,j is pruned, its corresponding feature map xi+1,j is removed, which reduces nik2hi+1wi+1 operations. The kernels that apply on the removed feature maps from the filters of the next convolutional layer are also removed, which saves an additional ni+2k2hi+2wi+2 operations. Pruning m filters of layer i will reduce m/ni+1 of the computation cost for both layers i and i+ 1.
3.1 DETERMINING WHICH FILTERS TO PRUNE WITHIN A SINGLE LAYER
Our method prunes the less useful filters from a well-trained model for computational efficiency while minimizing the accuracy drop. We measure the relative importance of a filter in each layer by calculating the sum of its absolute weights ∑ |Fi,j |, i.e., its `1-norm ‖Fi,j‖1. Since the number
of input channels, ni, is the same across filters, ∑ |Fi,j | also represents the average magnitude of its kernel weights. This value gives an expectation of the magnitude of the output feature map. Filters with smaller kernel weights tend to produce feature maps with weak activations as compared to the other filters in that layer. Figure 2(a) illustrates the distribution of filters’ absolute weights sum for each convolutional layer in a VGG-16 network trained on the CIFAR-10 dataset, where the distribution varies significantly across layers. We find that pruning the smallest filters works better in comparison with pruning the same number of random or largest filters (Section 4.4). Compared to other criteria for activation-based feature map pruning (Section 4.5), we find `1-norm is a good criterion for data-free filter selection.
The procedure of pruning m filters from the ith convolutional layer is as follows: 1. For each filter Fi,j , calculate the sum of its absolute kernel weights sj = ∑ni
l=1
∑ |Kl|.
2. Sort the filters by sj . 3. Prune m filters with the smallest sum values and their corresponding feature maps. The
kernels in the next convolutional layer corresponding to the pruned feature maps are also removed.
4. A new kernel matrix is created for both the ith and i+ 1th layers, and the remaining kernel weights are copied to the new model.
Relationship to pruning weights Pruning filters with low absolute weights sum is similar to pruning low magnitude weights (Han et al. (2015)). Magnitude-based weight pruning may prune away whole filters when all the kernel weights of a filter are lower than a given threshold. However, it requires a careful tuning of the threshold and it is difficult to predict the exact number of filters that will eventually be pruned. Furthermore, it generates sparse convolutional kernels which can be hard to accelerate given the lack of efficient sparse libraries, especially for the case of low-sparsity.
Relationship to group-sparse regularization on filters Recent work (Zhou et al. (2016); Wen et al. (2016)) apply group-sparse regularization ( ∑ni j=1 ‖Fi,j‖2 or `2,1-norm) on convolutional filters, which also favor to zero-out filters with small l2-norms, i.e. Fi,j = 0. In practice, we do not observe noticeable difference between the `2-norm and the `1-norm for filter selection, as the important filters tend to have large values for both measures (Appendix 6.1). Zeroing out weights of multiple filters during training has a similar effect to pruning filters with the strategy of iterative pruning and retraining as introduced in Section 3.4.
3.2 DETERMINING SINGLE LAYER’S SENSITIVITY TO PRUNING
To understand the sensitivity of each layer, we prune each layer independently and evaluate the resulting pruned network’s accuracy on the validation set. Figure 2(b) shows that layers that maintain their accuracy as filters are pruned away correspond to layers with larger slopes in Figure 2(a). On the contrary, layers with relatively flat slopes are more sensitive to pruning. We empirically determine the number of filters to prune for each layer based on their sensitivity to pruning. For deep networks such as VGG-16 or ResNets, we observe that layers in the same stage (with the same feature map size) have a similar sensitivity to pruning. To avoid introducing layer-wise meta-parameters, we use the same pruning ratio for all layers in the same stage. For layers that are sensitive to pruning, we prune a smaller percentage of these layers or completely skip pruning them.
3.3 PRUNING FILTERS ACROSS MULTIPLE LAYERS
We now discuss how to prune filters across the network. Previous work prunes the weights on a layer by layer basis, followed by iteratively retraining and compensating for any loss of accuracy (Han et al. (2015)). However, understanding how to prune filters of multiple layers at once can be useful: 1) For deep networks, pruning and retraining on a layer by layer basis can be extremely time-consuming 2) Pruning layers across the network gives a holistic view of the robustness of the network resulting in a smaller network 3) For complex networks, a holistic approach may be necessary. For example, for the ResNet, pruning the identity feature maps or the second layer of each residual block results in additional pruning of other layers.
To prune filters across multiple layers, we consider two strategies for layer-wise filter selection:
• Independent pruning determines which filters should be pruned at each layer independent of other layers.
• Greedy pruning accounts for the filters that have been removed in the previous layers. This strategy does not consider the kernels for the previously pruned feature maps while calculating the sum of absolute weights.
Figure 3 illustrates the difference between two approaches in calculating the sum of absolute weights. The greedy approach, though not globally optimal, is holistic and results in pruned networks with higher accuracy especially when many filters are pruned.
For simpler CNNs like VGGNet or AlexNet, we can easily prune any of the filters in any convolutional layer. However, for complex network architectures such as Residual networks (He et al. (2016)), pruning filters may not be straightforward. The architecture of ResNet imposes restrictions and the filters need to be pruned carefully. We show the filter pruning for residual blocks with projection mapping in Figure 4. Here, the filters of the first layer in the residual block can be arbitrarily pruned, as it does not change the number of output feature maps of the block. However, the correspondence between the output feature maps of the second convolutional layer and the identity feature maps makes it difficult to prune. Hence, to prune the second convolutional layer of the residual block, the corresponding projected feature maps must also be pruned. Since the identical feature maps are more important than the added residual maps, the feature maps to be pruned should be determined by the pruning results of the shortcut layer. To determine which identity feature maps are to be pruned, we use the same selection criterion based on the filters of the shortcut convolutional layers (with 1× 1 kernels). The second layer of the residual block is pruned with the same filter index as selected by the pruning of the shortcut layer.
3.4 RETRAINING PRUNED NETWORKS TO REGAIN ACCURACY
After pruning the filters, the performance degradation should be compensated by retraining the network. There are two strategies to prune the filters across multiple layers:
1. Prune once and retrain: Prune filters of multiple layers at once and retrain them until the original accuracy is restored. 2. Prune and retrain iteratively: Prune filters layer by layer or filter by filter and then retrain iteratively. The model is retrained before pruning the next layer for the weights to adapt to the changes from the pruning process.
We find that for the layers that are resilient to pruning, the prune and retrain once strategy can be used to prune away significant portions of the network and any loss in accuracy can be regained by retraining for a short period of time (less than the original training time). However, when some filters from the sensitive layers are pruned away or large portions of the networks are pruned away, it may not be possible to recover the original accuracy. Iterative pruning and retraining may yield better results, but the iterative process requires many more epochs especially for very deep networks.
4 EXPERIMENTS
We prune two types of networks: simple CNNs (VGG-16 on CIFAR-10) and Residual networks (ResNet-56/110 on CIFAR-10 and ResNet-34 on ImageNet). Unlike AlexNet or VGG (on ImageNet) that are often used to demonstrate model compression, both VGG (on CIFAR-10) and Residual networks have fewer parameters in the fully connected layers. Hence, pruning a large percentage of parameters from these networks is challenging. We implement our filter pruning method in Torch7 (Collobert et al. (2011)). When filters are pruned, a new model with fewer filters is created and the remaining parameters of the modified layers as well as the unaffected layers are copied into the new model. Furthermore, if a convolutional layer is pruned, the weights of the subsequent batch normalization layer are also removed. To get the baseline accuracies for each network, we train each model from scratch and follow the same pre-processing and hyper-parameters as ResNet (He et al. (2016)). For retraining, we use a constant learning rate 0.001 and retrain 40 epochs for CIFAR-10 and 20 epochs for ImageNet, which represents one-fourth of the original training epochs. Past work has reported up to 3× original training times to retrain pruned networks (Han et al. (2015)).
4.1 VGG-16 ON CIFAR-10
VGG-16 is a high-capacity network originally designed for the ImageNet dataset (Simonyan & Zisserman (2015)). Recently, Zagoruyko (2015) applies a slightly modified version of the model on CIFAR-10 and achieves state of the art results. As shown in Table 2, VGG-16 on CIFAR-10 consists of 13 convolutional layers and 2 fully connected layers, in which the fully connected layers do not occupy large portions of parameters due to the small input size and less hidden units. We use the model described in Zagoruyko (2015) but add Batch Normalization (Ioffe & Szegedy (2015))
layer after each convolutional layer and the first linear layer, without using Dropout (Srivastava et al. (2014)). Note that when the last convolutional layer is pruned, the input to the linear layer is changed and the connections are also removed.
As shown in Figure 2(b), each of the convolutional layers with 512 feature maps can drop at least 60% of filters without affecting the accuracy. Figure 2(c) shows that with retraining, almost 90% of the filters of these layers can be safely removed. One possible explanation is that these filters operate on 4× 4 or 2× 2 feature maps, which may have no meaningful spatial connections in such small dimensions. For instance, ResNets for CIFAR-10 do not perform any convolutions for feature maps below 8× 8 dimensions. Unlike previous work (Zeiler & Fergus (2014); Han et al. (2015)), we observe that the first layer is robust to pruning as compared to the next few layers. This is possible for a simple dataset like CIFAR-10, on which the model does not learn as much useful filters as on ImageNet (as shown in Figure. 5). Even when 80% of the filters from the first layer are pruned, the number of remaining filters (12) is still larger than the number of raw input channels. However, when removing 80% filters from the second layer, the layer corresponds to a 64 to 12 mapping, which may lose significant information from previous layers, thereby hurting the accuracy. With 50% of the filters being pruned in layer 1 and from 8 to 13, we achieve 34% FLOP reduction for the same accuracy.
4.2 RESNET-56/110 ON CIFAR-10
ResNets for CIFAR-10 have three stages of residual blocks for feature maps with sizes of 32× 32, 16× 16 and 8× 8. Each stage has the same number of residual blocks. When the number of feature maps increases, the shortcut layer provides an identity mapping with an additional zero padding for the increased dimensions. Since there is no projection mapping for choosing the identity feature maps, we only consider pruning the first layer of the residual block. As shown in Figure 6, most of the layers are robust to pruning. For ResNet-110, pruning some single layers without retraining even
improves the performance. In addition, we find that layers that are sensitive to pruning (layers 20, 38 and 54 for ResNet-56, layer 36, 38 and 74 for ResNet-110) lie at the residual blocks close to the layers where the number of feature maps changes, e.g., the first and the last residual blocks for each stage. We believe this happens because the precise residual errors are necessary for the newly added empty feature maps.
The retraining performance can be improved by skipping these sensitive layers. As shown in Table 1, ResNet-56-pruned-A improves the performance by pruning 10% filters while skipping the sensitive layers 16, 20, 38 and 54. In addition, we find that deeper layers are more sensitive to pruning than layers in the earlier stages of the network. Hence, we use a different pruning rate for each stage. We use pi to denote the pruning rate for layers in the ith stage. ResNet-56-pruned-B skips more layers (16, 18, 20, 34, 38, 54) and prunes layers with p1=60%, p2=30% and p3=10%. For ResNet-110, the first pruned model gets a slightly better result with p1=50% and layer 36 skipped. ResNet-110-pruned-B skips layers 36, 38, 74 and prunes with p1=50%, p2=40% and p3=30%. When there are more than two residual blocks at each stage, the middle residual blocks may be redundant and can be easily pruned. This might explain why ResNet-110 is easier to prune than ResNet-56.
4.3 RESNET-34 ON ILSVRC2012
ResNets for ImageNet have four stages of residual blocks for feature maps with sizes of 56 × 56, 28 × 28, 14 × 14 and 7 × 7. ResNet-34 uses the projection shortcut when the feature maps are down-sampled. We first prune the first layer of each residual block. Figure 7 shows the sensitivity of the first layer of each residual block. Similar to ResNet-56/110, the first and the last residual blocks of each stage are more sensitive to pruning than the intermediate blocks (i.e., layers 2, 8, 14, 16, 26, 28, 30, 32). We skip those layers and prune the remaining layers at each stage equally. In Table 1 we compare two configurations of pruning percentages for the first three stages: (A) p1=30%, p2=30%, p3=30%; (B) p1=50%, p2=60%, p3=40%. Option-B provides 24% FLOP reduction with about 1% loss in accuracy. As seen in the pruning results for ResNet-50/110, we can predict that ResNet-34 is relatively more difficult to prune as compared to deeper ResNets.
We also prune the identity shortcuts and the second convolutional layer of the residual blocks. As these layers have the same number of filters, they are pruned equally. As shown in Figure 7(b), these layers are more sensitive to pruning than the first layers. With retraining, ResNet-34-pruned-C prunes the third stage with p3=20% and results in 7.5% FLOP reduction with 0.75% loss in accuracy. Therefore, pruning the first layer of the residual block is more effective at reducing the overall FLOP
than pruning the second layer. This finding also correlates with the bottleneck block design for deeper ResNets, which first reduces the dimension of input feature maps for the residual layer and then increases the dimension to match the identity mapping.
4.4 COMPARISON WITH PRUNING RANDOM FILTERS AND LARGEST FILTERS
We compare our approach with pruning random filters and largest filters. As shown in Figure 8, pruning the smallest filters outperforms pruning random filters for most of the layers at different pruning ratios. For example, smallest filter pruning has better accuracy than random filter pruning for all layers with the pruning ratio of 90%. The accuracy of pruning filters with the largest `1-norms drops quickly as the pruning ratio increases, which indicates the importance of filters with larger `1-norms.
4.5 COMPARISON WITH ACTIVATION-BASED FEATURE MAP PRUNING
The activation-based feature map pruning method removes the feature maps with weak activation patterns and their corresponding filters and kernels (Polyak & Wolf (2015)), which needs sample data as input to determine which feature maps to prune. A feature map xi+1,j ∈ Rwi+1×hi+1 is generated by applying filter Fi,j ∈ Rni×k×k to feature maps of previous layer xi ∈ Rni×wi×hi , i.e., xi+1,j = Fi,j ∗ xi. Given N randomly selected images {xn1}Nn=1 from the training set, the statistics of each feature map can be estimated with one epoch forward pass of the N sampled data. Note that we calculate statistics on the feature maps generated from the convolution operations before batch normalization or non-linear activation. We compare our `1-norm based filter pruning with feature map pruning using the following criteria: σmean-mean(xi,j) = 1N ∑N n=1 mean(x n i,j), σmean-std(xi,j) =
1 N ∑N n=1 std(x n i,j), σmean-`1(xi,j) = 1 N ∑N n=1 ‖xni,j‖1, σmean-`2(xi,j) = 1 N ∑N n=1 ‖xni,j‖2 and
σvar-`2(xi,j) = var({‖xni,j‖2}Nn=1), where mean, std and var are standard statistics (average, standard deviation and variance) of the input. Here, σvar-`2 is the contribution variance of channel criterion proposed in Polyak & Wolf (2015), which is motivated by the intuition that an unimportant feature map has almost similar outputs for the whole training data and acts like an additional bias.
The estimation of the criteria becomes more accurate when more sample data is used. Here we use the whole training set (N = 50, 000 for CIFAR-10) to compute the statistics. The performance of feature map pruning with above criteria for each layer is shown in Figure 9. Smallest filter pruning outperforms feature map pruning with the criteria σmean-mean, σmean-`1 , σmean-`2 and σvar-`2 . The σmean-std criterion has better or similar performance to `1-norm up to pruning ratio of 60%. However, its performance drops quickly after that especially for layers of conv 1, conv 2 and conv 3. We find `1-norm is a good heuristic for filter selection considering that it is data free.
5 CONCLUSIONS
Modern CNNs often have high capacity with large training and inference costs. In this paper we present a method to prune filters with relatively low weight magnitudes to produce CNNs with reduced computation costs without introducing irregular sparsity. It achieves about 30% reduction in FLOP for VGGNet (on CIFAR-10) and deep ResNets without significant loss in the original accuracy. Instead of pruning with specific layer-wise hayperparameters and time-consuming iterative retraining, we use the one-shot pruning and retraining strategy for simplicity and ease of implementation. By performing lesion studies on very deep CNNs, we identify layers that are robust or sensitive to pruning, which can be useful for further understanding and improving the architectures.
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for their valuable feedback.
6 APPENDIX
6.1 COMPARISON WITH `2-NORM BASED FILTER PRUNING
We compare `1-norm with `2-norm for filter pruning. As shown in Figure 10, `1-norm works slightly better than `2-norm for layer conv 2. There is no significant difference between the two norms for other layers.
6.2 FLOP AND WALL-CLOCK TIME
FLOP is a commonly used measure to compare the computation complexities of CNNs. It is easy to compute and can be done statically, which is independent of the underlying hardware and software implementations. Since we physically prune the filters by creating a smaller model and then copy the weights, there are no masks or sparsity introduced to the original dense BLAS operations. Therefore the FLOP and wall-clock time of the pruned model is the same as creating a model with smaller number of filters from scratch.
We report the inference time of the original model and the pruned model on the test set of CIFAR-10 and the validation set of ILSVRC 2012, which contains 10,000 32× 32 images and 50,000 224× 224 images respectively. The ILSVRC 2012 dataset is used only for ResNet-34. The evaluation is conducted in Torch7 with Titan X (Pascal) GPU and cuDNN v5.1, using a mini-batch size 128. As shown in Table 3, the saved inference time is close to the FLOP reduction. Note that the FLOP number only considers the operations in the Conv and FC layers, while some calculations such as Batch Normalization and other overheads are not accounted.

This paper proposes a very simple idea (prune low-weight filters from ConvNets) in order to reduce FLOPs and memory consumption. The proposed method is experimented on with VGG-16 and ResNets on CIFAR10 and ImageNet.
Pros:
- Creates *structured* sparsity, which automatically improves performance without changing the underlying convolution implementation
- Very simple to implement
Cons:
- No evaluation of how pruning impacts transfer learning
I'm generally positive about this work. While the main idea is almost trivial, I am not aware of any other papers that propose exactly the same idea and show a good set of experimental results. Therefore I'm inclined to accept it. The only major downside is that the paper does not evaluate the impact of filter pruning on transfer learning. For example, there is not much interest in the tasks of CIFAR10 or even ImageNet. Instead, the main interest in both academia and industry is the value of the learned representation for transferring to other tasks. One might expect filter pruning (or any other kind of pruning) to harm transfer learning. It's possible that the while the main task has about the same performance, transfer learning is strongly hurt. This paper has missed an opportunity to explore that direction.
Nit: Fig 2 title says VGG-16 in (b) and VGG_BN in (c). Are these the same models?