This paper studies the cooperative training of two probabilistic models of signals such as images. Both models are parametrized by convolutional neural networks (ConvNets). The first network is a descriptor network, which is an exponential family model or an energy-based model, whose feature statistics or energy function are defined by a bottom-up ConvNet, which maps the observed signal to the feature statistics. The second network is a generator network, which is a nonlinear version of factor analysis. It is defined by a top-down ConvNet, which maps the latent factors to the observed signal. The maximum likelihood training algorithms of both the descriptor net and the generator net are in the form of alternating back-propagation, and both algorithms involve Langevin sampling. We observe that the two training algorithms can cooperate with each other by jumpstarting each other’s Langevin sampling, and they can be seamlessly interwoven into a CoopNets algorithm that can train both nets simultaneously.
1 INTRODUCTION
1.1 TWO CONVNETS OF OPPOSITE DIRECTIONS
We begin with a story that the reader of this paper can readily relate to. A student writes up an initial draft of a paper. His advisor then revises it. After that they submit the revised paper for review. The student then learns from his advisor’s revision, while the advisor learns from the outside review. In this story, the advisor guides the student, but the student does most of the work.
This paper is about two probabilistic models of signals such as images, and they play the roles of student and advisor as described above. Both models are parametrized by convolutional neural networks (ConvNets or CNNs) (LeCun et al., 1998; Krizhevsky et al., 2012). The two nets take two opposite directions. One is bottom-up, and the other is top-down, as illustrate by the following diagram:
Bottom-up ConvNet Top-down ConvNet features latent variables ⇑ ⇓
signal signal (a) Descriptor Net (b) Generator Net
(1)
The simultaneous training of such two nets was first studied by the recent work of Kim & Bengio (2016). These two nets belong to two major classes of probabilistic models. (a) The exponential family models or the energy-based models (LeCun et al., 2006) or the Markov random field models (Zhu et al., 1997), where the probability distribution is defined by feature statistics or energy function computed from the signal by a bottom-up process. (b) The latent variable models or the directed graphical models, where the signal is assumed to be a transformation of the latent factors that follow a known prior distribution. The latent factors generate the signal by a top-down process. A classical example is factor analysis.
The two classes of models have been contrasted by Zhu (2003); Teh et al. (2003); Ngiam et al. (2011). Zhu (2003) called the two classes of models the descriptive models and the generative
models respectively. Both classes of models can benefit from the high capacity of the multi-layer ConvNets. (a) In the exponential family models or the energy-based models, the feature statistics or the energy function can be defined by a bottom-up ConvNet that maps the signal to the features and the energy function (Ngiam et al., 2011; Xie et al., 2016). We call the resulting model a descriptive network or a descriptor net following Zhu (2003), because it is built on descriptive feature statistics. (b) In the latent variable models or the directed graphical models, the transformation from the latent factors to the signal can be defined by a top-down ConvNet (Dosovitskiy et al., 2015), which maps the latent factors to the signal. We call the resulting model a generative network or generator net following Goodfellow et al. (2014), who proposed such a model in their work on the generative adversarial networks (GAN).
1.2 TWO TRAINING ALGORITHMS AND THEIR COOPERATION
Fig. 1(a) and (b) display the flowcharts of the maximum likelihood learning algorithms for training the descriptor and generator nets . We call the two algorithms Algorithm D and Algorithm G respectively. Algorithm D (Xie et al., 2016) iterates two steps: Step D1 synthesizes examples by sampling from the current model by Langevin dynamics. Step D2 updates the parameters to shift the density from the synthesized examples towards the observed examples. Algorithm G (Han et al., 2017) also iterates two steps. Step G1 infers latent factors for each observed example by sampling from their posterior distribution by Langevin dynamics. Step G2 updates the parameters by a non-linear regression of the observed examples on their corresponding latent factors. We use Langevin dynamics for Markov chain Monte Carlo (MCMC) sampling because the gradient term of Langevin dynamics can be readily computed via back-propagation. Thus all the steps D1, D2 and G1, G2 are powered by back-propagation, and both Algorithms D and G are alternating back-propagation algorithms.
In this article, we propose to couple Algorithms D and G into a cooperative training algorithm that interweaves the steps of the two algorithms seamlessly. We call the resulting algorithm the CoopNets algorithm, and we show that it can train both nets simultaneously.
Figure 1(c) displays the flowchart of the CoopNets algorithm. The generator is like the student. It generates the initial draft of the synthesized examples. The descriptor is like the advisor. It revises the initial draft by initializing its Langevin dynamics from the initial draft in Step D1, which produces the revised draft of the synthesized examples. The descriptor learns from the outside review in Step D2, which is in the form of the difference between the observed examples and the revised
synthesized examples. The generator learns from how the descriptor revises the initial draft by reconstructing the revised draft in Step G2. For each synthesized example, the generator knows the latent factors that generate the initial draft, so that Step G1 can infer the latent factors by initializing its Langevin dynamics from their known values.
In the CoopNets algorithm, the generator fuels the MCMC of the descriptor by supplying initial synthesized examples, which can be obtained by direct ancestral sampling. The generator then learns from the revised synthesized examples with virtually known latent factors. The cooperation is thus beneficial to both nets.
2 RELATED WORK
Our work is inspired by the generative adversarial networks (GAN) (Goodfellow et al., 2014; Denton et al., 2015; Radford et al., 2015). In GAN, the generator net is paired with a discriminator net. The two nets play adversarial roles. In our work, the generator net and the descriptor net play cooperative roles, and they feed each other the initial, revised and reconstructed synthesized data. The learning of both nets is based on maximum likelihood, and the learning process is quite stable because of the cooperative nature and the consistent directions of the two maximum likelihood training algorithms.
Another method to train the generator network is variational auto-encoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014; Mnih & Gregor, 2014), which learns an inferential or recognition network to approximate the posterior distribution of the latent factors.
The connection between the descriptor net and the discriminator net has been explored by Xie et al. (2016), where the descriptor can be derived from the discriminator.
Our work is most similar to the recent work of Kim & Bengio (2016). In fact, the settings of the two nets are the same. In their work, the generator learns from the descriptor by minimizing the Kullback-Leibler divergence from the generator to the descriptor, which can be decomposed into an energy term and an entropy term. In our work, the two nets interact with each other via synthesized data, and the generator learns from the descriptor by reconstructing the revised draft of synthesized examples. Our method does not need to approximate the intractable entropy term.
Our work is related to the contrastive divergence algorithm (Hinton, 2002) for training the descriptor net. The contrastive divergence initializes the MCMC sampling from the observed examples. The CoopNets algorithm initializes the MCMC sampling from the examples supplied by the generator.
3 TWO NETS AND TWO TRAINING ALGORITHMS
3.1 DESCRIPTOR NET AND TRAINING ALGORITHM
Let Y be the D-dimensional signal, such as an image. The descriptor model is in the form of exponential tilting of a reference distribution (Xie et al., 2016):
PD(Y ;WD) = 1
Z(WD) exp [f(Y ;WD)] q(Y ), (2)
where q(Y ) is the reference distribution such as Gaussian white noise q(Y ) ∝ exp ( −‖Y ‖2/2s2 ) ,
f(Y ;WD) (f stands for features) is the feature statistics or energy function, defined by a ConvNet whose parameters are denoted by WD. This ConvNet is bottom-up because it maps the signal Y to a number. See the diagram in (1). Z(WD) = ∫ exp [f(Y ;WD)] q(Y )dY = Eq{exp[f(Y ;WD)]} is the normalizing constant, where Eq is the expectation with respect to q.
Suppose we observe training examples {Yi, i = 1, ..., n} from an unknown data distribution Pdata(Y ). The maximum likelihood training seeks to maximize the log-likelihood function LD(WD) = 1 n ∑n i=1 logPD(Yi;WD). If the sample size n is large, the maximum likelihood estimator minimizes KL(Pdata|PD), the Kullback-Leibler divergence from the data distribution Pdata to the model distribution PD. The gradient of the LD(WD) is
L′D(WD) = 1
n n∑ i=1 ∂ ∂WD f(Yi;WD)− EWD [ ∂ ∂WD f(Y ;WD) ] , (3)
where EWD denotes the expectation with respect to PD(Y ;WD).
The expectation in equation (3) is analytically intractable and has to be approximated by MCMC, such as Langevin dynamics, which iterates the following step:
Yτ+1 = Yτ − δ2
2 [ Yτ s2 − ∂ ∂Y f(Yτ ;WD) ] + δUτ , (4)
where τ indexes the time steps of the Langevin dynamics, δ is the step size, and Uτ ∼ N(0, ID) is the Gaussian white noise term.
We can run ñ parallel chains of Langevin dynamics according to (4) to obtain the synthesized examples {Ỹi, i = 1, ..., ñ}. The Monte Carlo approximation to L′D(WD) is
L′D(WD) ≈ 1
n n∑ i=1 ∂ ∂WD f(Yi;WD)− 1 ñ ñ∑ i=1 ∂ ∂WD f(Ỹi;WD), (5)
which is used to update WD.
Algorithm D (Xie et al., 2016) iterates the following two steps after initializing WD and {Ỹi, i = 1, ..., ñ}. Step D1: Run lD steps of Langevin from the current {Ỹ } according according to (4). Step D2: update W (t+1)D = W (t) D + γtL ′ D(W (t) D ) with learning rate γt. The convergence of such an algorithm follows Younes (1999).
3.2 GENERATOR NET AND TRAINING ALGORITHM
The generator net (Goodfellow et al., 2014) seeks to explain the signal Y of dimensionD by a vector of latent factors X of dimension d, and usually d D. The model is of the following form:
X ∼ N(0, Id), Y = g(X;WG) + , ∼ N(0, σ2ID). (6) g(X;WG) (g stands for generator) is a top-down ConvNet defined by the parameters WG . The ConvNet g maps the latent factors X to the signal Y . See the diagram in (1).
The joint density of model (6) is PG(X,Y ;WG) = PG(X)PG(Y |X;WG), and
logPG(X,Y ;WG) = − 1
2σ2 ‖Y − g(X;WG)‖2 −
1 2 ‖X‖2 + constant, (7)
where the constant term is independent of X , Y and WG . The marginal density is obtained by integrating out the latent factors X , i.e., PG(Y ;WG) = ∫ PG(X,Y ;WG)dX . The inference of X given Y is based on the posterior density PG(X|Y ;WG) = PG(X,Y ;WG)/PG(Y ;WG) ∝ PG(X,Y ;WG) as a function of X .
For the training data {Yi, i = 1, ..., n}, the generator net can be trained by maximizing the loglikelihood LG(WG) = 1n ∑n i=1 logPG(Yi;WG). For large sample, the learned WG minimizes the Kullback-Leibler divergence KL(Pdata|PG) from the data distribution Pdata to the model distribution PG . The gradient of LG(WG) is obtained according to the following identity
∂
∂WG logPG(Y ;WG) =
1
PG(Y ;WG)
∂
∂WG
∫ PG(Y,X;WG)dX
= 1
PG(Y ;WG)
∫ [ ∂
∂WG logPG(Y,X;WG)
] PG(Y,X;WG)dX
=
∫ [ ∂
∂WG logPG(Y,X;WG)
] PG(Y,X;WG)
PG(Y ;WG) dX
= EPG(X|Y ;WG)
[ ∂
∂WG logPG(X,Y ;WG)
] , (8)
which underlies the EM algorithm. In general, the expectation in (8) is analytically intractable, and has to be approximated by MCMC that samples from the posterior PG(X|Y ;WG), such as Langevin dynamics, which iterates
Xτ+1 = Xτ + δ2
2
∂
∂X logPG(Xτ , Y ;WG) + δUτ , (9)
where Uτ ∼ N(0, Id). With Xi sampled from PG(Xi | Yi,WG) for each observation Yi, the Monte Carlo approximation to L′G(WG) is
L′G(WG) ≈ 1
n n∑ i=1 ∂ ∂WG logPG(Xi, Yi;WG) = 1 n n∑ i=1 1 σ2 (Yi − g(Xi;WG)) ∂ ∂WG g(Xi;WG). (10)
Algorithm G (Han et al., 2017) iterates the following two steps after initializing WG and {Xi, i = 1, ..., n}. Step G1: run lG steps of Langevin from the current {Xi} according to (9). Step G2: update W
(t+1) G =W (t) G +γtL ′ G(W (t) G ) with learning rate γt. The convergence of such an algorithm follows
Younes (1999).
4 COOPNETS ALGORITHM: RECONSTRUCTING THE REVISION
In Algorithms D and G, both steps D1 and G1 are Langevin dynamics, which may be slow to converge. An interesting observation is that the two algorithms can cooperate with each other by jumpstarting each other’s Langevin sampling.
Specifically, in Step D1, we can initialize the synthesized examples by generating examples from the generator net. We first generate X̂i ∼ N(0, Id), and then generate Ŷi = g(X̂i;WG) + i, for i = 1, ..., ñ. If the current generator PG is close to the current descriptor PD, then the generated {Ŷi} should be a good initialization for sampling from the descriptor net, i.e., starting from the {Ŷi, i = 1, ..., ñ}, we run Langevin dynamics in Step D1 for lD steps to get {Ỹi, i = 1, ..., ñ}, which are revised versions of {Ŷi}. These {Ỹi} can be used as the synthesized examples from the descriptor net. We can then update WD according to Step D2 of Algorithm D.
In order to update WG of the generator net, we treat the {Ỹi, i = 1, ..., ñ} produced by the above Step D1 as the training data for the generator. Since these {Ỹi} are obtained by the Langevin dynamics initialized from the {Ŷi, i = 1, ..., ñ} produced by the generator net with known latent factors {X̂i, i = 1, ..., ñ}, we can update WG by learning from {(Ỹi, X̂i), i = 1, ..., ñ}, which is a supervised learning problem, or more specifically, a non-linear regression of Ỹi on X̂i. At W (t) G , the latent factors X̂i generates and thus reconstructs the initial example Ŷi. After updating WG , we want X̂i to reconstruct the revised example Ỹi. That is, we revise WG to absorb the revision from Ŷi to Ỹi, so that the generator shifts its density from {Ŷi} to {Ỹi}. The reconstruction error can tell us whether the generator has caught up with the descriptor by fully absorbing the revision.
The left diagram in (11) illustrates the basic idea.
X̂i
Ŷi Ỹi
W (t) G W (t+1) G
W (t) D
X̂i Xi
Ŷi Ỹi
W (t) G
W (t) G W (t+1) G
W (t) D (11)
In the two diagrams in (11), the double-line arrows indicate generation and reconstruction by the generator net, while the dashed-line arrows indicate Langevin dynamics for revision and inference in the two nets. The diagram on the right in (11) illustrates a more rigorous method, where we initialize the Langevin inference of {Xi, i = 1, ..., ñ} in Step G1 from {X̂i}, and then update WG in Step G2 based on {(Ỹi, Xi), i = 1, ..., ñ}. The diagram on the right shows how the two nets jumpstart each other’s Langevin dynamics.
Algorithm 1 describes the cooperative training that interweaves Algorithm D and Algorithm G. See Figure 1(c) for the flowchart of the CoopNets algorithm. In our experiments, we set lG = 0 and infer Xi = X̂i for simplicity, i.e., we follow the left diagram in (11).
See Appendix for a theoretical understanding of the convergence of the CoopNets algorithm.
Algorithm 1 CoopNets Algorithm Input:
(1) training examples {Yi, i = 1, ..., n} (2) numbers of Langevin steps lD ad lG (3) number of learning iterations T
Output: (1) estimated parameters WD and WG (2) synthesized examples {Ŷi, Ỹi, i = 1, ..., ñ}
1: Let t← 0, initialize WD and WG . 2: repeat 3: Step G0: For i = 1, ..., ñ, generate X̂i ∼ N(0, Id), and generate Ŷi = g(X̂i;W (t)G ) + i. 4: Step D1: For i = 1, ..., ñ, starting from Ŷi, Run lD steps of Langevin dynamics to obtain Ỹi, each step following equation (4). 5: Step G1: Treat the current {Ỹi, i = 1, ..., ñ} as the training data, for each i, infer Xi = X̂i.
Or more rigorously, starting from Xi = X̂i, run lG steps of Langevin dynamics to update Xi, each step following equation (9).
6: Step D2: Update W (t+1)D = W (t) D + γtL ′ D(W (t) D ), where L ′ D(W (t) D ) is computed according to (5). 7: Step G2: Update W (t+1)G = W (t) G + γtLG ′(W (t) G ), where LG
′(WG) is computed according to (10), except that Yi is replaced by Ỹi, and n by ñ.
8: Let t← t+ 1 9: until t = T
5 EXPERIMENTS
We use the MatConvNet of Vedaldi & Lenc (2015) for coding. For the descriptor net, we adopt the structure of Xie et al. (2016), where the bottom-up network consists of multiple layers of convolution by linear filtering, ReLU non-linearity, and down-sampling. We adopt the structure of the generator network of Radford et al. (2015); Dosovitskiy et al. (2015), where the top-down network consists of multiple layers of deconvolution by linear superposition, ReLU non-linearity, and up-sampling, with tanh non-linearity at the bottom-layer (Radford et al., 2015) to make the signals fall within [−1, 1].
5.1 QUANTITATIVE EXPERIMENT ON FACE COMPLETION
We conduct an experiment on learning from complete training images of human faces, and then testing the learned model on completing the occluded testing images. The structure of the generator network is the same as in (Radford et al., 2015; Dosovitskiy et al., 2015). We adopt a 4-layer descriptor net. The first layer has 96 5 × 5 filters with sub-sampling of 2, the second layers has 128 5 × 5 filters with sub-sampling of 2, the third layer has 256 5 × 5 filters with sub-sampling of 2, and the final layer is a fully connected layer with 50 channels as output. We use L=10 steps of Langevin revision dynamics within each learning iteration, and the Langevin step size is set at 0.002. The learning rate is 0.07. The training data are 10, 000 human faces randomly selected from CelebA dataset (Liu et al., 2015). We run 600 cooperative learning iterations. Figure 2 displays 144 synthesized human faces by the descriptor net.
To quantitatively test whether we have learned a good generator net g(X;WG) even though it has never seen the training images directly in the training stage, we apply it to the task of recovering the occluded pixels of testing images. For each occluded testing image Y , we use Step G1 of Algorithm G to infer the latent factors X . The only change is with respect to the term ‖Y − g(X;WG)‖2, where the sum of squares is over all the observed pixels of Y in back-propagation computation. We run 1000 Langevin steps, initializing X from N(0, Id). After inferring X , the completed image g(X;WG) is automatically obtained. We design 3 experiments, where we randomly place a 20×20, 30 × 30, or 40 × 40 mask on each 64 × 64 testing image. These 3 experiments are denoted by M20 M30, and M40 respectively (M for mask). We report the recovery errors and compare our method with 8 different image inpainting methods as well as the DCGAN of Radford et al. (2015).
For DCGAN, we use the parameter setting in Radford et al. (2015) except changing the number of learning iterations to 600. We use the same 10, 000 training images to learn DCGAN. After the model is learned, we keep the generator and use the same method as ours to infer latent factors X , and recover the unobserved pixels. In 8 inpainting methods, Methods 1 and 2 are based on Markov random field prior where the nearest neighbor potential terms are `2 and `1 differences respectively. Methods 3 to 8 are interpolation methods. Please refer to D’Errico (2004) for more details. Table 1 displays the recovery errors of the 3 experiments, where the error is measured by per pixel difference (relative to the range of pixel values) between the original image and the recovered image on the occluded region, averaged over 100 testing images. Fig. 3 displays some recovery results by our method. The first row shows the original images as the ground truth. The second row displays the testing images with occluded pixels. The third row displays the recovered images by the generator net trained by the CoopNets algorithm on the 10,000 training images.
5.2 QUALITATIVE EXPERIMENT ON SYNTHESIS
We conduct an experiment on synthesizing images of categories from Imagenet ILSVRC2012 dataset (Deng et al., 2009) and MIT places205 dataset (Zhou et al., 2014). We adopt a 4-layer descriptor net. The first layer has 64 5 × 5 filters with sub-sampling of 2, the second layers has 128 3 × 3 filters with sub-sampling of 2, the third layer has 256 3 × 3 filters with sub-sampling of 1, and the final layer is a fully connected layer with 100 channels as output. We set the number of Langevin dynamics steps in each learning iteration to 10 and the step size to 0.002. The learning rate is 0.07. For each category, we randomly choose 1,000 images as training data and resize the images to 64× 64. We run 1, 000 cooperative learning iterations to train the model. Figures 4 and 5 display the results for two categories, where for each category, we show 144 original images sampled from the training set, and 144 synthesized images generated by our method. The appendix contains more synthesis results.
As a comparison, we apply the Algorithm G alone and GAN code on the same 1,000 hotel room training images to learn the generator of the same structure as in CoopNets. Figure 6 displays the synthesis results.
We also try to synthesize images at high resolution (224× 224). We adopt a 4-layer descriptor net. The first layer has 128 15× 15 filters with sub-sampling of 3, the second layer has 256 3× 3 filters with sub-sampling of 2, the third layer has 512 3 × 3 filters with sub-sampling of 1, and the final layer is a fully connected layer with 100 channels as output. We enlarge the filters of the final layer of generator net to 14 × 14 to generate 224 × 224 images. The learning rate is 0.05. We run 1000 cooperative learning iterations to train the model. Figures 7 and 8 show the synthesized images of two categories from MIT places205 dataset.
6 CONCLUSION
The most unique feature of our work is that the two networks feed each other the synthesized data in the learning process, including initial, revised, and reconstructed synthesized data.
Another unique feature of our work is that the learning process interweaves the existing maximum likelihood learning algorithms for the two networks.
A third unique feature of our work is that the MCMC for the descriptor keeps rejuvenating the chains by refreshing the samples by independent replacements supplied by the generator, so that a single chain effectively amounts to an infinite number of chains or the evolution of the whole marginal distribution modeled by the generator.
CODE AND DATA
http://www.stat.ucla.edu/˜ywu/CoopNets/main.html
7 APPENDIX: CONVERGENCE
7.1 GENERATOR OF INFINITE CAPACITY
In the CoopNets algorithm, the descriptor learns from the observed examples, while the generator learns from the descriptor through the synthesized examples. Therefore, the descriptor is the driving force in terms of learning, although the generator is the driving force in terms of synthesis. In order to understand the convergence of learning, we can start from Algorithm D for learning the descriptor.
Algorithm D is a stochastic approximation algorithm (Robbins & Monro, 1951), except that the samples are generated by finite step MCMC transitions. According to Younes (1999), Algorithm D converges to the maximum likelihood estimate under suitable regularity conditions on the mixing of the transition kernel of the MCMC and the schedule of the learning rate γt, even if the number of Langevin steps lD is finite or small (e.g., lD = 1), and even if the number of parallel chains ñ is finite or small (e.g., ñ = 1). The reason is that the random fluctuations caused by the finite number of chains, ñ, and the limited mixing caused by the finite steps of MCMC, lD, are mitigated if the
learning rate γt is sufficiently small. At learning iteration t, let W (t) D be the estimated parameter of the descriptor. Let P (t+1)D be the marginal distribution of {Ỹi}. Even though P (t+1) D 6= PD(Y ;W (t) D ) because lD is finite (P (t+1) D = PD(Y ;W (t) D ) if lD → ∞), we still have W (t) D → ŴD in probability according to Younes (1999), where ŴD is the maximum likelihood estimate of WD.
The efficiency of Algorithm D increases if the number of parallel chains ñ is large because it leads to more accurate estimation of the expectation in the gradient L′D(WD) of equation (3), so that we can afford to use larger learning rate γt for faster convergence.
Now let us come back to the CoopNets algorithm. In order to understand how the descriptor net helps the training of the generator net, let us consider the idealized scenario where the number of parallel chains ñ → ∞, and the generator has infinite capacity, and in each iteration it estimates WG by maximum likelihood using the synthesized data from P (t+1)D . In this idealized scenario, the learned generator PG(Y ;W (t+1) G ) will reproduce P (t+1) D by minimizing KL(P (t+1) D (Y )|PG(Y ;WG)), with P (t+1) D serving as its data distribution. Then eventually the learned generator PG(Y, ŴG) will reproduce PD(Y ; ŴD). Thus the cooperative training helps the learning of the generator. Note that the learned generator PG(Y, ŴG) will not reproduce the distribution of the observed data Pdata, unless the descriptor is of infinite capacity too.
Conversely, the generator net also helps the learning of the descriptor net in the CoopNets algorithm. In Algorithm D, it is impractical to make the number of parallel chains ñ too large. On the other hand, it would be difficult for a small number of chains {Ỹi, i = 1, ..., ñ} to explore the state space. In the CoopNets algorithm, because PG(Y ;W (t) G ) reproduces P (t) D , we can generate a completely new batch of independent samples {Ŷi} from PG(Y ;W (t)G ), and revise {Ŷi} to {Ỹi} by Langevin dynamics, instead of running Langevin dynamics from the same old batch of {Ỹi} as in the original Algorithm D. This is like implementing an infinite number of parallel chains, because each iteration evolves a fresh batch of examples, as if each iteration evolves a new set of chains. By updating the generator WG , it is like we are updating the infinite number of parallel chains, because WG memorizes the whole distribution. Even if ñ in the CoopNets algorithm is small, e.g., ñ = 1, viewed from the perspective of Algorithm D, it is as if ñ→∞. Thus the above idealization ñ→∞ is sound.
7.2 GENERATOR OF FINITE CAPACITY
From an information geometry point of view, let D = {PD(Y ;WD),∀WD} be the manifold of the descriptor models, where each distribution PD(Y ;WD) is a point on this manifold. Then the maximum likelihood estimate of WD is a projection of the data distribution Pdata onto the manifold D. Let G = {PG(Y ;WG),∀WG} be the manifold of the generator models, where each distribution PG(Y ;WG) is a point on this manifold. Then the maximum likelihood estimate ofWG is a projection of the data distribution Pdata onto the manifold G. From now on, for notational simplicity and with a slight abuse of notation, we use WD to denote the descriptor distribution PD(Y ;WD), and use WG to denote the generator distribution PG(Y ;WG).
We assume both the observed data size n and the synthesized data size ñ are large enough so that we shall work on distributions or populations instead of finite samples. As explained above, assuming ñ→∞ is sound because the generator net can supply unlimited number of examples.
The Langevin revision dynamics runs a Markov chain from W (t)G towards W (t) D . Let LWD be the Markov transition kernel of lD steps of Langevin revisions towards WD. The distribution of the revised synthesized data is
P (t+1) D = LW (t)D ·W (t)G , (12)
where the notation L ·P denotes the marginal distribution obtained by running the Markov transition L fromP . The distributionP (t+1)D is in the middle between the two netsW (t) G andW (t) D , and it serves as the data distribution to train the generator, i.e., we project this distribution onto the manifold G = {PG(Y ;WG),∀WG} = {WG} (recall we use WG to denote the distribution PG(Y ;WG)) in the
information geometry picture, so that
W (t+1) G = argminG KL(P (t+1) D |WG). (13)
The learning process alternates between Markov transition in (12) and projection in (13), as illustrated by Figure 9.
In the case of lD →∞,
W (t) D → ŴD = argminD KL(Pdata|WD), (14)
W (t) G → ŴG = argminG KL(ŴD|WG). (15)
That is, we first project Pdata onto D, and from there continue to project onto G. Therefore, WD converges to the maximum likelihood estimate with Pdata being the data distribution, while WG converges to the maximum likelihood estimate with ŴD serving as the data distribution.
For finite lD, the algorithm may converge to the following fixed points. The fixed point for the generator satisfies
ŴG = argmin G KL(LŴD · ŴG |WG). (16)
The fixed point for the descriptor satisfies
ŴD = argmin D [ KL(Pdata|WD)−KL(LŴD · ŴG |WD) ] , (17)
which is similar to contrastive divergence (Hinton, 2002), except that ŴG takes the place of Pdata in the second Kullback-Leibler divergence. Because ŴG is supposed to be close to ŴD, the second Kullback-Leibler divergence is supposed to be small, hence our algorithm is closer to maximum likelihood learning than contrastive divergence.
Kim & Bengio (2016) learned the generator by gradient descent on KL(WG |W (t)D ) over G. The objective function is KL(WG |W (t)D ) = EWG [logPG(Y ;WG)]− EWG [logPD(Y ;W (t) D )], where the first term is the negative entropy that is intractable, and the second term is the expected energy that is tractable. Our learning method for the generator is consistent with the learning objective KL(WG |W (t)D ), because
KL(P (t+1) D |W (t) D ) ≤ KL(W (t) G |W (t) D ). (18)
In fact, KL(P (t+1)D |W (t) D ) → 0 monotonically as lD → ∞ due to the second law of thermodynamics. The reduction of the Kullback-Leibler divergence in (18) and the projection in (13) in our learning of the generator are consistent with the learning objective of reducing KL(WG |W (t)D ) in Kim & Bengio (2016). But the Monte Carlo implementation of L in our work avoids the need to approximate the intractable entropy term.
7.3 MORE SYNTHESIS RESULTS
We display more synthesis results at the resolution of 64 × 64.
ACKNOWLEDGEMENT
We thank Hansheng Jiang for her work on this project as a summer visiting student. We thank Tian Han for sharing the code on learning the generator network, and for helpful discussions.
The work is supported by NSF DMS 1310391, DARPA SIMPLEX N66001-15-C-4035, ONR MURI N00014-16-1-2007, and DARPA ARO W911NF-16-1-0579.

This paper introduces CoopNets, an algorithm which trains a Deep-Energy Model (DEM, the “descriptor”) with the help of an auxiliary directed bayes net, e.g. “the generator”. The descriptor is trained via standard maximum likelihood, with Langevin MCMC for sampling. The generator is trained to generate likely samples under the DEM in a single, feed-forward ancestral sampling step. It can thus be used to shortcut expensive MCMC sampling, hence the reference to “cooperative training”.
The above idea is interesting and novel, but unfortunately is not sufficiently validated by the experimental results. First and foremost, two out of the three experiments do not feature a train /test split, and ignore standard training and evaluation protocols for texture generation (see [R1]). Datasets are also much too small. As such these experiments only seem to confirm the ability of the model to overfit. On the third in-painting tasks, baselines are almost non-existent: no VAEs, RBMs, DEM, etc which makes it difficult to evaluate the benefits of the proposed approach.
In a future revision, I would also encourage the authors to answer the following questions experimentally. What is the impact of the missing rejection step in Langevin MCMC (train with, without ?). What is the impact of the generator on the burn-in process of the Markov chain (show sample auto-correlation). How bad is approximation of training the generator from ({\tilde{Y}, \hat{X}) instead of ({\tilde{Y}, \tilde{X}) ? Run comparative experiments.
The paper would also greatly benefit from a rewrite focusing on clarity, instead of hyperbole (“pioneering work” in reference to closely related, but non-peer reviewed work) and prose (“tale of two nets”). For example, the authors fail to specify the exact form of the energy function: this seems like a glaring omission.
PROS:
+ Interesting and novel idea
CONS:
- Improper experimental protocols
- Missing baselines
- Missing diagnostic experiments
[R1] Heess, N., Williams, C. K. I., and Hinton, G. E. (2009). Learning generative texture models with extended fields of-experts.


We introduce the “Energy-based Generative Adversarial Network” model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions. Similar to the probabilistic GANs, a generator is seen as being trained to produce contrastive samples with minimal energies, while the discriminator is trained to assign high energies to these generated samples. Viewing the discriminator as an energy function allows to use a wide variety of architectures and loss functionals in addition to the usual binary classifier with logistic output. Among them, we show one instantiation of EBGAN framework as using an auto-encoder architecture, with the energy being the reconstruction error, in place of the discriminator. We show that this form of EBGAN exhibits more stable behavior than regular GANs during training. We also show that a single-scale architecture can be trained to generate high-resolution images.
We introduce the “Energy-based Generative Adversarial Network” model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions. Similar to the probabilistic GANs, a generator is seen as being trained to produce contrastive samples with minimal energies, while the discriminator is trained to assign high energies to these generated samples. Viewing the discriminator as an energy function allows to use a wide variety of architectures and loss functionals in addition to the usual binary classifier with logistic output. Among them, we show one instantiation of EBGAN framework as using an auto-encoder architecture, with the energy being the reconstruction error, in place of the discriminator. We show that this form of EBGAN exhibits more stable behavior than regular GANs during training. We also show that a single-scale architecture can be trained to generate high-resolution images.
1 INTRODUCTION
1.1 ENERGY-BASED MODEL
The essence of the energy-based model (LeCun et al., 2006) is to build a function that maps each point of an input space to a single scalar, which is called “energy”. The learning phase is a datadriven process that shapes the energy surface in such a way that the desired configurations get assigned low energies, while the incorrect ones are given high energies. Supervised learning falls into this framework: for each X in the training set, the energy of the pair (X,Y ) takes low values when Y is the correct label and higher values for incorrect Y ’s. Similarly, when modeling X alone within an unsupervised learning setting, lower energy is attributed to the data manifold. The term contrastive sample is often used to refer to a data point causing an energy pull-up, such as the incorrect Y ’s in supervised learning and points from low data density regions in unsupervised learning.
1.2 GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains. The basic idea of GAN is to simultaneously train a discriminator and a generator. The discriminator is trained to distinguish real samples of a dataset from fake samples produced by the generator. The generator uses input from an easy-to-sample random source, and is trained to produce fake samples that the discriminator cannot distinguish from real data samples. During training, the generator receives the gradient of the output of the discriminator with respect to the fake sample. In the original formulation of GAN in Goodfellow et al. (2014), the discriminator produces a probability and, under certain conditions, convergence occurs when the distribution produced by the generator matches the data distribution. From a game theory point of view, the convergence of a GAN is reached when the generator and the discriminator reach a Nash equilibrium.
1.3 ENERGY-BASED GENERATIVE ADVERSARIAL NETWORKS
In this work, we propose to view the discriminator as an energy function (or a contrast function) without explicit probabilistic interpretation. The energy function computed by the discriminator can be viewed as a trainable cost function for the generator. The discriminator is trained to assign low energy values to the regions of high data density, and higher energy values outside these regions. Conversely, the generator can be viewed as a trainable parameterized function that produces samples in regions of the space to which the discriminator assigns low energy. While it is often possible to convert energies into probabilities through a Gibbs distribution (LeCun et al., 2006), the absence of normalization in this energy-based form of GAN provides greater flexibility in the choice of architecture of the discriminator and the training procedure.
The probabilistic binary discriminator in the original formulation of GAN can be seen as one way among many to define the contrast function and loss functional, as described in LeCun et al. (2006) for the supervised and weakly supervised settings, and Ranzato et al. (2007) for unsupervised learning. We experimentally demonstrate this concept, in the setting where the discriminator is an autoencoder architecture, and the energy is the reconstruction error. More details of the interpretation of EBGAN are provided in the appendix B.
Our main contributions are summarized as follows:
• An energy-based formulation for generative adversarial training. • A proof that under a simple hinge loss, when the system reaches convergence, the generator
of EBGAN produces points that follow the underlying data distribution. • An EBGAN framework with the discriminator using an auto-encoder architecture in which
the energy is the reconstruction error. • A set of systematic experiments to explore hyper-parameters and architectural choices that
produce good result for both EBGANs and probabilistic GANs. • A demonstration that EBGAN framework can be used to generate reasonable-looking high-
resolution images from the ImageNet dataset at 256×256 pixel resolution, without a multiscale approach.
2 THE EBGAN MODEL
Let pdata be the underlying probability density of the distribution that produces the dataset. The generator G is trained to produce a sample G(z), for instance an image, from a random vector z, which is sampled from a known distribution pz , for instance N (0, 1). The discriminator D takes either real or generated images, and estimates the energy value E ∈ R accordingly, as explained later. For simplicity, we assume that D produces non-negative values, but the analysis would hold as long as the values are bounded below.
2.1 OBJECTIVE FUNCTIONAL
The output of the discriminator goes through an objective functional in order to shape the energy function, attributing low energy to the real data samples and higher energy to the generated (“fake”) ones. In this work, we use a margin loss, but many other choices are possible as explained in LeCun et al. (2006). Similarly to what has been done with the probabilistic GAN (Goodfellow et al., 2014), we use a two different losses, one to train D and the other to train G, in order to get better quality gradients when the generator is far from convergence. Given a positive margin m, a data sample x and a generated sample G(z), the discriminator loss LD and the generator loss LG are formally defined by:
LD(x, z) = D(x) + [m−D ( G(z) ) ]+ (1)
LG(z) = D ( G(z) ) (2)
where [·]+ = max(0, ·). Minimizing LG with respect to the parameters of G is similar to maximizing the second term of LD. It has the same minimum but non-zero gradients when D(G(z)) ≥ m.
2.2 OPTIMALITY OF THE SOLUTION
In this section, we present a theoretical analysis of the system presented in section 2.1. We show that if the system reaches a Nash equilibrium, then the generator G produces samples that are indistinguishable from the distribution of the dataset. This section is done in a non-parametric setting, i.e. we assume that D and G have infinite capacity.
Given a generator G, let pG be the density distribution of G(z) where z ∼ pz . In other words, pG is the density distribution of the samples generated by G. We define V (G,D) = ∫ x,z LD(x, z)pdata(x)pz(z)dxdz and U(G,D) = ∫ z LG(z)pz(z)dz. We train the discriminatorD to minimize the quantity V and the generatorG to minimize the quantityU . A Nash equilibrium of the system is a pair (G∗, D∗) that satisfies:
V (G∗, D∗) ≤ V (G∗, D) ∀D (3) U(G∗, D∗) ≤ U(G,D∗) ∀G (4)
Theorem 1. If (D∗, G∗) is a Nash equilibrium of the system, then pG∗ = pdata almost everywhere, and V (D∗, G∗) = m.
Proof. First we observe that
V (G∗, D) = ∫ x pdata(x)D(x)dx+ ∫ z pz(z) [m−D(G∗(z))]+ dz (5)
= ∫ x ( pdata(x)D(x) + pG∗(x) [m−D(x)]+ ) dx. (6)
The analysis of the function ϕ(y) = ay+b(m−y)+ (see lemma 1 in appendix A for details) shows: (a) D∗(x) ≤ m almost everywhere. To verify it, let us assume that there exists a set of measure non-zero such that D∗(x) > m. Let D̃(x) = min(D∗(x),m). Then V (G∗, D̃) < V (G∗, D∗) which violates equation 3. (b) The function ϕ reaches its minimum in m if a < b and in 0 otherwise. So V (G∗, D) reaches its minimum when we replace D∗(x) by these values. We obtain
V (G∗, D∗) = m ∫ x 1pdata(x)<pG∗ (x)pdata(x)dx+m ∫ x 1pdata(x)≥pG∗ (x)pG∗(x)dx (7)
= m ∫ x ( 1pdata(x)<pG∗ (x)pdata(x) + ( 1− 1pdata(x)<pG∗ (x) ) pG∗(x) ) dx (8)
= m ∫ x pG∗(x)dx+m ∫ x 1pdata(x)<pG∗ (x)(pdata(x)− pG∗(x))dx (9)
= m+m ∫ x 1pdata(x)<pG∗ (x)(pdata(x)− pG∗(x))dx. (10)
The second term in equation 10 is non-positive, so V (G∗, D∗) ≤ m. By putting the ideal generator that generates pdata into the right side of equation 4, we get∫
x
pG∗(x)D ∗(x)dx ≤ ∫ x pdata(x)D ∗(x)dx. (11)
Thus by (6), ∫ x pG∗(x)D ∗(x)dx+ ∫ x pG∗(x)[m−D∗(x)]+dx ≤ V (G∗, D∗) (12)
and since D∗(x) ≤ m, we get m ≤ V (G∗, D∗). Thus, m ≤ V (G∗, D∗) ≤ m i.e. V (G∗, D∗) = m. Using equation 10, we see that can only happen if ∫ x 1pdata(x)<pG(x)dx = 0, which is true if and only if pG = pdata almost everywhere (this is because pdata and pG are probabilities densities, see lemma 2 in the appendix A for details).
Theorem 2. A Nash equilibrium of this system exists and is characterized by (a) pG∗ = pdata (almost everywhere) and (b) there exists a constant γ ∈ [0,m] such that D∗(x) = γ (almost everywhere).1.
1This is assuming there is no region where pdata(x) = 0. If such a region exists, D∗(x) may have any value in [0,m] for x in this region.
Proof. See appendix A.
2.3 USING AUTO-ENCODERS
In our experiments, the discriminator D is structured as an auto-encoder:
D(x) = ||Dec(Enc(x))− x||. (13)
The diagram of the EBGAN model with an auto-encoder discriminator is depicted in figure 1. The choice of the auto-encoders for D may seem arbitrary at the first glance, yet we postulate that it is conceptually more attractive than a binary logistic network:
• Rather than using a single bit of target information to train the model, the reconstruction-based output offers a diverse targets for the discriminator. With the binary logistic loss, only two targets are possible, so within a minibatch, the gradients corresponding to different samples are most likely far from orthogonal. This leads to inefficient training, and reducing the minibatch sizes is often not an option on current hardware. On the other hand, the reconstruction loss will likely produce very different gradient directions within the minibatch, allowing for larger minibatch size without loss of efficiency. • Auto-encoders have traditionally been used to represent energy-based model and arise naturally. When trained with some regularization terms (see section 2.3.1), auto-encoders have the ability to learn an energy manifold without supervision or negative examples. This means that even when an EBGAN auto-encoding model is trained to reconstruct a real sample, the discriminator contributes to discovering the data manifold by itself. To the contrary, without the presence of negative examples from the generator, a discriminator trained with binary logistic loss becomes pointless.
2.3.1 CONNECTION TO THE REGULARIZED AUTO-ENCODERS
One common issue in training auto-encoders is that the model may learn little more than an identity function, meaning that it attributes zero energy to the whole space. In order to avoid this problem, the model must be pushed to give higher energy to points outside the data manifold. Theoretical and experimental results have addressed this issue by regularizing the latent representations (Vincent et al., 2010; Rifai et al., 2011; MarcAurelio Ranzato & Chopra, 2007; Kavukcuoglu et al., 2010). Such regularizers aim at restricting the reconstructing power of the auto-encoder so that it can only attribute low energy to a smaller portion of the input points.
We argue that the energy function (the discriminator) in the EBGAN framework is also seen as being regularized by having a generator producing the contrastive samples, to which the discriminator ought to give high reconstruction energies. We further argue that the EBGAN framework allows more flexibility from this perspective, because: (i)-the regularizer (generator) is fully trainable instead of being handcrafted; (ii)-the adversarial training paradigm enables a direct interaction between the duality of producing contrastive sample and learning the energy function.
2.4 REPELLING REGULARIZER
We propose a “repelling regularizer” which fits well into the EBGAN auto-encoder model, purposely keeping the model from producing samples that are clustered in one or only few modes of pdata. Another technique “minibatch discrimination” was developed by Salimans et al. (2016) from the same philosophy.
Implementing the repelling regularizer involves a Pulling-away Term (PT) that runs at a representation level. Formally, let S ∈ Rs×N denotes a batch of sample representations taken from the encoder output layer. Let us define PT as:
fPT (S) = 1 N(N − 1) ∑ i ∑ j 6=i ( STi Sj ‖Si‖‖Sj‖ )2 . (14)
PT operates on a mini-batch and attempts to orthogonalize the pairwise sample representation. It is inspired by the prior work showing the representational power of the encoder in the auto-encoder alike model such as Rasmus et al. (2015) and Zhao et al. (2015). The rationale for choosing the cosine similarity instead of Euclidean distance is to make the term bounded below and invariant to scale. We use the notation “EBGAN-PT” to refer to the EBGAN auto-encoder model trained with this term. Note the PT is used in the generator loss but not in the discriminator loss.
3 RELATED WORK
Our work primarily casts GANs into an energy-based model scope. On this direction, the approaches studying contrastive samples are relevant to EBGAN, such as the use of noisy samples (Vincent et al., 2010) and noisy gradient descent methods like contrastive divergence (Carreira-Perpinan & Hinton, 2005). From the perspective of GANs, several papers were presented to improve the stability of GAN training, (Salimans et al., 2016; Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Mathieu et al., 2015).
Kim & Bengio (2016) propose a probabilistic GAN and cast it into an energy-based density estimator by using the Gibbs distribution. Quite unlike EBGAN, this proposed framework doesn’t get rid of the computational challenging partition function, so the choice of the energy function is required to be integratable.
4 EXPERIMENTS
4.1 EXHAUSTIVE GRID SEARCH ON MNIST
In this section, we study the training stability of EBGANs over GANs on a simple task of MNIST digit generation with fully-connected networks. We run an exhaustive grid search over a set of architectural choices and hyper-parameters for both frameworks.
Formally, we specify the search grid in table 1. We impose the following restrictions on EBGAN models: (i)-using learning rate 0.001 and Adam (Kingma & Ba, 2014) for both G and D; (ii)nLayerD represents the total number of layers combiningEnc andDec. For simplicity, we fixDec to be one layer and only tune the Enc #layers; (iii)-the margin is set to 10 and not being tuned. To analyze the results, we use the inception score (Salimans et al., 2016) as a numerical means reflecting the generation quality. Some slight modification of the formulation were made to make figure 2 visually more approachable while maintaining the score’s original meaning, I ′ = ExKL(p(y)||p(y|x))2 (more details in appendix C). Briefly, higher I ′ score implies better generation quality.
Histograms We plot the histogram of I ′ scores in figure 2. We further separated out the optimization related setting from GAN’s grid (optimD, optimG and lr) and plot the histogram of each subgrid individually, together with the EBGAN I ′ scores as a reference, in figure 3. The number of experiments for GANs and EBGANs are both 512 in every subplot. The histograms evidently show that EBGANs are more reliably trained.
2This form of the “inception score” is only used to better analyze the grid search in the scope of this work, but not to compare with any other published work.
Digits generated from the configurations presenting the best inception score are shown in figure 4.
4.2 SEMI-SUPERVISED LEARNING ON MNIST
We explore the potential of using the EBGAN framework for semi-supervised learning on permutation-invariant MNIST, collectively on using 100, 200 and 1000 labels. We utilized a bottom-
layer-cost Ladder Network (LN) (Rasmus et al., 2015) with the EGBAN framework (EBGAN-LN). Ladder Network can be categorized as an energy-based model that is built with both feedforward and feedback hierarchies powered by stage-wise lateral connections coupling two pathways.
One technique we found crucial in enabling EBGAN framework for semi-supervised learning is to gradually decay the margin value m of the equation 1. The rationale behind is to let discriminator punish generator less when pG gets closer to the data manifold. One can think of the extreme case where the contrastive samples are exactly pinned on the data manifold, such that they are “not contrastive anymore”. This ultimate status happens when m = 0 and the EBGAN-LN model falls back to a normal Ladder Network. The undesirability of a non-decay dynamics for using the discriminator in the GAN or EBGAN framework is also indicated by Theorem 2: on convergence, the discriminator reflects a flat energy surface. However, we posit that the trajectory of learning a EBGAN-LN model does provide the LN (discriminator) more information by letting it see contrastive samples. Yet the optimal way to avoid the mentioned undesirability is to make sure m has been decayed to 0 when the Nash Equilibrium is reached. The margin decaying schedule is found by hyper-parameter search in our experiments (technical details in appendix D).
From table 2, it shows that positioning a bottom-layer-cost LN into an EBGAN framework profitably improves the performance of the LN itself. We postulate that within the scope of the EBGAN framework, iteratively feeding the adversarial contrastive samples produced by the generator to the energy function acts as an effective regularizer; the contrastive samples can be thought as an extension to the dataset that provides more information to the classifier. We notice there was a discrepancy between the reported results between Rasmus et al. (2015) and Pezeshki et al. (2015), so we report both results along with our own implementation of the Ladder Network running the same setting. The specific experimental setting and analysis are available in appendix D.
4.3 LSUN & CELEBA
We apply the EBGAN framework with deep convolutional architecture to generate 64 × 64 RGB images, a more realistic task, using the LSUN bedroom dataset (Yu et al., 2015) and the large-scale face dataset CelebA under alignment (Liu et al., 2015). To compare EBGANs with DCGANs (Radford et al., 2015), we train a DCGAN model under the same configuration and show its generation side-by-side with the EBGAN model, in figures 5 and 6. The specific settings are listed in appendix C.
4.4 IMAGENET
Finally, we trained EBGANs to generate high-resolution images on ImageNet (Russakovsky et al., 2015). Compared with the datasets we have experimented so far, ImageNet presents an extensively larger and wilder space, so modeling the data distribution by a generative model becomes very challenging. We devised an experiment to generate 128 × 128 images, trained on the full ImageNet-1k dataset, which contains roughly 1.3 million images from 1000 different categories. We also trained a network to generate images of size 256 × 256, on a dog-breed subset of ImageNet, using the wordNet IDs provided by Vinyals et al. (2016). The results are shown in figures 7 and 8. Despite the difficulty of generating images on a high-resolution level, we observe that EBGANs are able to learn about the fact that objects appear in the foreground, together with various background components resembling grass texture, sea under the horizon, mirrored mountain in the water, buildings, etc. In addition, our 256 × 256 dog-breed generations, although far from realistic, do reflect some knowledge about the appearances of dogs such as their body, furs and eye.
5 OUTLOOK
We bridge two classes of unsupervised learning methods – GANs and auto-encoders – and revisit the GAN framework from an alternative energy-based perspective. EBGANs show better convergence pattern and scalability to generate high-resolution images. A family of energy-based loss functionals presented in LeCun et al. (2006) can easily be incorporated into the EBGAN framework. For the future work, the conditional setting (Denton et al., 2015; Mathieu et al., 2015) is a promising setup to explore. We hope the future research will raise more attention on a broader view of GANs from the energy-based perspective.
ACKNOWLEDGMENT
We thank Emily Denton, Soumith Chitala, Arthur Szlam, Marc’Aurelio Ranzato, Pablo Sprechmann, Ross Goroshin and Ruoyu Sun for fruitful discussions. We also thank Emily Denton and Tian Jiang for their help with the manuscript.
A APPENDIX: TECHNICAL POINTS OF SECTION 2.2
Lemma 1. Let a, b ≥ 0, ϕ(y) = ay + b [m− y]+. The minimum of ϕ on [0,+∞) exists and is reached in m if a < b, and it is reached in 0 otherwise (the minimum may not be unique).
Proof. The function ϕ is defined on [0,+∞), its derivative is defined on [0,+∞)\{m} and ϕ′(y) = a− b if y ∈ [0,m) and ϕ′(y) = a if y ∈ (m,+∞). So when a < b, the function is decreasing on [0,m) and increasing on (m,+∞). Since it is continuous, it has a minimum in m. It may not be unique if a = 0 or a− b = 0. On the other hand, if a ≥ b the function ϕ is increasing on [0,+∞), so 0 is a minimum.
Lemma 2. If p and q are probability densities, then ∫ x 1p(x)<q(x)dx = 0 if and only if∫
x 1p(x) 6=q(x)dx = 0.
Proof. Let’s assume that ∫ x 1p(x)<q(x)dx = 0. Then∫
x
1p(x)>q(x)(p(x)− q(x))dx (15)
= ∫ x (1− 1p(x)≤q(x))(p(x)− q(x))dx (16)
= ∫ x p(x)dx− ∫ x q(x)dx+ ∫ x 1p(x)≤q(x)(p(x)− q(x))dx (17)
= 1− 1 + ∫ x ( 1p(x)<q(x) + 1p(x)=q(x) ) (p(x)− q(x))dx (18)
= ∫ x 1p(x)<q(x)(p(x)− q(x))dx+ ∫ x 1p(x)=q(x)(p(x)− q(x))dx (19) = 0 + 0 = 0 (20)
So ∫ x 1p(x)>q(x)(p(x) − q(x))dx = 0 and since the term in the integral is always non-negative, 1p(x)>q(x)(p(x) − q(x)) = 0 for almost all x. And p(x) − q(x) = 0 implies 1p(x)>q(x) = 0, so 1p(x)>q(x) = 0 almost everywhere. Therefore ∫ x 1p(x)>q(x)dx = 0 which completes the proof, given the hypothesis.
Proof of theorem 2 The sufficient conditions are obvious. The necessary condition on G∗ comes from theorem 1, and the necessary condition on D∗(x) ≤ m is from the proof of theorem 1. Let us now assume that D∗(x) is not constant almost everywhere and find a contradiction. If it is not, then there exists a constant C and a set S of non-zero measure such that ∀x ∈ S, D∗(x) ≤ C and ∀x 6∈ S, D∗(X) > C. In addition we can choose S such that there exists a subset S ′ ⊂ S of non-zero measure such that pdata(x) > 0 on S ′ (because of the assumption in the footnote). We can build a generator G0 such that pG0(x) ≤ pdata(x) over S and pG0(x) < pdata(x) over S ′. We compute
U(G∗, D∗)− U(G0, D∗) = ∫ x (pdata − pG0)D∗(x)dx (21)
= ∫ x (pdata − pG0)(D∗(x)− C)dx (22)
= ∫ S (pdata − pG0)(D∗(x)− C)dx+ ∫ RN\S (pdata − pG0)(D∗(x)− C)dx (23) > 0 (24)
which violates equation 4.
B APPENDIX: MORE INTERPRETATIONS ABOUT GANS AND ENERGY-BASED LEARNING
TWO INTERPRETATIONS OF GANS
GANs can be interpreted in two complementary ways. In the first interpretation, the key component is the generator, and the discriminator plays the role of a trainable objective function. Let us imagine that the data lies on a manifold. Until the generator produces samples that are recognized as being on the manifold, it gets a gradient indicating how to modify its output so it could approach the manifold. In such scenario, the discriminator acts to punish the generator when it produces samples that are outside the manifold. This can be understood as a way to train the generator with a set of possible desired outputs (e.g. the manifold) instead of a single desired output as in traditional supervised learning.
For the second interpretation, the key component is the discriminator, and the generator is merely trained to produce contrastive samples. We show that by iteratively and interactively feeding contrastive samples, the generator enhances the semi-supervised learning performance of the discriminator (e.g. Ladder Network), in section 4.2.
C APPENDIX: EXPERIMENT SETTINGS
MORE DETAILS ABOUT THE GRID SEARCH
For training both EBGANs and GANs for the grid search, we use the following setting:
• Batch normalization (Ioffe & Szegedy, 2015) is applied after each weight layer, except for the generator output layer and the discriminator input layer (Radford et al., 2015). • Training images are scaled into range [-1,1]. Correspondingly the generator output layer is followed by a Tanh function. • ReLU is used as the non-linearity function. • Initialization: the weights in D fromN (0, 0.002) and in G fromN (0, 0.02). The bias are initial-
ized to be 0.
We evaluate the models from the grid search by calculating a modified version of the inception score, I ′ = ExKL(p(y)||p(y|x)), where x denotes a generated sample and y is the label predicted by a MNIST classifier that is trained off-line using the entire MNIST training set. Two main changes were made upon its original form: (i)-we swap the order of the distribution pair; (ii)-we omit the e(·) operation. The modified score condenses the histogram in figure 2 and figure 3. It is also worth noting that although we inherit the name “inception score” from Salimans et al. (2016), the evaluation isn’t related to the “inception” model trained on ImageNet dataset. The classifier is a regular 3-layer ConvNet trained on MNIST.
The generations showed in figure 4 are the best GAN or EBGAN (obtaining the best I ′ score) from the grid search. Their configurations are:
• figure 4(a): nLayerG=5, nLayerD=2, sizeG=1600, sizeD=1024, dropoutD=0, optimD=SGD, optimG=SGD, lr=0.01. • figure 4(b): nLayerG=5, nLayerD=2, sizeG=800, sizeD=1024, dropoutD=0, optimD=ADAM, optimG=ADAM, lr=0.001, margin=10. • figure 4(c): same as (b), with λPT = 0.1.
LSUN & CELEBA
We use a deep convolutional generator analogous to DCGAN’s and a deep convolutional autoencoder for the discriminator. The auto-encoder is composed of strided convolution modules in the feedforward pathway and fractional-strided convolution modules in the feedback pathway. We leave the usage of upsampling or switches-unpooling (Zhao et al., 2015) to future research. We also followed the guidance suggested by Radford et al. (2015) for training EBGANs. The configuration of the deep auto-encoder is:
• Encoder: (64)4c2s-(128)4c2s-(256)4c2s • Decoder: (128)4c2s-(64)4c2s-(3)4c2s
where “(64)4c2s” denotes a convolution/deconvolution layer with 64 output feature maps and kernel size 4 with stride 2. The margin m is set to 80 for LSUN and 20 for CelebA.
IMAGENET
We built deeper models in both 128×128 and 256×256 experiments, in a similar fashion to section 4.3,
• 128× 128 model: – Generator: (1024)4c-(512)4c2s-(256)4c2s-(128)4c2s(64)4c2s-(64)4c2s-(3)3c
– Noise #planes: 100-64-32-16-8-4 – Encoder: (64)4c2s-(128)4c2s-(256)4c2s-(512)4c2s – Decoder: (256)4c2s-(128)4c2s-(64)4c2s-(3)4c2s – Margin: 40
• 256× 256 model: – Generator: (2048)4c-(1024)4c2s-(512)4c2s-(256)4c2s-(128)4c2s(64)4c2s-(64)4c2s-(3)3c
– Noise #planes: 100-64-32-16-8-4-2 – Encoder: (64)4c2s-(128)4c2s-(256)4c2s-(512)4c2s – Decoder: (256)4c2s-(128)4c2s-(64)4c2s-(3)4c2s – Margin: 80
Note that we feed noise into every layer of the generator where each noise component is initialized into a 4D tensor and concatenated with current feature maps in the feature space. Such strategy is also employed by Salimans et al. (2016).
D APPENDIX: SEMI-SUPERVISED LEARNING EXPERIMENT SETTING
BASELINE MODEL
As stated in section 4.2, we chose a bottom-layer-cost Ladder Network as our baseline model. Specifically, we utilize an identical architecture as reported in both papers (Rasmus et al., 2015; Pezeshki et al., 2015); namely a fully-connected network of size 784-1000-500-250-250-250, with batch normalization and ReLU following each linear layer. To obtain a strong baseline, we tuned the weight of the reconstruction cost with values from the set { 5000784 , 2000 784 , 1000 784 , 500 784}, while fixing the weight on the classification cost to 1. In the meantime, we also tuned the learning rate with values {0.002, 0.001, 0.0005, 0.0002, 0.0001}. We adopted Adam as the optimizer with β1 being set to 0.5. The minibatch size was set to 100. All the experiments are finished by 120,000 steps. We use the same learning rate decay mechanism as in the published papers – starting from the two-thirds of total steps (i.e., from step #80,000) to linearly decay the learning rate to 0. The result reported in section 4.2 was done by the best tuned setting: λL2 = 1000 784 , lr = 0.0002.
EBGAN-LN MODEL
We place the same Ladder Network architecture into our EBGAN framework and train this EBGANLN model the same way as we train the EBGAN auto-encoder model. For technical details, we started training the EBGAN-LN model from the margin value 16 and gradually decay it to 0 within the first 60,000 steps. By the time, we found that the reconstruction error of the real image had already been low and reached the limitation of the architecture (Ladder Network itself); besides the generated images exhibit good quality (shown in figure 10). Thereafter we turned off training the generator but kept training the discriminator for another 120,000 steps. We set the initial learning rates to be 0.0005 for discriminator and 0.00025 for generator. The other setting is kept consistent with the best baseline LN model. The learning rate decay started at step #120,000 (also two-thirds of the total steps).
OTHER DETAILS
• Notice that we used the 28×28 version (unpadded) of the MNIST dataset in the EBGAN-LN experiment. For the EBGAN auto-encoder grid search experiments, we used the zero-padded version, i.e., size 32×32. No phenomenal difference has been found due to the zero-padding. • We generally took the `2 norm of the discrepancy between input and reconstruction for the loss term in the EBGAN auto-encoder model as formally written in section 2.1. However, for the EBGAN-LN experiment, we followed the original implementation of Ladder Network using a vanilla form of `2 loss. • Borrowed from Salimans et al. (2016), the batch normalization is adopted without the learned parameter γ but merely with a bias term β. It still remains unknown whether such trick could affect learning in some non-ignorable way, so this might have made our baseline model not a strict reproduction of the published models by Rasmus et al. (2015) and Pezeshki et al. (2015).
E APPENDIX: TIPS FOR SETTING A GOOD ENERGY MARGIN VALUE
It is crucial to set a proper energy margin valuem in the framework of EBGAN, from both theoretical and experimental perspective. Hereby we provide a few tips:
• Delving into the formulation of the discriminator loss made by equation 1, we suggest a numerical balance between its two terms which concern real and fake sample respectively. The second term is apparently bounded by [0,m] (assuming the energy function D(x) is non-negative). It is desirable to make the first term bounded in a similar range. In theory, the upper bound of the first term is essentially determined by (i)-the capacity of D; (ii)-the complexity of the dataset. • In practice, for the EBGAN auto-encoder model, one can run D (the auto-encoder) alone on the real sample dataset and monitor the loss. When it converges, the consequential loss implies a rough limit on how well such setting of D is capable to fit the dataset. This usually suggests a good start for a hyper-parameter searching on m. • m being overly large results in a training instability/difficulty, while m being too small is prone to the mode-dropping problem. This property of m is depicted in figure 9. • One successful technique, as we introduced in appendix D, is to start from a large m and gradually decayed it to 0 along training proceeds. Unlike the feature matching semisupervised learning technique proposed by Salimans et al. (2016), we show in figure 10 that not only does the EBGAN-LN model achieve a good semi-supervised learning performance, it also produces satisfactory generations.
Abstracting away from the practical experimental tips, the theoretical understanding of EBGAN in section 2.2 also provides some insight for setting a feasible m. For instance, as implied by Theorem 2, setting a large m results in a broader range of γ to which D∗(x) may converge. Instability may come after an overly large γ because it generates two strong gradients pointing to opposite directions, from loss 1, which would demand more finicky optimization setting.
F APPENDIX: MORE GENERATION
LSUN AUGMENTED VERSION TRAINING
For LSUN bedroom dataset, aside from the experiment on the whole images, we also train an EBGAN auto-encoder model based on dataset augmentation by cropping patches. All the patches are of size 64× 64 and cropped from 96× 96 original images. The generation is shown in figure 11.
COMPARISON OF EBGANS AND EBGAN-PTS
To further demonstrate how the pull-away term (PT) may influence EBGAN auto-encoder model training, we chose both the whole-image and augmented-patch version of the LSUN bedroom dataset, together with the CelebA dataset to make some further experimentation. The comparison of EBGAN and EBGAN-PT generation are showed in figure 12, figure 13 and figure 14. Note
that all comparison pairs adopt identical architectural and hyper-parameter setting as in section 4.3. The cost weight on the PT is set to 0.1.

This paper proposes a novel extension of generative adversarial networks that replaces the traditional binary classifier discriminator with one that assigns a scalar energy to each point in the generator's output domain. The discriminator minimizes a hinge loss while the generator attempts to generate samples with low energy under the discriminator. The authors show that a Nash equilibrium under these conditions yields a generator that matches the data distribution (assuming infinite capacity). Experiments are conducted with the discriminator taking the form of an autoencoder, optionally including a regularizer that penalizes generated samples having a high cosine similarity to other samples in the minibatch.
Pros:
* The paper is well-written.
* The topic will be of interest to many because it sets the stage for the exploration of a wider variety of discriminators than currently used for training GANs.
* The theorems regarding optimality of the Nash equilibrium appear to be correct.
* Thorough exploration of hyperparameters in the MNIST experiments.
* Semi-supervised results show that contrastive samples from the generator improve classification performance.
Cons:
* The relationship to other works that broaden the scope of the discriminator (e.g. [1]) or use a generative network to provide contrastive samples to an energy-based model ([2]) is not made clear in the paper.
* From visual inspection alone it is difficult to conclude whether EB-GANs produce better samples than DC-GANs on the LSUN and CelebA datasets.
* It is difficult to assess the effect of the PT regularizer beyond visual inspection as the Inception score results are computed with the vanilla EB-GAN.
Specific Comments
* Sec 2.3: It is unclear to me why a reconstruction loss will necessarily produce very different gradient directions.
* Sec 2.4: It is confusing that "pulling-away" is abbreviated as "PT".
* Sec 4.1: It seems strange that the Inception model (trained on natural images) is being used to compute KL scores for MNIST. Using an MNIST-trained CNN to compute Inception-style scores seems to be more appropriate here.
* Figure 3: There is little variation across the histograms, so this figure is not very enlightening.
* Appendix A: In the proof of theorem 2, it is unclear to me why a Nash equilibrium of the system exists.
Typos / Minor Comments
* Abstract: "probabilistic GANs" should probably be "traditional" or "classical" GANs.
* Theorem 2: "A Nash equilibrium ... exists"
* Sec 3: Should be "Several papers were presented"
Overall, I have some concerns with the related work and experimental evaluation sections, but I feel the model is novel enough and is well-justified by the optimality proofs and the quality of the generated samples.
[1] Springenberg, Jost Tobias. "Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks." arXiv preprint arXiv:1511.06390 (2015).
[2] Kim, Taesup, and Yoshua Bengio. "Deep Directed Generative Models with Energy-Based Probability Estimation." arXiv preprint arXiv:1606.03439 (2016).