Inverse problems are typically solved by first defining a model and then choosing an inference procedure. With this separation of modeling from inference, inverse problems can be framed in a modular way. For example, variational inference can be applied to a broad class of models. The modularity, however, typically goes away after model parameters have been trained under a chosen inference procedure. During training, model and inference often interact in a way that the model parameters will ultimately be adapted to the chosen inference procedure, posing the two components inseparable after training. But if model and inference become inseperable after training, why separate them in the first place? We propose a novel learning framework which abandons the dichotomy between model and inference. Instead, we introduce Recurrent Inference Machines (RIM), a class of recurrent neural networks (RNN), that directly learn to solve inverse problems. We demonstrate the effectiveness of RIMs in experiments on various image reconstruction tasks. We show empirically that RIMs exhibit the desirable convergence behavior of classical inference procedures, and that they can outperform state-ofthe-art methods when trained on specialized inference tasks. Our approach bridges the gap between inverse problems and deep learning, providing a framework for fast progression in the field of inverse problems.
1 INTRODUCTION
Inverse Problems are a broad class of problems which can be encountered in all scientific disciplines, from the natural sciences to engineering. The task in inverse problems is to reconstruct a signal from observations that are subject to a known (or inferred) corruption process known as the forward model. A typical example of an inverse problem is the linear measurement problem
y = Ax + n, (1)
where x is the signal of interest, A is an m × d corruption matrix, n is an additive noise vector, and y is the actual measurement. If A is a wide matrix such that m d, this problem is typically ill-posed. Many signal reconstruction problems can be phrased in terms of the linear measurement problem such as image denoising, super-resolution, deconvolution and so on. The general form of A typically defines the problem class. If A is an identity matrix the problem is a denoising problem, while in tomography A represents a Fourier transform and a consecutive sub-sampling of the Fourier coefficients.
Inverse problems are often formulated as an optimization problem of the form
min x d(y,Ax) + λR(x), (2)
where d(y,Ax) is the data fidelity term that enforces x to satisfy the observations y, and R(x) is a regularization term which restricts the solution to comply with a predefined model over x.
The difficulties that arise in this framework are two-fold: (1) it is difficult to choose R(x) such that it is an appropriate model for complex signals such as natural images, and (2) even under a well chosen R(x) the optimization procedure might become difficult.
Compressed sensing approaches give up on a versatileR(x) in order to define a convex optimization procedure. The idea is that the signal x has a sparse representation in some basis Ψ such that x = Ψu and that the optimization problem can be rephrased as
min u d(y,AΨu) + λ ‖u‖1 , (3)
where ‖·‖1 is the sparsity inducing L1-norm (Donoho, 2006a). Under certain classes of d(y,AΨu) such as quadratic errors the optimization problem becomes convex. Results from the compressed sensing literature offer provable bounds on the reconstruction performance for sparse signals of this form (Candès et al., 2006; Donoho, 2006b). The basis Ψ can also be learned from data (Aharon et al., 2006; Elad & Aharon, 2006).
Other approaches interpret equation (2) in terms of probabilities such that finding the solution is a matter of performing maximum a posteriori (MAP) estimation (Figueiredo et al., 2007). In those cases d(y,AΨu) takes the form of a log-likelihood and R(x) takes the form of a parametric logprior log pθ(x) over variable x such that the minimization becomes:
max x
log p(y|A,x) + log pθ(x). (4)
This allows for more expressiveness of R(x) and for the possibility of learning the prior pθ(x) from data. However, with more expressive priors optimization will become more difficult as well. In fact, only for a few trivial prior-likelihood pairs will inference remain convex. In practice one often has to resort to approximations of the objective and to approximate double-loop algorithms in order to allow for scalable inference (Nickisch & Seeger, 2009; Zoran & Weiss, 2011).
In this work we take a radically different approach to solving inverse problems. We move away from the idea that it is beneficial to separate learning a prior (regularizer) from the optimization to do the reconstruction. The usual thinking is that this separation allows for greater modularity and the possibility to interchange one of these two complementary components in order to build new algorithms. In practice however, we observe that the optimization procedure almost always has to be adapted to the model choice to achieve good performance (Aharon et al., 2006; Elad & Aharon, 2006; Nickisch & Seeger, 2009; Zoran & Weiss, 2011). In fact, it is well known that the optimization procedure used for training should match the one used during testing because the model has adapted itself to perform well under that optimization procedure (Kumar et al., 2005; Wainwright, 2006).
What we need is a single framework which allows us to backpropagate through the optimization procedure when we learn the free parameters. Hence, We propose to look at inverse problems as a direct mapping from observations to estimated signal,
x̂ = fφ(A,y) (5)
where x̂ is an estimate of signal x from observations (A,y). Here we define φ as a set of learnable parameters which define the inference algorithm as well as constraints on x. The goal is thus to define map whose parameters are directly optimized for solving the inverse problem itself. It has the benefits of both having high expressive power (if the map fφ is complex enough) as well as being fast at inference time.
This paradigm shift allows us to learn and combine the effect of a prior, the reconstruction fidelity and an inference method without the need to explicitly define the functional form of all components. The whole procedure is simply interpreted as a single RNN. As a result, there is no need for sparsity assumptions, the introduction of model constraints to allow for convexity, or even for double-loop algorithms (Gregor & LeCun, 2010). In fact the proposed framework allows for use of current deep learning approaches which have high expressive power without trading off scalability. It further allows us to move all the manual parameter tuning - which is still common in traditional approaches (Zoran & Weiss, 2011) - away from the inference phase and into the learning phase. We believe this framework can be an important asset to introduce deep learning into the domain of inverse problems.
2 RECURRENT INFERENCE MACHINES
The goal of this work is to find an inverse model as described in equation (5). Often, however, it will be intractable to find (5) directly, even with modern non-linear function approximators. For high-dimensional y and x, which are typically considered in inverse problems, it will simply not be possible to fit matrix A into memory explicitly, but instead matrix A will be replaced by an operator that acts on x. An example is the Discrete Fourier Transform (DFT). Instead of using a Fourier matrix which is quadratic in the size of x, DFTs are typically performed using the Fast Fourier Transform (FFT) algorithm which reduces computational cost and memory consumption significantly. The use of operators, however, does not allow us to feed A into (5) anymore, but instead we will have to resort to an iterative approach that alternates between updates of x and
evaluation of Ax. This is precisely what is typically done in gradient-based inference methods, and we will motivate our framework from there.
2.1 GRADIENT-BASED INFERENCE
Recall from equation (4) that inverse problems can be interpreted in terms of probability such that optimization is an iterative approach to MAP inference. In its most simple form each consecutive estimate of x is then computed through a recursive function of the form
xt+1 = xt + γt∇ ( log p (y|A,x) + log pθ (x) ) (xt) (6)
where we make use of the fact that p(x|A,y) ∝ p(y|A,x)pθ(x) and γt is the step size or learning rate at iteration t. Further, A is a (partially-)observable covariate, p(y|A,x) is the likelihood function for a given inference problem, and pθ (x) is a prior over signal x. In many cases where either the likelihood term or the prior term deviate from standard models, optimization will not be convex. In constrast, the approach presented in this work is completely freed from ideas about convexity, as will be shown in the next section.
2.2 RECURRENT FUNCTION DEFINITION
The central insight of this work is that update equation (6) can be generalized such that xt+1 = xt + gφ(∇y|x,xt) (7)
where we denote∇ log p(y|A,x)(xt) by∇y|x for readability and φ is a set of learnable parameters that govern the updates of x. In this representation, prior parameters θ and learning rate parameters γ have been merged into one set of trainable parameters φ.
To recover the original update equation (6), gφ(∇y|x,xt) is written as gφ(∇y|x,xt) = γt ( ∇y|x +∇x ) (8)
where we make use of ∇x to denote ∇ log pθ(x)(xt). It will be useful to dissect the terms on the right-hand side of (8) to make sense of the usefulness of the modification.
First notice, that in equation (6) we never explicitly evaluate the prior, but only evaluate its gradient in order to perform updates. If never used, learning a prior appears to be unnecessary, and instead it appears more reasonable to directly learn a gradient function ∇x = fθ(xt) ∈ Rd. The advantage of working solely with gradients is that they do no require the evaluation of an (often) intractable normalization constant of pθ(x).
A second observation is that the step sizes γt are usually subject to either a chosen schedule or chosen through a deterministic algorithm such as a line search. That means the step sizes are always chosen according to a predefined model Γ. Interestingly, this model is usually not learned. In order to make inference faster and improve performance we suggest to learn the model Γ as well.
In (7) we have made the prior pθ(x) and the the step size model Γ implicit in function gφ(∇y|x,ηt). We explicitly keep ∇y|x as an input to (7) because - as opposed to Λ and pθ(x) - it represents extrinsic information that is injected into the model. It allows for changes in the likelihood model p(y|x) without the need to retrain parameters φ of the inference model gφ. Figure 1 gives a visual summary of the insights from this section.
2.3 OUTPUT CONSTRAINTS
In many problem domains the range of values for variable x is naturally constraint. For example, images typically have pixels with strictly positive values. In order to model this constraint we make use of nonlinear link functions as they are typically used in neural networks, such that
x = Ψ(η) (9) where Ψ(·) is any differentiable link function and η is the space in which RIMs iterate such that update equation (7) is replaced by
ηt+1 = ηt + gφ(∇y|η,ηt) (10) As a result x can be constraint to a certain range of values through Ψ(·), whereas iterations are performed in the unconstrained space of η
2.4 RECURRENT NETWORKS
A useful extension of (7) is to introduce a latent state variable st into the procedure. This latent variable is typically used as a utility in recurrent neural networks to learn temporal dependencies in data processing. With an additional latent variable the update equations become
ηt+1 = ηt + hφ ( ∇y|η,ηt, st+1 ) (11) st+1 = h ∗ φ ( ∇y|η,ηt, st ) (12)
where h∗φ(·) is the update model for state variable s. The variable s will allow the procedure to have memory in order to track progression, curvature, approximate a preconditioning matrix Tt (such as in BFGS) and determine a stopping criterion among other things. The concept of a temporal memory is quite limited in classical inference methods, which will allow RIMs to have a potential advantage over these methods.
2.5 TRAINING
In order to learn a step-wise inference procedure it will be necessary to simulate the inference steps during training. I.e. during training, an RIM will perform a number of inference steps T . At each step the model will produce a prediction as depicted in figure Figure 1. Each of those predictions is then subject to a loss, which encourages the model to produce predictions that improve over time. In it’s simplest form we can define a loss which is simply a weighted sum of the individual prediction losses at each time step such that
Ltotal(φ) = T∑ t=1 wtL(xt(φ),x) (13)
is the total loss. Here, L(·) is a base loss function such as the mean square error, wt is a positive scalar and xt(φ) is a prediction at time t. In this work we follow Andrychowicz et al. (2016) in setting wt = 1 for all time steps.
3 RELATED WORK
The RIM framework can be seen as an auto-encoder framework in which only the decoder is trained, whereas the encoder is given by a known corruption process. In terms of the training procedure this makes RIMs very similar to denoising auto-encoders (Vincent et al., 2008). Though initially with the objective of regularization in mind, denoising auto-encoders have been shown to be effectively used as generative models (Vincent et al., 2010). The difference of RIMs to denoising auto-encoders and also more recently developed auto-encoders such as Kingma & Welling (2014); Rezende et al. (2014) is that RIMs enforce coupling between encoder and decoder both, during training and test time. In it’s typical form, decoder and encoder of an auto-encoder are only coupled during training time, while there is no information flow during test time (Kingma & Welling, 2014; Rezende et al., 2014; Vincent et al., 2008; 2010). An exception is the work from Gregor et al. (2016) which is conceptually strongly related to RIMs. There, an RNN model is used to generate static data by drawing on a fixed canvas. An error signal is propagated throughout the generation process.
There have been approaches in the past which aim to formulate a framework in which an inference procedure is learned. One of the best known frameworks is LISTA (Gregor & LeCun, 2010) which aims to learn a model that reconstructs sparse codes from data. LISTA models try to fit into the classical framework of doing inference as described in 1, whereas RIMs are completely removed from assumptions about sparsity. A recent paper by Andrychowicz et al. (2016) aims to train RNNs as optimizers for non-convex optimization problems. Though introduced with a different intention, RIMs can be seen as a generalization of this approach, in which the model - in addition to the gradient information - is aware about the absolute position of a prediction in variable space(see equation (7)).
4 EXPERIMENTAL RESULTS
We evaluate our method on various kinds of image restoration tasks which can each be formulated in terms of linear measurement problems as described in equation (1). We first analyze the properties
of our proposed method on a set of restoration tasks from random projections. Later we compare our model on two well known image restoration tasks: image denoising and image super-resolution.
4.1 MODELS
If not specified otherwise we use the same RNN architecture for all experiments presented in this work. The chosen RNN consists of three convolutional hidden layers and a final convolutional output layer. All convolutional filters were chosen to be of size 3 x 3 pixels. The first hidden layer consists of convolutions with stride 2 (64 features), subsequent batch normalization and a tanh nonlinearity. The second hidden layer represents the RNN part of the model. We chose a gated recurrent unit (GRU) (Chung et al., 2014) with 256 features. The third hidden layer is a transpose convolution layer with 64 features which aims to recover the original image dimensions of the signal, followed again by a batch normalization layer and a tanh nonlinearity. All models have been trained on a fixed number of iterations of 20 steps. All methods were implemented in Tensorflow1.
4.2 DATA
All experiments were run on the BSD-300 data set (Martin et al., 2001)2. For training we extracted patches of size 32 x 32 pixels with stride 4 from the 200 training images available in the data set. In total this amounts to a data set of about 400 thousand image patches with highly redundant information. All models were trained over only two epochs, i.e. each unique image patch was seen by a model only twice during training. Validation was performed on a held-out data set of 1000 image patches.
For testing we either used the whole test set of 100 images from BSDS-300 or we used only a subset of 68 images which was introduced by Roth & Black (2005) and which is commonly used in the image restoration community 3.
4.3 IMAGE RESTORATION
All tasks addressed in this work assume a linear measurement problem of the form as described in equation (1) with additive (isotropic) Gaussian noise. In this case the gradient of the likelihood takes the form
∇y|x = 1
σ2 AT (y −Ax) (14)
where σ2 is the noise variance. For very small σ this gradient diverges. In order to make the gradient more stable also for small σ we chose to rewrite it as
∇y|x = 1
σ2 + AT (y −Ax) (15)
where = softplus(φ ) and φ is a trainable parameter. As a link function Ψ (see (9)) we chose the logistic sigmoid nonlinearity4 and we used the mean square error as training loss.
4.4 MULTI-TASK LEARNING WITH RANDOM PROJECTIONS
To analyze the properties of our proposed framework in terms of convergence and to test whether all components of the model are useful, we first trained the model to reconstruct image patches from noisy random projections of grayscale image patches. We consider three types of random projection matrices: (1) Gaussian ensembles with elements drawn from a standard normal distribution, (2) binary ensembles with entries of values {−1, 1} drawn from a Bernulli distribution with p = 0.5, and (3) Fourier ensembles with randomly sampled rows from a Fourier matrix (see Donoho (2006b)).
We trained three models on these tasks: (1) a Recurrent Inference Machine (RIM) as described in 2, (2) a gradient-descent network (GDN) which does not use the current estimate as an input (compare
1https://www.tensorflow.org 2https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/ 3http://www.visinf.tu-darmstadt.de/vi research/code/foe.en.jsp 4All training data was rescaled to be in the range [0, 1]
Andrychowicz et al. (2016)), and (3) a feed-forward network (FFN) which uses the same inputs as the RIM but where we replaced the GRU unit with a ReLu layer in order to remove state-dependence. Model (2) and (3) are simplifications of RIM in order to test the influence of each of the removed model components on prediction performance.
Figure 2 shows the reconstruction performance of all three models on random projections. In all tasks the RIM clearly outperforms both other models, showing overall consistent convergence behavior. The FFN performs well on easier tasks but starts to show degrading performance over time on more difficult tasks. This suggests that the state information of RIM plays an important role on the convergence behavior as well as overall performance. The GDN shows worst performance among all three models. For all tasks, the performance of GDN starts to degrade clearly after the 20 time steps that were used during training. We hypothesize that the model is able to compensate some of the missing information about the current estimate of x through state variable s during training, but the model is not able to transfer this ability to episodes with more iterations.
These results suggests that both the current estimate as well as the recurrent state carry useful information for performing inference. We will therefor only consider fully fledged RIMs from here on.
4.5 IMAGE DENOISING
After evaluating our model on 32 x 32 pixel image patches we wanted to see how reconstruction performance generalizes to full sized images and to an out of domain problem. We chose to reuse the RIM that was trained on the random projections task to perform image denoising. In this section we will call this model RIM-3task. To test the hypothesis that inference should be trained task specific, we further trained a model RIM-denoise solely on the denoising task. Table 2 shows the denoising performance through the mean PSNR on the BSD-300 test set for both models as compared to state-of-the-art methods in image denoising. The RIM-3task model shows very competitive results with other methods on all noise levels. This exemplifies that the model indeed has learned something reminiscent of a prior, as it was never directly trained on this task. The RIM-denoise model further improves upon the performance of RIM-3task and it outperforms most other methods on all noise levels. This is to say that the same RIM was used to perform denoising on different noise levels, and this model does not require any hand tuning after training.
Table 2 shows denoising perfomance on image that have been 8-bit quantized after adding noise(see Schmidt et al. (2016)). In this case performance slightly deteriorates for both models, though still making competitive with stateof-the-art methods. This effect could possibly be accommodated through further training, or by adjusting the forward model. Figure 3 gives some qualitative results on the denoising performance for one of the test images from BSD-300 as compared to the method from Zoran & Weiss (2011). RIM is able to produce more naturalistic images with less visible artifacts. The state variable in our RIM model allows for a growing receptive field size over time, which could explain the good long range interactions that the model shows.
Many denoising algorithms are solely tested on gray-scale images. Sometimes this is due to additional difficulties that multi-channel problems bring for some inference approaches. To show that it is straightforward to apply RIMs to multi-channel problems we trained a model to denoise RGB images. The denoising performance can be seen in table 1. The model is able to exploit correlations across color channels which allows for an additional boost in reconstruction performance.
4.6 IMAGE SUPER-RESOLUTION
We further tested our approach on the well known image super-resolution task. We trained a single RIM 5 on 36 x 36 pixel image patches from the BSD-300 training set to perform image super-
5The architecture of this model was slightly simplified in comparison to the previous problems. Instead of strided convolutions, we chose a trous convolutions. This model is more flexible and used only about 500.000 parameters. Previous experiments will be updated with the same model architecture.
resolution for factors 2, 3, and 46. We followed the same testing protocol as in Huang et al. (2015), and we used the test images that were retrieved from their website 7. Table 3 shows a comparison with some state-of-the-art methods on super-resolution for the BSD-300 test set. Figure 4 shows a qualitative example of super-resolution performance. The other deep learning method in this comparison, SRCNN Dong et al. (2014), is outperformed by RIM on all scales. Interestingly SRCNN was trained for each scale independently whereas we only trained one RIM for all scales. The chosen RIM has only about 500.000 parameters which amounts to about 2MB of disk space, which makes this architecture very attractive also for mobile computing.
6We reimplemented MATLABs bicubic interpolation kernel in order to apply a forward model (subsampling) in TensorFlow which agrees with the forward model in Huang et al. (2015).
7https://sites.google.com/site/jbhuang0604/publications/struct sr
Metric Scale Bicubic SRCNN A+ SelfExSR RIM (Ours)
PSNR 2x 29.55± 0.35 31.11± 0.39 31.22± 0.40 31.18± 0.39 31.39± 0.39 3x 27.20± 0.33 28.20± 0.36 28.30± 0.37 28.30± 0.37 28.51± 0.37 4x 25.96± 0.33 26.70± 0.34 26.82± 0.35 26.85± 0.36 27.01± 0.35
SSIM 2x 0.8425± 0.0078 0.8835± 0.0062 0.8862± 0.0063 0.8855± 0.0064 0.8885± 0.0062 3x 0.7382± 0.0114 0.7794± 0.0102 0.7836± 0.0104 0.7843± 0.0104 0.7888± 0.0101 4x 0.6672± 0.0131 0.7018± 0.0125 0.7089± 0.0125 0.7108± 0.0124 0.7156± 0.0125
Table 3: Image super-resolution performance on RGB images from BSD-300 test set. Mean and standard deviation (of the mean) of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) Wan (2004). Standard deviation of the mean was estimated from 10.000 boostrap samples. Test protocol and images taken from Huang et al. (2015). Only the three best performing methods from Huang et al. (2015) were chosen for comparison: SRCNN Dong et al. (2014), A+ Timofte et al. (2015), SelfExSR Huang et al. (2015). Best mean values in bold.
5 DISCUSSION
In this work, we introduce a general learning framework for solving inverse problems with deep learning approaches. We establish this framework by abandoning the traditional separation between model and inference. Instead, we propose to learn both components jointly without the need to define their explicit functional form. This paradigm shift enables us to bridge the gap between the fields of deep learning and inverse problems. We believe that this framework can have a major impact on many inverse problems, for example in medical imaging and radio astronomy. Although we have focused on linear image reconstruction tasks in this work, the framework can be applied to inverse problems of all kinds, such as non-linear inverse problems.
ACKNOWLEDGMENTS
The research was funded by the DOME project (Astron & IBM) and the Netherlands Organization for Scientific Research (NWO). The authors are greatful for helpful comments from Thomas Kipf, Mijung Park, Rajat Thomas, and Karen Ullrich.

Unfortunately, even after reading the authors' response to my pre-review question, I feel this paper in its current form lacks sufficient novelty to be accepted to ICLR.
Fundamentally, the paper suggests that traditional iterative algorithms for specific class of problems (ill-posed image inverse problems) can be replaced by discriminatively trained recurrent networks. As R3 also notes, un-rolled networks for iterative inference aren't new: they've been used to replace CRF-type inference, and _also_ to solve image inverse problems (my refs [1-3]). Therefore, I'd argue that the fundamental idea proposed by the paper isn't new---it is just that the paper seeks to 'formalize' it as an approach for inverse problems (although, there is nothing specific about the analysis that ties it to inverse problems: the paper only shows that the RIM can express gradient descent over prior + likelihood objective).
I also did not find the claims about benefits over prior approaches very compelling. The comment about parameter sharing works both ways---it is possible that untying the parameters leads to better performance over a fewer number of 'iterations', and given that the 'training set' is synthetically generated, learning a larger number of parameters doesn't seem to be an issue. Also, I'd argue that sharing the parameters is the 'obvious' approach, and the prior methods choose to not tie the parameters to get better accuracy.
The same holds for being able to handle different noise levels / scale sizes. A single model can always be trained to handle multiple forms of degradation---its just that its likely to do better when it's trained for specific degradation model/level. But more importantly, there is no evidence in the current set of experiments that shows that this is a property of the RIM architecture. (Moreover, this claim goes against one of the motivations of the paper of not training a single prior for different observation models ... but to train the entire inference architecture end-to-end).
It is possible that the proposed method does offer practical benefits beyond prior work---but these benefits don't come from the idea of simply unrolling iterations, which is not novel. I would strongly recommend that the authors consider a significant re-write of the paper---with a detailed discussion of prior work mentioned in the comments that highlights, with experiments, the specific aspects of their recurrent architecture that enables better recovery for inverse problems. I would also suggest that to claim the mantle of 'solving inverse problems', the paper consider a broader set of inverse tasks---in-painting, deconvolution, different noise models, and possibly working with multiple observations (like for HDR).


CALIBRATING ENERGY-BASED GENERATIVE ADVER-
In this paper we propose equipping Generative Adversarial Networks with the ability to produce direct energy estimates for samples. Specifically, we develop a flexible adversarial training framework, and prove this framework not only ensures the generator converges to the true data distribution, but also enables the discriminator to retain the density information at the global optimum. We derive the analytic form of the induced solution, and analyze its properties. In order to make the proposed framework trainable in practice, we introduce two effective approximation techniques. Empirically, the experiment results closely match our theoretical analysis, verifying that the discriminator is able to recover the energy of data distribution.
1 INTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) represent an important milestone on the path towards more effective generative models. GANs cast generative model training as a minimax game between a generative network (generator), which maps a random vector into the data space, and a discriminative network (discriminator), whose objective is to distinguish generated samples from real samples. Multiple researchers Radford et al. (2015); Salimans et al. (2016); Zhao et al. (2016) have shown that the adversarial interaction with the discriminator can result in a generator that produces compelling samples. The empirical successes of the GAN framework were also supported by the theoretical analysis of Goodfellow et al., who showed that, under certain conditions, the distribution produced by the generator converges to the true data distribution, while the discriminator converges to a degenerate uniform solution.
While GANs have excelled as compelling sample generators, their use as general purpose probabilistic generative models has been limited by the difficulty in using them to provide density estimates or even unnormalized energy values for sample evaluation.
It is tempting to consider the GAN discriminator as a candidate for providing this sort of scoring function. Conceptually, it is a trainable sample evaluation mechanism that – owing to GAN training paradigm – could be closely calibrated to the distribution modeled by the generator. If the discriminator could retain fine-grained information of the relative quality of samples, measured for instance by probability density or unnormalized energy, it could be used as an evaluation metric. Such data-driven evaluators would be highly desirable for problems where it is difficult to define evaluation criteria that correlate well with human judgment. Indeed, the real-valued discriminator of the recently introduced energy-based GANs Zhao et al. (2016) might seem like an ideal candidate energy function. Unfortunately, as we will show, the degenerate fate of the GAN discriminator at the optimum equally afflicts the energy-based GAN of Zhao et al..
In this paper we consider the questions: (i) does there exists an adversarial framework that induces a non-degenerate discriminator, and (ii) if so, what form will the resulting discriminator take? We introduce a novel adversarial learning formulation, which leads to a non-degenerate discriminator while ensuring the generator distribution matches the data distribution at the global optimum. We derive a general analytic form of the optimal discriminator, and discuss its properties and their
∗Part of this work was completed while author was at Maluuba Research
relationship to the specific form of the training objective. We also discuss the connection between the proposed formulation and existing alternatives such as the approach of Kim & Bengio (2016). Finally, for a specific instantiation of the general formulation, we investigate two approximation techniques to optimize the training objective, and verify our results empirically.
2 RELATED WORK
Following a similar motivation, the field of Inverse Reinforcement Learning (IRL) (Ng & Russell, 2000) has been exploring ways to recover the “intrinsic” reward function (analogous to the discriminator) from observed expert trajectories (real samples). Taking this idea one step further, apprenticeship learning or imitation learning (Abbeel & Ng, 2004; Ziebart et al., 2008) aims at learning a policy (analogous to the generator) using the reward signals recovered by IRL. Notably, Ho & Ermon draw a connection between imitation learning and GAN by showing that the GAN formulation can be derived by imposing a specific regularization on the reward function. Also, under a special case of their formulation, Ho & Ermon provide a duality-based interpretation of the problem, which inspires our theoretical analysis. However, as the focus of (Ho & Ermon, 2016) is only on the policy, the authors explicitly propose to bypass the intermediate IRL step, and thus provide no analysis of the learned reward function.
The GAN models most closely related to our proposed framework are energy-based GAN models of Zhao et al. (2016) and Kim & Bengio (2016). In the next section, We show how one can derive both of these approaches from different assumptions regarding regularization of the generative model.
3 ALTERNATIVE FORMULATION OF ADVERSARIAL TRAINING
3.1 BACKGROUND
Before presenting the proposed formulation, we first state some basic assumptions required by the analysis, and introduce notations used throughout the paper.
Following the original work on GANs (Goodfellow et al., 2014), our analysis focuses on the nonparametric case, where all models are assumed to have infinite capacities. While many of the nonparametric intuitions can directly transfer to the parametric case, we will point out cases where this transfer fails. We assume a finite data space throughout the analysis, to avoid technical machinery out of the scope of this paper. Our results, however, can be extended to continuous data spaces, and our experiments are indeed performed on continuous data.
Let X be the data space under consideration, and P = {p | p(x) ≥ 0,∀x ∈ X , ∑ x∈X p(x) = 1} be the set of all proper distributions defined on X . Then, pdata ∈ P : X 7→ R and pgen ∈ P : X 7→ R will denote the true data distribution and the generator distribution. Ex∼p f(x) denotes the expectation of the quantity f(x) w.r.t. x drawn from p. Finally, the term “discriminator” will refer to any structure that provides training signals to the generator based on some measure of difference between the generator distribution and the real data distribution, which which includes but is not limited to f -divergence.
3.2 PROPOSED FORMULATION
In order to understand the motivation of the proposed approach, it is helpful to analyze the optimization dynamics near convergence in GANs first.
When the generator distribution matches the data distribution, the training signal (gradient) w.r.t. the discriminator vanishes. At this point, assume the discriminator still retains density information, and views some samples as more real and others as less. This discriminator will produce a training signal (gradient) w.r.t. the generator, pushing the generator to generate samples that appear more real to the discriminator. Critically, this training signal is the sole driver of the generator’s training. Hence, the generator distribution will diverge from the data distribution. In other words, as long as the discriminator retains relative density information, the generator distribution cannot stably match the data distribution. Thus, in order to keep the generator stationary as the data distribution, the discriminator must assign flat (exactly the same) density to all samples at the optimal.
From the analysis above, the fundamental difficulty is that the generator only receives a single training signal (gradient) from the discriminator, which it has to follow. To keep the generator stationary, this single training signal (gradient) must vanish, which requires a degenerate discriminator. In this work, we propose to tackle this single training signal constraint directly. Specifically, we introduce a novel adversarial learning formulation which incorporates an additional training signal to the generator, such that this additional signal can
• balance (cancel out) the discriminator signal at the optimum, so that the generator can stay stationary even if the discriminator assigns non-flat density to samples
• cooperate with the discriminator signal to make sure the generator converges to the data distribution, and the discriminator retains the correct relative density information
The proposed formulation can be written as the following minimax training objective,
max c min pgen∈P E x∼pgen
[ c(x) ] − E x∼pdata [ c(x) ] +K(pgen), (1)
where c(x) : X 7→ R is the discriminator that assigns each data point an unbounded scalar cost, and K(pgen) : P 7→ R is some (functionally) differentiable, convex function of pgen. Compared to the original GAN, despite the similar minimax surface form, the proposed fomulation has two crucial distinctions.
Firstly, while the GAN discriminator tries to distinguish “fake” samples from real ones using binary classification, the proposed discriminator achieves that by assigning lower cost to real samples and higher cost to “fake” one. This distinction can be seen from the first two terms of Eqn. (1), where the discriminator c(x) is trained to widen the expected cost gap between “fake” and real samples, while the generator is adversarially trained to minimize it. In addition to the different adversarial mechanism, a calibrating term K(pgen) is introduced to provide a countervailing source of training signal for pgen as we motivated above. For now, the form of K(pgen) has not been specified. But as we will see later, its choice will directly decide the form of the optimal discriminator c∗(x).
With the specific optimization objective, we next provide theoretical characterization of both the generator and the discriminator at the global optimum.
Define L(pgen, c) = Ex∼pgen [ c(x) ] −Ex∼pdata [ c(x) ] +K(pgen), then L(pgen, c) is the Lagrange dual function of the following optimization problem
min pgen∈P K(pgen)
s.t. pgen(x)− pdata(x) = 0,∀x ∈ X (2)
where c(x),∀x appears in L(pgen, c) as the dual variables introduced for the equality constraints. This duality relationship has been observed previously in (Ho & Ermon, 2016, equation (7)) under the adversarial imitation learning setting. However, in their case, the focus was fully on the generator side (induced policy), and no analysis was provided for the discriminator (reward function).
In order to characterize c∗, we first expand the set constraint on pgen into explicit equality and inequality constraints:
min pgen K(pgen)
s.t. pgen(x)− pdata(x) = 0,∀x − pgen(x) ≤ 0,∀x∑ x∈X pgen(x)− 1 = 0.
(3)
Notice thatK(pgen) is a convex function of pgen(x) by definition, and both the equality and inequality constraints are affine functions of pgen(x). Thus, problem (2) is a convex optimization problem. What’s more, since (i) domK is open, and (ii) there exists a feasible solution pgen = pdata to (3), by the refined Slater’s condition (Boyd & Vandenberghe, 2004, page 226), we can further verify that strong duality holds for (3). With strong duality, a typical approach to characterizing the optimal solution is to apply the Karush-Kuhn-Tucker (KKT) conditions, which gives rise to this theorem:
Proposition 3.1. By the KKT conditions of the convex problem (3), at the global optimum, the optimal generator distribution p∗gen matches the true data distribution pdata, and the optimal discriminator c∗(x) has the following form:
c∗(x) = − ∂K(pgen)
∂pgen(x) ∣∣∣∣ pgen=pdata − λ∗ + µ∗(x),∀x ∈ X ,
where µ∗(x) = { 0, pdata(x) > 0
ux, pdata(x) = 0 ,
λ∗ ∈ R, is an under-determined real number independent of x, ux ∈ R+, is an under-determined non-negative real number.
(4)
The detailed proof of proposition 3.1 is provided in appendix A.1. From (4), we can see the exact form of the optimal discriminator depends on the term K(pgen), or more specifically its gradient. But, before we instantiateK(pgen) with specific choices and show the corresponding forms of c∗(x), we first discuss some general properties of c∗(x) that do not depend on the choice of K.
Weak Support Discriminator. As part of the optimal discriminator function, the term µ∗(x) plays the role of support discriminator. That is, it tries to distinguish the support of the data distribution, i.e. SUPP(pdata) = {x ∈ X | pdata(x) > 0}, from its complement set with zeroprobability, i.e. SUPP(pdata){ = {x ∈ X | pdata(x) = 0}. Specifically, for any x ∈ SUPP(pdata) and x′ ∈ SUPP(pdata){, it is guaranteed that µ∗(x) ≤ µ∗(x′). However, because µ∗(·) is underdetermined, there is nothing preventing the inequality from degenerating into an equality. Therefore, we name it the weak support discriminator. But, in all cases, µ∗(·) assigns zero cost to all data points within the support. As a result, it does not possess any fine-grained density information inside of the data support. It is worth pointing out that, in the parametric case, because of the smoothness and the generalization properties of the parametric model, the learned discriminator may generalize beyond the data support.
Global Bias. In (4), the term λ∗ is a scalar value shared for all x. As a result, it does not affect the relative cost among data points, and only serves as a global bias for the discriminator function.
Having discussed general properties, we now consider some specific cases of the convex function K, and analyze the resulting optimal discriminator c∗(x) in detail.
1. First, let us consider the case where K is the negative entropy of the generator distribution, i.e. K(pgen) = −H(pgen). Taking the derivative of the negative entropy w.r.t. pgen(x), we have
c∗ent(x) = − log pdata(x)− 1− λ∗ + µ∗(x),∀x ∈ X , (5) where µ∗(x) and λ∗ have the same definitions as in (4). Up to a constant, this form of c∗ent(x) is exactly the energy function of the data distribution pdata(x). This elegant result has deep connections to several existing formulations, which include max-entropy imitation learning (Ziebart et al., 2008) and the directed-generator-trained energybased model (Kim & Bengio, 2016). The core difference is that these previous formulations are originally derived from maximum-likelihood estimation, and thus the minimax optimization is only implicit. In contrast, with an explicit minimax formulation we can develop a better understanding of the induced solution. For example, the global bias λ∗ suggests that there exists more than one stable equilibrium the optimal discriminator can actually reach. Further, µ∗(x) can be understood as a support discriminator that poses extra cost on generator samples which fall in zero-probability regions of data space.
2. When K(pgen) = 12 ∑ x∈X pgen(x) 2 = 12‖pgen‖ 2 2, which can be understood as posing `2 regular-
ization on pgen, we have ∂K(pgen) ∂pgen(x) ∣∣ pgen=pdata = pdata(x), and it follows
c∗`2(x) = −pdata(x)− λ ∗ + µ∗(x),∀x ∈ X , (6)
with µ∗(x), λ∗ similarly defined as in (4). Surprisingly, the result suggests that the optimal discriminator c∗`2(x) directly recovers the negative probability −pdata(x), shifted by a constant. Thus, similar to the entropy solution (5), it fully retains the relative density information of data points within the support.
However, because of the under-determined term µ∗(x), we cannot recover the distribution density pdata exactly from either c∗`2 or c ∗ ent if the data support is finite. Whether this ambiguity can
be resolved is beyond the scope of this paper, but poses an interesting research problem. 3. Finally, let’s consider consider a degenerate case, where K(pgen) is a constant. That is, we dont
provide any additional training signal for pgen at all. With K(pgen) = const, we simply have
c∗cst(x) = λ ∗ + µ∗(x),∀x ∈ X , (7)
whose discriminative power is fully controlled by the weak support discriminator µ∗(x). Thus, it follows that c∗cst(x) won’t be able to discriminate data points within the support of pdata, and its power to distinguish data from SUPP(pdata) and SUPP(pdata){ is weak. This closely matches the intuitive argument in the beginning of this section. Note that when K(pgen) is a constant, the objective function (1) simplifies to:
max c min pgen∈P E x∼pgen
[ c(x) ] − E x∼pdata [ c(x) ] , (8)
which is very similar to the EBGAN objective (Zhao et al., 2016, equation (2) and (4)). As we show in appendix A.2, compared to the objective in (8), the EBGAN objective puts extra constraints on the allowed discriminator function. In spite of that, the EBGAN objective suffers from the single-training-signal problem and does not guarantee that the discriminator will recover the real energy function (see appendix A.2 for detailed analysis).
As we finish the theoretical analysis of the proposed formulation, we want to point out that simply adding the same term K(pgen) to the original GAN formulation will not lead to both a generator that matches the data distribution, and a discriminator that retains the density information (see appendix A.3 for detailed analysis).
4 PARAMETRIC INSTANTIATION WITH ENTROPY APPROXIMATION
While the discussion in previous sections focused on the non-parametric case, in practice we are limited to a finite amount of data, and the actual problem involves high dimensional continuous spaces. Thus, we resort to parametric representations for both the generator and the discriminator. In order to train the generator using standard back-propagation, we do not parametrize the generator distribution directly. Instead, we parametrize a directed generator network that transforms random noise z ∼ pz(z) to samples from a continuous data space Rn. Consequently, we don’t have analytical access to the generator distribution, which is defined implicitly by the generator network’s noise→data mapping. However, the regularization term K(pgen) in the training objective (1) requires the generator distribution. Faced with this problem, we focus on the max-entropy formulation, and exploit two different approximations of the regularization term K(pgen) = −H(pgen).
4.1 NEAREST-NEIGHBOR ENTROPY GRADIENT APPROXIMATION
The first proposed solution is built upon an intuitive interpretation of the entropy gradient. Firstly, since we construct pgen by applying a deterministic, differentiable transform gθ to samples z from a fixed distribution pz , we can write the gradient of H(pgen) with respect to the generator parameters θ as follows:
−∇θH(pgen) = Ez∼pz [∇θ log pgen(gθ(z))] = Ez∼pz [ ∂gθ(z)
∂θ
∂ log pgen(gθ(z))
∂gθ(z)
] , (9)
where the first equality relies on the “reparametrization trick”. Equation 9 implies that, if we can compute the gradient of the generator log-density log pgen(x) w.r.t. any x = gθ(z), then we can directly construct the Monte-Carlo estimation of the entropy gradient ∇θH(pgen) using samples from the generator.
Intuitively, for any generated data x = gθ(z), the term ∂ log pgen(x)
∂x essentially describes the direction of local change in the sample space that will increase the log-density. Motivated by this intuition, we propose to form a local Gaussian approximation pigen of pgen around each point xi in a batch of samples {x1, ..., xn} from the generator, and then compute the gradient ∂ log pgen(xi)∂xi based on the
Gaussian approximation. Specifically, each local Gaussian approximation pigen is formed by finding the k nearest neighbors of xi in the batch {x1, ..., xn}, and then placing an isotropic Gaussian distribution at their mean (i.e. maximimum likelihood). Based on the isotropic Gaussian approximation, the resulting gradient has the following form
∂ log pgen(xi)
∂xi ≈ µi − xi, where µi =
1
k ∑ x′∈KNN(xi) x′ is the mean of the Gaussian (10)
Finally, note the scale of this gradient approximation may not be reliable. To fix this problem, we normalize the approximated gradient into unit norm, and use a single hyper-parameter to model the scale for all x, leading to the following entropy gradient approximation
−∇θH(pgen) ≈ α 1
k ∑ xi=gθ(zi) µi − xi ‖µi − xi‖2
(11)
where α is the hyper-parameter and µi is defined as in equation (10).
An obvious weakness of this approximation is that it relies on Euclidean distance to find the k nearest neighbors. However, Euclidean distance is usually not the proper metric to use when the effective dimension is very high. As the problem is highly challenging, we leave it for future work.
4.2 VARIATIONAL LOWER BOUND ON THE ENTROPY
Another approach we consider relies on defining and maximizing a variational lower bound on the entropy H(pgen(x)) of the generator distribution. We can define the joint distribution over observed data and the noise variables as pgen(x, z) = pgen(x | z)pgen(z), where simply pgen(z) = pz(z) is a fixed prior. Using the joint, we can also define the marginal pgen(x) and the posterior pgen(z | x). We can also write the mutual information between the observed data and noise variables as:
I(pgen(x); pgen(z)) = H(pgen(x))−H(pgen(x | z)) = H(pgen(z))−H(pgen(z | x)),
(12)
where H(pgen(. | .)) denotes the conditional entropy. By reorganizing terms in this definition, we can write the entropy H(pgen(x)) as:
H(pgen(x)) = H(pgen(z))−H(pgen(z | x)) +H(pgen(x | z)) (13)
We can think of pgen(x | z) as a peaked Gaussian with a fixed, diagonal covariance, and hence its conditional entropy is constant and can be dropped. Furthermore, H(pgen(z)) is also assumed to be fixed a priori. Hence, we can maximize H(pgen(x)) by minimizing the conditional entropy:
H(pgen(z | x)) = Ex∼pgen(x) [ Ez∼pgen(z|x) [− log pgen(z | x)] ] (14)
Optimizing this term is still problematic, because (i) we do not have access to the posterior pgen(z | x), and (ii) we cannot sample from it. Therefore, we instead minimize a variational upper bound defined by an approximate posterior qgen(z | x):
H(pgen(z | x)) = Ex∼pgen(x) [ Ez∼pgen(z|x) [− log qgen(z | x)]− KL(pgen(z | x)‖qgen(z | x)) ] ≤ Ex∼pgen(x) [ Ez∼pgen(z|x) [− log qgen(z | x)]
] = U(qgen).
(15)
We can also rewrite the variational upper bound as: U(qgen) = Ex,z∼pgen(x,z) [− log qgen(z | x)] = Ez∼pgen(z) [ Ex∼pgen(x|z) [− log qgen(z | x)] ] , (16)
which can be optimized efficiently with standard back-propagation and Monte Carlo integration of the relevant expectations based on independent samples drawn from the joint pgen(x, z). By minimizing this upper bound on the conditional entropy H(pgen(z | x)), we are effectively maximizing a variational lower bound on the entropy H(pgen(x)).
5 EXPERIMENTS
In this section, we verify our theoretical results empirically on several synthetic and real datasets. In particular, we evaluate whether the discriminator obtained from the entropy-regularized adversarial training can capture the density information (in the form of energy), while making sure the generator distribution matches the data distribution. For convenience, we refer to the obtained models as EGAN-Ent. Our experimental setting follows closely recommendations from Radford et al. (2015), except in Sec. 5.1 where we use fully-connected models (see appendix B.1 for details). 1
5.1 SYNTHETIC LOW-DIMENSIONAL DATA
First, we consider three synthetic datasets in 2-dimensional space, which are drawn from the following distributions: (i) Mixture of 4 Gaussians with equal mixture weights, (ii) Mixture of 200 Gaussians arranged as two spirals (100 components each spiral), and (iii) Mixture of 2 Gaussians with highly biased mixture weights, P (c1) = 0.9, P (c2) = 0.1. We visualize the ground-truth energy of these distributions along with 100K training samples in Figure 1. Since the data lies in 2-dimensional space, we can easily visualize both the learned generator (by drawing samples) and the discriminator for direct comparison and evaluation. We evaluate here our EGAN-Ent model using both approximations: the nearest-neighbor based approximation (EGAN-Ent-NN) and the variational-inference based approximation (EGAN-Ent-VI), and compare them with two baselines: the original GAN and the energy based GAN with no regularization (EGAN-Const).
Experiment results are summarized in Figure 2 for baseline models, and Figure 3 for the proposed models. As we can see, all four models can generate perfect samples. However, for the discriminator, both GAN and EGAN-Const lead to degenerate solution, assigning flat energy inside the empirical data support. In comparison, EGAN-Ent-VI and EGAN-Ent-NN clearly capture the density information, though to different degrees. Specifically, on the equally weighted Gaussian mixture and the two-spiral mixture datasets, EGAN-Ent-NN tends to give more accurate and fine-grained solutions compared to EGAN-Ent-VI. However, on the biased weighted Gaussian mixture dataset, EGAN-Ent-VI actually fails to captures the correct mixture weights of the two modes, incorrectly assigning lower energy to the mode with lower probability (smaller weight). In contrast, EGANEnt-NN perfectly captures the bias in mixture weight, and obtains a contour very close to the ground truth.
To better quantify these differences, we present detailed comparison based on KL divergence in appendix B.2. What’s more, the performance difference between EGAN-Ent-VI and EGAN-Ent-NN on biased Gaussian mixture reveals the limitations of the variational inference based approximation, i.e. providing inaccurate gradients. Due to space consideratiosn, we refer interested readers to the appendix B.3 for a detailed discussion.
5.2 RANKING NIST DIGITS
In this experiment, we verify that the results in synthetic datasets can translate into data with higher dimensions. While visualizing the learned energy function is not feasible in high-dimensional space, we can verify whether the learned energy function learns relative densities by inspecting the ranking of samples according to their assigned energies. We train on 28× 28 images of a single handwritten
1For more details, please refer to https://github.com/zihangdai/cegan_iclr2017.
digit from the NIST dataset. 2 We compare the ability of EGAN-Ent-NN with both EGAN-Const and GAN on ranking a set of 1,000 images, half of which are generated samples and the rest are real test images. Figures 4 and 5 show the top-100 and bottom-100 ranked images respectively for each model, after training them on digit 1. We also show in Figure 7 the mean of all training samples, so we can get a sense of what is the most common style (highest density) of digit 1 in NIST. We can notice that all of the top-ranked images by EGAN-Ent-NN look similar to the mean sample. In addition, the lowest-ranked images are clearly different from the mean image, with either high (clockwise or counter-clockwise) rotation degrees from the mean, or an extreme thickness level. We do not see such clear distinction in other models. We provide in the appendix B.4 the ranking of the full set of images.
5.3 SAMPLE QUALITY ON NATURAL IMAGE DATASETS
In this last set of experiments, we evaluate the visual quality of samples generated by our model in two datasets of natural images, namely CIFAR-10 and CelebA. We employ here the variationalbased approximation for entropy regularization, which can scale well to high-dimensional data. Figure 6 shows samples generated by EGAN-Ent-VI. We can see that despite the noisy gradients provided by the variational approximation, our model is able to generate high-quality samples.
2https://www.nist.gov/srd/nist-special-database-19, which is an extended version of MNIST with an average of over 74K examples per digit.
We futher validate the quality of our model’s samples on CIFAR-10 using the Inception score proposed by (Salimans et al., 2016) 3. Table 1 shows the scores of our EGAN-Ent-VI, the best GAN model from Salimans et al. (2016) which uses only unlabeled data, and an EGAN-Const model which has the same architecture as our model. We notice that even without employing suggested techniques in Salimans et al. (2016), energy-based models perform quite similarly to the GAN model. Furthermore, the fact that our model scores higher than EGAN-Const highlights the importance of entropy regularization in obtaining good quality samples.
6 CONCLUSION
In this paper we have addressed a fundamental limitation in adversarial learning approaches, which is their inability of providing sensible energy estimates for samples. We proposed a novel adversarial learning formulation which results in a discriminator function that recovers the true data energy. We provided a rigorous characterization of the learned discriminator in the non-parametric setting, and proposed two methods for instantiating it in the typical parametric setting. Our experimental results verify our theoretical analysis about the discriminator properties, and show that we can also obtain samples of state-of-the-art quality.
7 ACKNOWLEDGEMENTS
We would like to thank the developers of Theano (Theano Development Team, 2016) for developing such a powerful tool for scientific computing. Amjad Almahairi was supported by funding from Maluuba Research.
3Using the evaluation script released in https://github.com/openai/improved-gan/
Figure 7: mean digit
A SUPPLEMENTARY MATERIALS FOR SECTION 3
A.1 OPTIMAL DISCRIMINATOR FORM UNDER THE PROPOSED FORMULATION
Proof of proposition 3.1. Refining the Lagrange L(pgen, c) by introducing additional dual variables for the probability constraints (the second and third), the new Lagrange function has the form
L(pgen, c, µ, λ) = K(pgen)+ ∑ x∈X c(x) ( pgen(x)−pdata(x) ) − ∑ x∈X µ(x)pgen(x)+λ( ∑ x∈X pgen(x)−1) (17) where c(x) ∈ R,∀x, µ(x) ∈ R+,∀x, and λ ∈ R are the dual variables. The KKT conditions for the optimal primal and dual variables are as follows
∂K(pgen)
∂pgen(x) ∣∣∣∣ pgen=pdata + c∗(x)− µ∗(x) + λ∗ = 0, ∀x (stationarity)
µ∗(x)p∗gen(x) = 0, ∀x (complement slackness) µ∗(x) ≥ 0, ∀x (dual feasibility)
p∗gen(x) ≥ 0, p∗gen(x) = pdata(x), ∀x (primal feasibility)∑ x∈X pgen(x) = 1 (primal feasibility)
(18)
Rearranging the conditions above, we get p∗gen(x) = pdata(x),∀x ∈ X as well as equation (4), which concludes the proof.
A.2 OPTIMAL CONDITIONS OF EBGAN
In (Zhao et al., 2016), the training objectives of the generator and the discriminator cannot be written as a single minimax optimization problem since the margin structure is only applied to the objective of the discriminator. In addition, the discriminator is designed to produce the mean squared reconstruction error of an auto-encoder structure. This restricted the range of the discriminator output to be non-negative, which is equivalent to posing a set constraint on the discriminator under the non-parametric setting.
Thus, to characterize the optimal generator and discriminator, we adapt the same analyzing logic used in the proof sketch of the original GAN (Goodfellow et al., 2014). Specifically, given a specific generator distribution pgen, the optimal discriminator function given the generator distribution c∗(x; pgen) can be derived by examining the objective of the discriminator. Then, the conditional optimal discriminator function is substituted into the training objective of pgen, simplifying the “adversarial” training as a minimizing problem only w.r.t. pgen, which can be well analyzed.
Firstly, given any generator distribution pgen, the EBGAN training objective for the discriminator can be written as the following form
c∗(x; pgen) = argmax c∈C − E pgen max(0,m− c(x))− E pdata c(x)
= argmax c∈C E pgen min(0, c(x)−m)− E pdata
c(x) (19)
where C = {c : c(x) ≥ 0,∀x ∈ X} is the set of allowed non-negative discriminator functions. Note this set constraint comes from the fact the mean squared reconstruction error as discussed above.
Since the problem (19) is independent w.r.t. each x, the optimal solution can be easily derived as
c∗(x; pgen) =  0, pgen(x) < pdata(x) m, pgen(x) > pdata(x) αx, pgen(x) = pdata(x) > 0
βx, pgen(x) = pdata(x) = 0
(20)
where αx ∈ [0,m] is an under-determined number, a βx ∈ [0,∞) is another under-determined nonnegative real number, and the subscripts in m,αx, βx reflect that fact that these under-determined values can be distinct for different x.
This way, the overall training objective can be cast into a minimization problem w.r.t. pgen,
p∗gen = argmin pgen∈P E x∼pgen c∗(x; pgen)− E x∼pdata c∗(x; pgen)
= argmin pgen∈P ∑ x∈X [ pgen(x)− pdata(x) ] c∗(x; pgen)
(21)
where the second term of the first line is implicitly defined as the problem is an adversarial game between pgen and c. Proposition A.1. The global optimal of the EBGAN training objective is achieved if and only if pgen = pdata. At that point, c∗(x) is fully under-determined.
Proof. The proof is established by showing contradiction.
Firstly, assume the optimal p∗gen 6= pdata. Thus, there must exist a non-equal setX6= = {x | pdata(x) 6= p∗gen(x)}, which can be further splitted into two subsets, the greater-than set X> = {x | p∗gen(x) > pdata(x)}, and the less-than set X< = {x | p∗gen(x) < pdata(x)}. Similarly, we define the equal set X= = {x : p∗gen(x) = pdata(x)}. Obviously, X> ⋃ X< ⋃ X= = X .
Let L(pgen) = ∑ x∈X [ pgen(x) − pdata(x) ] c∗(x; pgen), substituting the results from equation (20) into (21), the L(pgen)∗ can be written as
L(p∗gen) = ∑
x∈X< ⋃ X< ⋃ X=
[ p∗gen(x)− pdata(x) ] c∗(x; p∗gen)
= ∑ x∈X< [ p∗gen(x)− pdata(x) ] c∗(x; p∗gen) + ∑ x∈X> [ p∗gen(x)− pdata(x) ] c∗(x; p∗gen)
= m ∑ x∈X> p∗gen(x)− pdata(x)
> 0
(22)
However, when p′gen = pdata, we have
L(p′gen) = 0 < L(p ∗ gen) (23)
which contradicts the optimal (miminum) assumption of p∗gen. Hence, the contradiction concludes that at the global optimal, p∗gen = pdata. By equation (20), it directly follows that c
∗(x; p∗gen) = αx, which completes the proof.
A.3 ANALYSIS OF ADDING ADDITIONAL TRAINING SIGNAL TO GAN FORMULATION
To show that simply adding the same training signal to GAN will not lead to the same result, it is more convenient to directly work with the formulation of f -GAN (Nowozin et al., 2016, equation (6)) family, which include the original GAN formulation as a special case.
Specifically, the general f -GAN formulation takes the following form
max c min pgen∈P E x∼pgen
[ f?(c(x)) ] − E x∼pdata [ c(x) ] , (24)
where the f?(·) denotes the convex conjugate (Boyd & Vandenberghe, 2004) of the f -divergence function. The optimal condition of the discriminator can be found by taking the variation w.r.t. c, which gives the optimal discriminator
c∗(x) = f ′( pdata(x)
pgen(x) ) (25)
where f ′(·) is the first-order derivative of f(·). Note that, even when we add an extra term L(pgen) to equation (24), since the term K(pgen) is a constant w.r.t. the discriminator, it does not change the result given by equation (25) about the optimal discriminator. As a consequence, for the optimal
discriminator to retain the density information, it effectively means pgen 6= pdata. Hence, there will be a contradiction if both c∗(x) retains the density information, and the generator matches the data distribution.
Intuitively, this problem roots in the fact that f -divergence is quite “rigid” in the sense that given the pgen(x) it only allows one fixed point for the discriminator. In comparison, the divergence used in our proposed formulation, which is the expected cost gap, is much more flexible. By the expected cost gap itself, i.e. without the K(pgen) term, the optimal discriminator is actually under-determined.
B SUPPLEMENTARY MATERIALS FOR SECTION 5
B.1 EXPERIMENT SETTING
Here, we specify the neural architectures used for experiements presented in Section 5.
Firstly, for the Egan-Ent-VI model, we parameterize the approximate posterior distribution qgen(z | x) with a diagonal Gaussian distribution, whose mean and covariance matrix are the output of a trainable inference network, i.e.
qgen(z | x) = N (µ, Iσ2) µ, log σ = f infer(x)
(26)
where f infer denotes the inference network, and I is the identity matrix. Note that the Inference Network only appears in the Egan-Ent-VI model.
For experiments with the synthetic datasets, the following fully-connected feed forward neural networks are employed
• Generator: FC(4,128)-BN-ReLU-FC(128,128)-BN-ReLU-FC(128,2) • Discriminator: FC(2,128)-ReLU-FC(128,128)-ReLU-FC(128,1) • Inference Net: FC(2,128)-ReLU-FC(128,128)-ReLU-FC(128,4*2)
where FC and BN denote fully-connected layer and batch normalization layer respectively. Note that since the input noise to the generator has dimension 4, the Inference Net output has dimension 4*2, where the first 4 elements correspond the inferred mean, and the last 4 elements correspond to the inferred diagonal covariance matrix in log scale.
For the handwritten digit experiment, we closely follow the DCGAN (Radford et al., 2015) architecture with the following configuration
• Generator: FC(10,512*7*7)-BN-ReLU-DC(512,256;4c2s)-BN-ReLU -DC(256,128;4c2s)-BN-ReLU-DC(128,1;3c1s)-Sigmoid
• Discriminator: CV(1,64;3c1s)-BN-LRec-CV(64,128;4c2s)-BN-LRec -CV(128,256;4c2s)-BN-LRec-FC(256*7*7,1)
• Inference Net: CV(1,64;3c1s)-BN-LRec-CV(64,128;4c2s)-BN-LRec -CV(128,256;4c2s)-BN-LRec-FC(256*7*7,10*2)
Here, LRec is the leaky rectified non-linearity recommended by Radford et al. (2015). In addition, CV(128,256,4c2s) denotes a convolutional layer with 128 input channels, 256 output channels, and kernel size 4 with stride 2. Similarly, DC(256,128,4c2s) denotes a corresponding transposed convolutional operation. Compared to the original DCGAN architecture, the discriminator under our formulation does not have the last sigmoid layer which squashes a scalar value into a probability in [0, 1].
For celebA experiment with 64× 64 color images, we use the following architecture
• Generator: FC(10,512*4*4)-BN-ReLU-DC(512,256;4c2s)-BN-ReLU-DC(256,128;4c2s) -BN-ReLU-DC(256,128;4c2s)-BN-ReLU-DC(128,3;4c2s)-Tanh
• Discriminator: CV(3,64;4c2s)-BN-LRec-CV(64,128;4c2s)-BN-LRec-CV(128,256;4c2s) -BN-LRec-CV(256,256;4c2s)-BN-LRec-FC(256*4*4,1)
• Inference Net: CV(3,64;4c2s)-BN-LRec-CV(64,128;4c2s)-BN-LRec-CV(128,256;4c2s) -BN-LRec-CV(256,256;4c2s)-BN-LRec-FC(256*4*4,10*2)
For Cifar10 experiment, where the image size is 32× 32, similar architecture is used
• Generator: FC(10,512*4*4)-BN-ReLU-DC(512,256;4c2s)-BN-ReLU-DC(256,128;3c1s) -BN-ReLU-DC(256,128;4c2s)-BN-ReLU-DC(128,3;4c2s)-Tanh
• Discriminator: CV(3,64;3c1s)-BN-LRec-CV(64,128;4c2s)-BN-LRec-CV(128,256;4c2s) -BN-LRec-CV(256,256;4c2s)-BN-LRec-FC(256*4*4,1)
• Inference Net: CV(3,64;3c1s)-BN-LRec-CV(64,128;4c2s)-BN-LRec-CV(128,256;4c2s) -BN-LRec-CV(256,256;4c2s)-BN-LRec-FC(256*4*4,10*2)
Given the chosen architectures, we follow Radford et al. (2015) and use Adam as the optimization algorithm. For more detailed hyper-parameters, please refer to the code.
B.2 QUANTITATIVE COMPARISON OF DIFFERENT MODELS
In order to quantify the quality of recovered distributions, we compute the pairwise KL divergence of the following four distributions:
• The real data distribution with analytic form, denoted as pdata • The empirical data distribution approximated from the 100K training data, denoted as pemp • The generator distribution approximated from 100K generated data, denoted as pgen • The discriminator distribution re-normalized from the learned energy, denoted as pdisc
Since the synthetic datasets are two dimensional, we approximate both the empirical data distribution and the generator distribution using the simple histogram estimation. Specifically, we divide the canvas into a 100-by-100 grid, and assign each sample into its nearest grid cell based on euclidean distance. Then, we normalize the number of samples in each cell into a proper distribution. When recovering the discriminator distribution from the learned energy, we assume that µ∗(x) = 0 (i.e. infinite data support), and discretize the distribution into the same grid cells
pdisc(x) = exp(−c∗(x))∑
x′∈Grid exp(−c∗(x′)) ,∀x ∈ Grid
Based on these approximation, Table 2 summarizes the results. For all measures related to the discriminator distribution, EGAN-Ent-VI and EGAN-Ent-NN significantly outperform the other two baseline models, which matches our visual assessment in Figure 2 and 3. Meanwhile, the generator distributions learned from our proposed framework also achieve relatively lower divergence to both the empirical data distribution and the true data distribution.
B.3 COMPARISON OF THE ENTROPY (GRADIENT) APPROXIMATION METHODS
In order to understand the performance difference between EGAN-Ent-VI and EGAN-Ent-NN, we analyze the quality of the entropy gradient approximation during training. To do that, we visualize some detailed training information in Figure 8.
As we can see in figure 8a, the viarational entropy gradient approximation w.r.t. samples is not accurate:
• It is inaccurate in terms of gradient direction. Ideally, the direction of the entropy gradient should be pointing from the center of its closest mode towards the surroundings, with
the direction orthogonal to the implicit contour in Fig. (1,2). However, the direction of gradients in the Fig. (2,3) does not match this. • It is inaccurate in magnitude. As we can see, the entropy approximation gradient (Fig.
(2,3)) has much larger norm than the discriminator gradient (Fig. (2,2)). As a result, the total gradient (Fig. (2,4)) is fully dominated by the entropy approximation gradient. Thus, it usually takes much longer for the generator to learn to generate rare samples, and the training also proceeds much slower compared to the nearest neighbor based approximation.
In comparison, the nearest neighbor based gradient approximation is much more accurate as shown in 8b. As a result, it leads to more accurate energy contour, as well as faster training. What’s more, from Figure 8b Fig. (2,4), we can see the entropy gradient does have the cancel-out effect on the discriminator gradient, which again matches our theory.
B.4 RANKING NIST DIGITS
Figure 9 shows the ranking of all 1000 generated and real images (from the test set) for three models: EGAN-Ent-NN, EGAN-Const, and GAN. We can clearly notice that in EGAN-Ent-NN the topranked digits look very similar to the mean digit. From the upper-left corner to the lower-right corner, the transition trend is: the rotation degree increases, and the digits become increasingly thick or thin compared to the mean. In addition, samples in the last few rows do diverge away from the mean image: either highly diagonal to the right or left, or have different shape: very thin or thick, or typewriter script. Other models are not able to achieve a similar clear distinction for high versus low probability images. Finally, we consistently observe the same trend in modeling other digits, which are not shown in this paper due to space constraint.
B.5 CLASSIFIER PERFORMANCE AS A PROXY MEASURE
As mentioned in Section 5, evaluating the proposed formulation quantitatively on high-dimensional data is extremely challenging. Here, in order to provide more quantitative intuitions on the learned discriminator at convergence, we adopt a proxy measure. Specifically, we take the last-layer activation of the converged discriminator network as fixed pretrained feature, and build a linear classifier upon it. Hypothetically, if the discriminator does not degenerate, the extracted last-layer feature should maintain more information about the data points, especially compared to features from degenerated discriminators. Following this idea, we first train EGAN-Ent-NN, EGAN-Const, and GAN on the MNIST till convergence, and then extract the last-layer activation from their discriminator networks as fixed feature input. Based on fixed feature, a randomly initialized linear classifier is trained to do classification on MNIST. Based on 10 runs (with different initialization) of each of the three models, the test classification performance is summarized in Table 3. For comparison purpose, we also include a baseline where the input features are extracted from a discriminator network with random weights.
Based on the proxy measure, EGAN-Ent-NN seems to maintain more information of data, which suggests that the discriminator from our proposed formulation is more informative. Despite the positive result, it is important to point out that maintaining information about categories does not necessarily mean maintaining information about the energy (density). Thus, this proxy measure should be understood cautiously.

The submission explores several alternatives to provide the generator function in generative adversarial training with additional gradient information. The exposition starts by describing a general formulation about how this additional gradient information (termed K(p_gen) could be added to the generative adversarial training objective function (Equation 1). Next, the authors prove that the shape of the optimal discriminator does indeed depend on the added gradient information (Proposition 3.1), which is unsurprising. Finally, the authors propose three particular alternatives to construct K(p_gen): the negative entropy of the generator distribution, the L2 norm of the generator distribution, and a constant function (which resembles the EBGAN objective of Zhao et al, 2016).
The exposition moves then to an experimental evaluation of the method, which sets K(p_gen) to be the approximate entropy of the generator distribution. At this point, my intuition is that the objective function under study is the vanilla GAN objective, plus a regularization term that encourages diversity (high entropy) in the generator distribution. The hope of the authors is that this regularization will transform the discriminator into an estimate of the energy landscape of the data distribution.
The experimental evaluation proceeds by 1) showing the contour plots of the obtained generator distribution for a 2D problem, 2) studying the generation diversity in MNIST digits, and 3) showing some samples for CIFAR-10 and CelebA. The 2D problem results are convincing, since one can clearly observe that the discriminator scores translate into unnormalized values of the density function. The MNIST results offer good intuition also: the more prototypical digits are assigned larger scores (unnormalized densities) by the discriminator, and the less prototypical digits are assigned smaller scores. The sample experiments from Section 5.3 are less convincing, since no samples from baseline models are provided for comparison.
To this end, I would recommend the authors to clarify three aspects. First, we have seen that entropy regularization leads to a discriminator that estimates the energy landscape of the data distribution. But, how does this regularization reshape the generator function? It would be nice to see the mean MNIST digit according to the generator, and some other statistics if possible. Second, how do the samples produced by the proposed methods compare (visually speaking) to the state-of-the art? Third, what are the *shortcomings* of this method versus vanilla GAN? Too much computational overhead? What are the qualitative and quantitative differences between the two entropy estimators proposed in the manuscript?
Overall, a clearly written paper. I vote for acceptance.
As an open question to the authors: What breakthroughs should we pursue to derive a GAN objective where the discriminator is an estimate of the data density function, after training?