MULTILAYER RECURRENT NETWORK MODELS OF PRI- MATE RETINAL GANGLION CELL RESPONSES
Developing accurate predictive models of sensory neurons is vital to understanding sensory processing and brain computations. The current standard approach to modeling neurons is to start with simple models and to incrementally add interpretable features. An alternative approach is to start with a more complex model that captures responses accurately, and then probe the fitted model structure to understand the neural computations. Here, we show that a multitask recurrent neural network (RNN) framework provides the flexibility necessary to model complex computations of neurons that cannot be captured by previous methods. Specifically, multilayer recurrent neural networks that share features across neurons outperform generalized linear models (GLMs) in predicting the spiking responses of parasol ganglion cells in the primate retina to natural images. The networks achieve good predictive performance given surprisingly small amounts of experimental training data. Additionally, we present a novel GLM-RNN hybrid model with separate spatial and temporal processing components which provides insights into the aspects of retinal processing better captured by the recurrent neural networks.
1 INTRODUCTION
Our understanding of sensory processing in the brain is most straightforwardly reflected in our ability to model the process by which stimuli presented at the sensory periphery are transformed into the spiking activity of populations of neurons. For decades, researchers have interrogated stimulus-response
∗These authors contributed equally.
neural properties using simplified targeted stimuli, such as bars, spots, or gratings. While these types of stimuli uncovered many interesting aspects of visual computation, they have several limitations (Barlow & Levick, 1965). These stimuli may not fully drive important components of neural response, and modeling efforts have often assumed a quasi-linear mapping from stimulus to firing rate. Subsequent efforts to characterize cells relied on white noise stimulation and building models through reverse correlation (de Boer R & Kuyper, 1968; Marmarelis & Naka, 1972; Chichilnisky, 2001). A standard model used to relate white noise to spiking responses is the linear-nonlinear-Poisson (LN) or generalized linear model (GLM) which consists of a spatiotemporal linear filtering of the stimulus followed by a nonlinearity and probabilistic spike generation (Chichilnisky, 2001; Simoncelli et al., 2004; Schwartz et al., 2006). Although this family of models have advanced our understanding, they do not optimally capture neural responses, especially to natural scenes which can lead to more complex responses than white noise stimuli (David et al., 2004). Even in the retina, early in the visual processing stream, these commonly-used models capture retinal ganglion cell (RGC) responses to natural stimuli less accurately than to white noise (Heitman et al., 2016).
Recently, deep neural networks have been used to dramatically improve performance on a diverse array of machine learning tasks (Krizhevsky et al., 2012; LeCun et al., 2015). Furthermore, these networks bear a loose resemblance to real neural networks, and provide a sufficiently rich model class that can still be roughly constrained to match the biological architecture (Kriegeskorte, 2015). Most previous research at this intersection of neuroscience and artificial neural networks has focused on training networks on a certain task, such as object recognition, and then comparing the computations performed in different layers of the artificial network to those performed by real neurons (Yamins et al., 2014). Here we take a different approach: we fit multilayer models directly to the spiking responses of neurons, an approach that has not been explored in detail (but see (McIntosh et al., 2016) for some recent independent parallel developments).
2 APPROACH
We fit a range of models, detailed below, to spiking responses of primate RGCs. Our baseline comparisons are the GLM architectures that have been widely used to construct previous neural models (Pillow et al., 2008), though here we focus on individual neuronal responses (we leave modeling of correlations between neurons for future work). We focused on RNNs as a flexible framework in which to model more complex temporal and spatial nonlinearities. We also explored a number of network architectures involving features or weights shared across observed neurons. Given the complexity of the network architectures, we reasoned that sharing statistical strength across neurons by learning a shared feature space might improve predictive performance. This is conceptually a form of multitask learning - we are using a shared representation to achieve better generalization (Baxter, 2000). Motivated by previous research showing significant differences in the processing properties of the two cell types examined, ON and OFF parasol retinal ganglion cells, we fit separate models for each of these cell types (Chichilnisky & Kalmar, 2002).
3 METHODS
3.1 DATA COLLECTION
We fit spiking responses of OFF and ON parasol retinal ganglion cells to natural scenes. Recordings were performed on isolated retina using a large-scale multi-electrode recording system (Litke et al., 2004; Frechette et al., 2005; Field et al., 2007). A standard spike sorting algorithm was used to identify spikes from different cells from the voltage signals on each electrode during visual stimulation (Litke et al., 2004). We focus on two separate experiments (the same experimental procedure in two separate retinas) here; analyses of other datasets yielded similar results. Models were fit separately for the two experiments due to animal to animal variability in cell properties, such as receptive field size and firing rate. Almost all spike sorted cells were used for training (exp 1 = 118 OFF cells, 66 ON cells; exp 2 = 142 OFF cells, 103 ON cells): two cells were removed due to data quality issues (see sec 3.3). Performance metrics in this paper are reported for the same subset of cells used in a previous study (Heitman et al., 2016). These cells passed passed a manual screen for spike sorting accuracy, demonstrated stable light responses, and met a convergence criteria in prior linear-nonlinear modeling (exp 1 = 10 OFF cells, 18 ON cells; exp 2 = 65 OFF cells, 14 ON cells). The naturalistic movie
stimulus consisted of images from the Van Hateren database shown for one second each, with spatial jitter based on eye movements during fixation by awake macaque monkeys (Z.M. Hafed and R.J. Krauzlis, personal communication), (van Hateren & van der Schaaf, 1998). An example stimulus can be found at https://youtu.be/sG_18Uz_6OE. 59 distinct natural scenes movies of length one minute (the training data) were interleaved with 59 repetitions of a 30 second movie (the test data). Interleaving ensured that the test movie repetitions spanned the same period of time as the training data and therefore experienced the same range of experimental conditions (in case of neural response drifts over time). The first 4 movies shown (2 training movies and 2 repetitions of the test movie) were excluded to avoid initial transients. Test metrics are reported for the last 29 seconds of the 30 second test movie for the same reason. For further details on the experimental set-up, data preprocessing, and visual stimulation, see Heitman et al. (2016).
3.2 MODEL TRAINING
All models were implemented in Theano and trained on a combination of CPUs and GPUs (Theano Development Team, 2016). Training was performed using the Adam optimizer on the mean squared error (MSE) between predicted firing rate and true spikes (Kingma & Ba, 2014). We also experimented with optimizing a Poisson likelihood; this led to qualitatively similar results but occasionally less stable fits, so we focus on the MSE results here. All recurrent dynamics and temporal filters operated on time bins of 8.33 ms (the frame rate of the movie). Spike history terms and performance metrics were calculated for 0.833 ms bins. We used the same split of training and validation data for both experiments: 104 thirty-second movies as training data and 10 thirty-second movies as a held-out validation set.
During training, the performance on the held-out validation set is checked after every pass through the training data. After each iteration through the training data, if the model exhibits significantly better validation performance than our previous best, we reset the minimum number of iterations to be twice the current iteration number. If we make it through those iterations without another significant improvement, we stop. We train for a maximum of 150 epochs, where we define one epoch as one pass through all the training data. The model with the best validation performance is saved and used to assess test performance. All models with shared parameters were trained on a combined MSE over
all neurons and the parameters picked were those which minimized validation MSE for all neurons. For individual LNs/GLMs/RNNs, the validation MSE was minimized for each neuron separately.
3.3 RECEPTIVE FIELD CENTER ESTIMATION
In all models used in this paper, we estimate the receptive field (RF) center of each neuron in order to identify the appropriate portion of the image to use as input. We calculate a 250 ms long spike triggered average (STA) using reverse correlation of the neuron’s spikes with a white noise stimulus. We reduce the noise in this STA by using a rank 1 approximation (singular value decomposition followed by reconstruction using the primary temporal and spatial components). We then smooth each frame of the STA via convolution with a Gaussian spatial filter. The center location is defined as the pixel location that has the maximum absolute magnitude over time. The center locations were visually assessed to check accuracy of the algorithm. Rare cases where the algorithm failed to identify the correct center indicated neurons that responded to very little of the image as their receptive field was more than half-way displaced out of the image. These two neurons (two Exp 1 ON cells) were removed from further analysis. If the receptive field center is close to the edge of the image, the image patch is padded with the average training stimulus value.
3.4 PERFORMANCE EVALUATION
To quantitatively evaluate the accuracy of model spike predictions, we used the fraction of explainable variance, which has been described in previous literature (Heitman et al., 2016). Average firing rates over time are obtained after generating spikes from the model in 0.833 ms bins and smoothing with a Gaussian temporal filter (SD=10ms). The fraction of variance is computed as
F (r, rs) = 1− ∑
t(r(t)− rs(t))2∑ t(r(t)− µ)2
(1)
where r(t) is the smoothed recorded firing rate, rs(t) is the smoothed predicted firing rate, and µ is the average recorded rate. Finally, to account for the reproducibility of responses over repeated trials, we normalize by the fraction of variance captured by using the average firing rate on the odd (ro) trials of the repeated test movie to predict responses on the even (re) trials:
FV = F (r, rs)
F (re, ro) . (2)
4 MODEL ANALYSIS
4.1 NETWORK ARCHITECTURES
Individual LNs and GLMs: The linear-nonlinear model (LN) consists of a spatiotemporal filtering of the 31x31x30 movie patch (Xt, width by height by time) surrounding the estimated center of the neuron’s receptive field plus a bias term (b), followed by a sigmoid nonlinearity (f ), and Poisson spike generation to produce the responses rt. The generalized linear model (GLM), given by
rt ∼ Poiss [ f ( ~wTs (Xt ~wt) + b+ ∑ i hirt−i )] , (3)
has the same architecture with the addition of a post-spike history filter h before the nonlinearity f (Pillow et al., 2008). We used a rank 1 approximation of the full spatiotemporal filter (higher rank models did not significantly improve fits on a subset of examined neurons), resulting in a vectorized 31x31 spatial filter (~ws) and a 30 bin temporal filter (~wt) which spans 250 ms (Heitman et al., 2016). The post-spike history filter consists of a weighted sum of a basis of 20 raised cosines spanning approximately 100 ms (Pillow et al., 2008). The models with spike history were fit by initializing with the model fit without spike history. The filter either operates on the recorded spikes (training and validation) or the spikes generated by the model (testing). The nonlinearity is the logistic sigmoid: f = L/(1 + exp(−x)), which has been shown to improve fitting over an exponential nonlinearity for modeling RGC responses to natural scenes (Heitman et al., 2016).
Shared LN: In this model, the architecture is similar to the individual LNs but all cells of a given type (OFF or ON) share the same temporal and spatial filters (Figure 1A; note that the spatial filters are displaced to the RF center of each individual RGC). All other parameters are individually tuned for each observed neuron. There is an additional gain term that weights the output of the filtering individually for each observed neuron.
Two-layer RNN, 50 units: In this architecture, there are two recurrent neural network (RNN) layers between the image patch and Poisson neural unit:
~h (1) j,t = max(0, U1~sj,t + V1 ~h (1) j,t−1 + ~c) (4)
~h (2) j,t = max(0, U2 ~h (1) j,t + V2 ~h (2) j,t−1 + ~d) (5) rj,t ∼ Poiss [ f(~wTj ~h (2) j,t + bj) ] . (6)
The activity of the 50 units in the first RNN layer at time t is given by ~h(1)j,t in Eqn. 4. These units are rectified linear, and receive input from the vectorized 31x31 image patch surrounding the center of neuron j’s receptive field, ~sj,t, with weights U1, along with input from the other units in the layer with weights V1 and a bias ~c. The output of the first RNN is then fed into a second RNN with similar architecture. The firing rate for each observed neuron in the final layer is then given by Eqn. 6, and is a weighted sum of the recurrent units plus a bias bj , followed by a softplus nonlinearity f = log(1 + exp(−x)). Note that all parameters are shared across neurons except for the weights to the final layer and the final bias terms (~wj and bj).
GLM-RNN Hybrid: The GLM-RNN hybrid model consists of a spatial filter followed by a two-layer RNN. The architecture resembles that of the full two-layer RNN with 50 units, except the input to the first layer is a scalar (post multiplication with the spatial filter) at each time step instead of the full image patch; thus the RNN in this model is responsible for shaping the temporal properties of the output, but does not affect spatial processing after the first linear spatial filtering stage. All weights
are shared across neurons except for weights to the final layer (~wj) and the final bias terms (bj):
yj,t = ~w T s ~sj,t (7)
~h (1) j,t = max(0, ~u1yj,t + V1 ~h (1) j,t−1 + ~c) (8)
~h (2) j,t = max(0, U2h (1) j,t + V2 ~h (2) j,t−1 + ~d) (9) rj,t ∼ Poiss [ f(~wTj ~h (2) t + bj) ] . (10)
4.2 MODEL PERFORMANCE
RNNs of varying architectures consistently outperformed LNs and GLMs in predicting neural spiking responses to a novel natural scene movie for both OFF and ON parasol retinal ganglion cells in both experiments (Figure 2). A shared two-layer recurrent network consistently captures around 80% of the explainable variance across experiments and cell types. Other recurrent architectures (1-3 layer RNNs and a 2 layer LSTM) led to similar levels of performance (Supplementary Figure 6). The increase in performance according to the fraction of explainable variance metric was not an average effect: almost all neurons were significantly better predicted by the RNN (Figure 2B). A 2 layer RNN model with additional trained spike history filters outperformed GLMs and LNs according to a normalized log likelihood metric (Supplementary Figure 7).
Inspection of the mean predicted firing rate traces for LNs and RNNs in Figure 3 reveals that the recurrent network seems to be capturing the timing of firing more precisely. The LN often predicts a general increase in firing rate at the correct times, but the RNN captures the sudden increase in firing rate followed by decay which often occurs when the image changes. On the other hand, the LN models sometimes predict modest increases or decreases in firing rate that the recurrent nets miss.
Understanding why the recurrent models improve performance is a challenging task due to the black-box nature of deep networks. The first layer filters (U1, from image patches to recurrent units) have an interpretable structure resembling traditional receptive fields expected in the retina
(Supplementary Figure 8). However, the computations performed by the recurrent units are difficult to tease apart, because the weights are less interpretable. Thus, instead of attempting a mechanistic explanation of the internals of the RNN, we focused on what additional captured information resulted in the improved RNN performance.
One possibility is that capturing nonlinear effects in parts of the image far from the receptive field center improved predictions (McIlwain, 1964; Passaglia et al., 2009). We restricted the size of the image patch surrounding each receptive field center from 31x31 to 15x15 (Supplementary Figure 9). Shared RNNs trained on the smaller image patch size did as well, or better, than those trained on the larger patch across almost all combinations of cell type and experiment. (We see a similar small improvement when training the LN models on the small patch.) Thus we concluded that long-range nonlinear spatial interactions do not contribute to the increased performance produced by the RNNs.
We also investigated whether nonlinear spatial interactions or nonlinear temporal processing primarily contributed to better predictions. To accomplish this, we constructed a GLM-RNN hybrid, described previously, in which a single spatial filter precedes a two-layer RNN - effectively allowing only temporal nonlinearities to be captured. This model improved prediction over the LNs and GLMs but did not reach full RNN performance. The amount by which this model closed the gap differed for different experiments and cell types. We quantified this by computing the difference between multitask RNN and multitask LN performance for each neuron and the difference between multitask hybrid and multitask LN performance. We divide the latter by the former (on a cell-by-cell basis) to obtain the ratios summarized in Figure 2C. The hybrid model closed greater than half of the gap on average between multitask LN and RNN performance, indicating that the richer temporal dynamics of the RNN model account for a large part of the difference between RNN and LN performance, though spatial nonlinearities play a role too.
5 MODEST TRAINING DATA LENGTH SUFFICES FOR GOOD PERFORMANCE
Deep networks can be complex and often require large amounts of data to adequately train: convolutional neural networks used for object recognition are trained on over a million images (Krizhevsky et al., 2012). Standard neuroscience experiments yield limited data sets, so it is crucial to assess whether we have enough data to adequately fit our network architectures. We trained the RNN on varying amounts of data, and ran several different iterations of the network to explore variation over random initializations and randomly chosen training sets. These results are shown for both ON and OFF cells in Figure 4. Surprisingly small amounts of training data resulted in good predictive abilities. For larger amounts of training data, different iterations resulted in very similar mean fraction of variance values, indicating fairly robust fitting in these models. See Supplementary Figure 10 for further details.
6 BENEFITS OF MULTITASK FRAMEWORK
We investigated whether the multitask framework with shared parameters across neurons actually helps to improve predictive performance with reasonable amounts of experimental data. First, we quantified the benefits of parameter-sharing in the simple LN model. This is a highly constrained
framework: every cell has the same spatial and temporal filter. The shared LN does not improve performance for most neurons (Figure 5A).
We expected the multitask framework to be more helpful applied to the RNN model because in this case we are sharing features but not all parameters across neurons. Indeed, the multitask RNN consistently outperformed RNNs trained individually on single neurons (Figure 5B); individuallytrained RNNs also had much more variable losses than did the multitask-trained RNNs. In a realistic experimental setting with limited data, the multitask framework is a useful way to leverage all of the data collected for all neurons.
7 CONCLUSION
Using retinal neurons responding to natural scenes as an example, we showed that: using deep networks to model neural spiking responses can significantly improve prediction over current state-ofthe-art models; sharing information across neurons in a multi-task framework leads to better and more stable predictions; and these models work well even given relatively small amounts of experimental data. We believe that the multitask RNN framework presented here will enable new, richer models of complex nonlinear spiking computations in other brain areas.
While one could argue that we have merely exchanged the black box of the brain for another black box, just having a more predictive model is an important tool for research: these predictive models of the primate retina can be used in retinal prosthetics research, to probe decoding, and as a first stage of processing in the modeling of higher visual areas. Additionally, the recurrent network is more accessible and available for experimentation and quantitative analysis. For example, the trained neural network models may guide choices for more accurate simpler models by identifying key computational features that are important to include. Training smaller models on the denoised compression of spiking data (the predicted firing rate) may help them to learn features they otherwise would not (Ba & Caruana, 2014). The deep network approach allows one to determine types of information important to the neuron without having to build an exact mechanistic model of how such information is incorporated, as demonstrated by our finding that both spatial and temporal nonlinearities are not fully captured by the standard pseudo-linear models. We hope in future work to gain a more thorough and quantitative understanding of the dynamics captured by the recurrent networks and to extend this approach to higher sensory areas.
ACKNOWLEDGMENTS
Funding for this research was provided by the National Science Foundation Graduate Research Fellowship Program under grant No. DGE-114747 (NB), Grant Number No. DGE-16-44869 (EB), the National Science Foundation IGERT Training Grant No. 0801700 (NB), the National Institutes of Health Grant EY017992 (EJC), NSF CRCNS IIS-1430239 (LP, EJC) and Google Faculty Research awards (LP, EJC); in addition, this work was supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DoI/IBC) contract number D16PC00003 (LP). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/IBC, or the U.S. Government.

This paper fits models to spike trains of retinal ganglion cells that are driven by natural images. I think the title should thus include the word “activity” at the end for otherwise it is actually formally incorrect.
Anyhow, this paper proposes more specifically a recurrent network for this time series prediction and compares it to what seems to be the previous approach of a generalized linear model. Overall the stated paradigm is that when one can predict the spikes well then one can look into the model and learn how nature does it.
In general the paper sounds plausible, though I am not convinced that I learned a lot. The results in figure 2 show that the RNN model can predict the spikes a bit better. So this is nice. But now what? You have shown that a more complicated model can produce better fits to the data, though there are of course still some variations to the real data. Your initial outline was that a better predictive model helps you to better understand the neural processing in the retina. So tell us what you learned. I am not a specialist of the retina, but I know that there are several layers and recurrencies in the retina, so I am not so surprised that the new model is better than the GLM.
It seems that more complicated recurrent models such as LSTM do not improve the performance according to a statement in the paper. However, comparisons on this level are also difficult as a more complex models needs more data. Hence, I would actually expect that more layers and even a more detailed model of the retina could eventually improve the prediction even further.
I was also a bit puzzled that all the neurons in the network share all the same parameters (weights). While the results show that these simplified models can capture a lot of the spike train characteristics, couldn’t a model with free parameters eventually outperform this one (with correspondingly more training data)?


IMPROVING GENERATIVE ADVERSARIAL NETWORKS WITH DENOISING FEATURE MATCHING
We propose an augmented training procedure for generative adversarial networks designed to address shortcomings of the original by directing the generator towards probable configurations of abstract discriminator features. We estimate and track the distribution of these features, as computed from data, with a denoising auto-encoder, and use it to propose high-level targets for the generator. We combine this new loss with the original and evaluate the hybrid criterion on the task of unsupervised image synthesis from datasets comprising a diverse set of visual categories, noting a qualitative and quantitative improvement in the “objectness” of the resulting samples.
1 INTRODUCTION
Generative adversarial networks (Goodfellow et al., 2014a) (GANs) have become well known for their strength at realistic image synthesis. The objective function for the generative network is an implicit function of a learned discriminator network, estimated in parallel with the generator, which aims to tell apart real data from synthesized. Ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable.
In practice, GANs are well known for being quite challenging to train effectively. The relative model capacities of the generator and discriminator must be carefully balanced in order for the generator to effectively learn. Compounding the problem is the lack of an unambiguous and computable convergence criterion. Nevertheless, particularly when trained on image collections from relatively narrow domains such as bedroom scenes (Yu et al., 2015) and human faces (Liu et al., 2015), GANs have been shown to produce very compelling results.
For diverse image collections comprising a wider variety of the visual world, the results have generally been less impressive. For example, samples from models trained on ImageNet (Russakovsky et al., 2014) roughly match the local and global statistics of natural images but yield few recognizable objects. Recent work (Salimans et al., 2016) has sought to address this problem by training the discriminator in a semi-supervised fashion, granting the discriminator’s internal representations knowledge of the class structure of (some fraction of) the training data it is presented. This technique markedly increases sample quality, but is unsatisfying from the perspective of GANs as a tool for unsupervised learning.
We propose to augment the generator’s training criterion with a second training objective which guides the generator towards samples more like those in the training set by explicitly modeling the data density in addition to the adversarial discriminator. Rather than deploy a second computationally expensive convolutional network for this task, the additional objective is computed in the space of features learned by the discriminator. In that space, we train a denoising auto-encoder, a family of models which is known to estimate the energy gradient of the data on which it is trained. We evaluate the denoising auto-encoder on samples drawn from the generator, and use the “denoised” features as targets – nearby feature configurations which are more likely than those of the generated sample, according to the distribution estimated by the denoiser.
We show that this yields generators which consistently produce recognizable objects on the CIFAR10 dataset without the use of label information as in Salimans et al. (2016). The criterion appears to improve stability and possesses a degree of natural robustness to the well known “collapse” pathology. We further investigate the criterion’s performance on two larger and more diverse collections of images, and validate our qualitative observations quantitatively with the Inception score proposed in Salimans et al. (2016).
2 BACKGROUND
2.1 GENERATIVE ADVERSARIAL NETWORKS
The generative adversarial networks paradigm (Goodfellow et al., 2014a) estimates generative samplers by means of a training procedure which pits a generator G against a discriminator D. D is trained to tell apart training examples from samples produced by G, while G is trained to increase the probability of its samples being incorrectly classified as data. In the original formulation, the training procedure defines a continuous minimax game
arg min G arg max D
Ex∼D logD(x) + Ez∼p(z) log (1−D (G(z))) (1)
where D is a data distribution on Rn, D is a function that maps Rn to the unit interval, and G is a function that maps a noise vector z ∈ Rm, drawn from a simple distribution p(z), to the ambient space of the training data, Rn. The idealized algorithm can be shown to converge and to minimize the Jensen-Shannon divergence between the data generating distribution and the distribution parameterized by G.
Goodfellow et al. (2014a) found that in practice, minimizing (1) with respect to the parameters of G proved difficult, and elected instead to optimize an alternate objective,
arg max G Ez∼p(z) logD (G(z)) (2)
at the same time as D is optimized as above. logD(G(z)) yields more favourably scaled persample gradients forGwhenD confidently identifies a sample as counterfeit, avoiding the vanishing gradients arising in that case with the − log(1−D(G(z))) objective. Subsequent authors have investigated applications and extensions of GANs; for a review of this body of literature, see Warde-Farley & Goodfellow (2016). Of particular note for our purposes is Radford et al. (2015), who provide a set of general guidelines for the successful training of generative adversarial networks, and Salimans et al. (2016), who build upon these techniques with a number of useful heuristics and explore a variant in which the discriminator D is trained to correctly classify labeled training data, resulting in gradients with respect to the discriminator evidently containing a great deal of information relevant to generating “object-like” samples.
2.2 CHALLENGES AND LIMITATIONS OF GANS
While Goodfellow et al. (2014a) provides a theoretical basis for the GAN criterion, the theory relies on certain assumptions that are not satisfied in practice. Proofs demonstrate convergence of the GAN criterion in the unconstrained space of arbitrary functions; in practice, finitely parameterized families of functions such as neural networks are employed. As a consequence, the “inner loop” of the idealized algorithm – maximizing (1) with respect to (the parameters of) D, is infeasible to perform exactly, and in practice only one or a few gradient steps stand in for this maximization. This results in a de facto criterion for G which minimizes a lower bound on the correct objective (Goodfellow, 2014).
A commonly observed failure mode is that of full or partial collapse, where G maps a large fraction of probable regions under p(z) to only a few, low-volume regions of Rn; in the case of images, this manifests as the appearance of many near-duplicate images in independent draws from G, as well as a lower diversity of samples and modes than what is observed in the dataset. As G and D are typically trained via mini-batch stochastic gradient descent, several authors have proposed heuristics that penalize such duplication within each mini-batch (Salimans et al., 2016; Zhao et al., 2016).
GANs represent a departure from traditional probabilistic models based on maximum likelihood and its approximations in that they parameterize a sampler directly and lack a closed form for the likelihood. This makes objective, quantitative evaluation difficult. While previous results in the literature have reported approximate likelihoods based on Parzen window estimates, Theis et al. (2015) has convincingly argued that these estimates can be quite misleading for high-dimensional data. In this work, we adopt the Inception score proposed by Salimans et al. (2016), which uses a reference Inception convolutional neural network (Szegedy et al., 2015) to compute
I({x}N1 ) = exp (E [DKL(p(y|x)‖p(y)])) (3) where p(y|x) is provided by the output of the Inception network and p(y) = ∫ x p(x)p(y|x)dx u 1 N ∑ p(y|xi). Note that this score can be made larger by a low-entropy per-sample posterior (i.e. the Inception network classifies a given sample with greater certainty) as well as a higher entropy aggregate posterior (i.e. the Inception network identifies a wide variety of classes among the samples presented to it). Salimans et al. (2016) found this score correlated well with human evaluations of samplers trained on CIFAR-10; we therefore employ the Inception score here as a quantitative measure of visual fidelity of the samples, following the previous work’s protocol of evaluating the average Inception score over 10 independent groups of 5,000 samples each. Error estimates correspond to standard deviations, in keeping with previously reported results.
3 IMPROVING UNSUPERVISED GAN TRAINING ON DIVERSE DATASETS
In this work, we focus on the apparent difficulty of training GANs to produce “object-like” samples when trained on diverse collections of natural images. While Salimans et al. (2016) make progress on this problem by employing labeled data and training the discriminator, here we aim to make progress on the unsupervised case. Nevertheless, our methods would be readily applicable to supervised, semi-supervised or (with slight modifications) conditional setting.
We begin from the slightly subtle observation that in realistic manifestations of the GAN training procedure, the discriminator’s (negative) gradient with respect to a sample points in a direction of (infinitesimal) local improvement with respect to the discriminator’s estimate of the sample being data; it does not necessarily point in the direction of a draw from the data distribution. Indeed, the literature is replete with instances of gradient descent with respect to the input of a classification model, particularly wide-domain natural image classifiers, producing ghostly approximations to a particular class exemplar (Le et al., 2012; Erhan et al., 2009; Yosinski et al., 2015) when this procedure is carried out without additional guidance, to say nothing of the problems posed by adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014b) and fooling examples (Nguyen et al., 2015).
While the gradient of the loss function defined by the discriminator may be a source of information mostly relevant to very local improvements, the discriminator itself is a potentially valuable source of compact descriptors of the training data. Many authors have noted the remarkable versatility of high-level features learned by convolutional networks (Donahue et al., 2014; Yosinski et al., 2014) and the degree to which high-level semantics can be reconstructed from even the deepest layers of a network (Dosovitskiy & Brox, 2016). Although non-stationary, the distribution of the highlevel activations of the discriminator when evaluated on data is ripe for exploitation as an additional source of knowledge about salient aspects of the data distribution.
We propose in this work to track this distribution with a denoising auto-encoder r(·) trained on the discriminator’s hidden states when evaluated on training data. Alain & Bengio (2014) showed that a denoising auto-encoder trained on data from a distribution q(h) estimates via r(h) − h the gradient of the true log-density, ∂ log q(h)∂h . Hence, if we train the denoising auto-encoder on the transformed training data h = Φ(x) with x ∼ D, then r(Φ(x′))− Φ(x′) with x′ = G(z) indicates in which direction x′ should be changed in order to make h = Φ(x′) more like those features seen with the data. Minimizing ||r(Φ(x′)) − Φ(x′)||2 with respect to x′ would thus push x′ towards higher probability configurations according to the data distribution in the feature space Φ(x). We thus evaluate the discriminator features Φ(x), and the denoising auto-encoder, on samples from the generator, and treat the denoiser’s output reconstruction as a fixed target for the generator. We refer to this procedure as denoising feature matching, and employ it as a learning signal for the generator in addition to the traditional GAN generator objective.
Formally, letG be the generator parameterized by θG, andD = d◦Φ be our discriminator composing feature extractor Φ(·) : Rn → Rk and a classifier d(·) : Rk → [0, 1]. Let C(·) : Rk → Rk be a corruption function to be applied at the input of the denoising auto-encoder when it is trained to denoise. The parameters of the discriminator D, comprising the parameters of both d and Φ, is trained as in Goodfellow et al. (2014a), while the generator is trained according to
arg min θG
Ez∼p(z) [ λdenoise‖Φ(G(z))− r(Φ(G(z)))‖|2 − λadv logD(G(z)) ] (4)
where r(G(z)) is treated as constant with respect to gradient computations. Simultaneously, the denoiser r(·) is trained according to the objective
arg min θr
Ex∼D‖Φ(x)− r(C(Φ(x)))‖2 (5)
3.1 EFFECT OF Φ
The theory surrounding denoising auto-encoders applies when estimating a denoising function from a data distribution p(vecx). Here, we propose to estimate the denoising auto-encoder in the space of discriminator features, giving rise to a distribution q(Φ(x)). A natural question is what effect this has on the gradient being backpropagated. This is difficult to analyze in general, as for most choices the mapping Φ will not be invertible, though it is instructive to examine the invertible case. Assuming an invertible Φ : Rn → Rn, let J = ∂Φ(x)∂x be the Jacobian of Φ, and q(Φ(x)) = p(x)|J |. By the inverse function theorem, J is also invertible (and is in fact the Jacobian of the inverse Φ−1). Applying the chain rule and re-arranging terms, taking advantage of the invertibility of J , we arrive at a straightforward relationship between the score of q and the score of p:
∂ log q(Φ(x)) ∂Φ(x) = ∂ log [p(x) |J |] ∂Φ(x) (6)
= ∂ log p(x) ∂Φ(x) + ∂ log ∣∣∣∂Φ(x)∂x ∣∣∣ ∂Φ(x)
(7)
=
( ∂ log p(x)
∂x + ∂ log |J | ∂x
) J−1 (8)
where
∂ log |J | ∂xk = Tr
( J−1 dJ
dxk
) (9)
and dJdxk is a matrix of scalar derivatives of elements of J with respect to xk. Thus, we see that the gradient backpropagated to the generator in an ideal setting is the gradient of the data distribution p(x) along with an additive term which accounts for the changes in the rate of volume expansion/contraction in Φ locally around x. In practice, Φ is not invertible, but the added benefit of the denoiser-targeted gradient appears to reduce underfitting to the modes of p in the generator, irrespective of any distortions Φ may introduce.
4 RELATED WORK
Denoising feature matching was originally inspired by feature matching introduced by Salimans et al. (2016) as an alternative training criterion for GAN generators, namely (in our notation)
arg min θG ∣∣‖Ex∼D [Φ(x)]− Ez∼p(z) [Φ(G(z))] ‖∣∣2 (10) Feature matching is equivalent to linear maximum mean discrepancy (Gretton et al., 2006), employing linear first moment matching in the space of discriminator features Φ(·) rather than the more familiar kernelized formulation. When performed on features in the penultimate layer, Salimans et al. (2016) found that the feature matching criterion was useful for the purpose of improving
results on semi-supervised classification, using classification of samples from the generator as a sophisticated form of data augmentation. Feature matching was, however, less successful at producing samples with high visual fidelity. This is somewhat unsurprising given that the criterion is insensitive to higher-order statistics of the respective feature distributions. Indeed, a degenerate G which deterministically reproduces a single sample m̂ such that Φ(m̂) = Ex∈DΦ(x) trivially minimizes (10); in practice the joint training dynamics of D and G do not appear to yield such degenerate solutions.
Rather than aiming to merely reduce linear separability between data and samples in the feature space defined by Φ(·), denoising feature matching selects a more probable (according to the feature distribution implied by the data, as captured by the denoiser) feature space target for each sample produced by G and regresses G towards it. While an early loss of entropy in G could result in the generator locking on to one or a few attractors in the denoiser’s energy landscape, we observe that this does not happen when used in conjunction with the traditional GAN objective, and in fact that the combination of the two objectives is notably robust to the collapses often observed in GAN training, even without taking additional measures to prevent them.
This work also draws inspiration from Alain & Bengio (2014), which showed that a suitably trained denoiser learns an operator which locally maps a sample towards regions of high probability under the data distribution. They further showed that a suitably trained1 reconstruction function r(·) behaves such that
r(x)− x ∝ ∂ log p(x) ∂x
(11)
That is, r(x) − x estimates the score of the data generating distribution, up to a multiplicative constant. Our use of denoising auto-encoders necessarily departs from idealized conditions in that the denoiser is estimated online from an ever-changing distribution of features.
Several approaches to GAN-like models have cast the problem in terms of learning an energy function. Kim & Bengio (2016) extends GANs by modeling the data distribution simultaneously with an energy function parameterized by a deep neural network (playing the role of the discriminator) and the traditional generator, carrying out learning with a learning rule resembling that of the Boltzmann machine (Ackley et al., 1985), where the “negative phase” gradient is estimated from samples from the generator. The energy-based GAN formulation of Zhao et al. (2016) resembles our work in their use of an auto-encoder which is trained to faithfully reconstruct (in our case, a corrupted, function of) the training data. The energy-based GAN replaces the discriminator with an auto-encoder, which is trained to assign low energy (L2 reconstruction error) to training data and higher energy to samples from G. To discourage generator collapses, a “pull-away term” penalizes the normalized dot product in a feature space defined by the auto-encoder’s internal representation. In this work, we preserve the discriminator, trained in the usual discriminative fashion, and in fact preserve the traditional generator loss, instead augmenting it with a source of complementary information provided by targets obtained from the denoiser. The energy-based GAN can be viewed as training the generator to seek fixed points of the autoencoding function (i.e. by backpropagating through the decoder and encoder in order to decrease reconstruction error), whereas we treat the output of r(·) as constant with respect to the optimization as in Lee et al. (2015). That is to say, rather than using backpropagation to steer the dynamics of the autoencoder, we instead employ our denoising autoencoder to augment the gradient information obtained by ordinary backpropagation.
Closest to our own approach, concurrent work on model-based super-resolution by Sønderby et al. (2016) trains a denoising auto-encoder on high-resolution ground truth and evaluates it on synthesized super-resolution images, using the difference between the original synthesized image and the denoiser’s output as an additional training signal for refining the output of the super-resolution network. Both Sønderby et al. (2016) and our own work are motivated by the results of Alain & Bengio (2014) discussed above. Aside from addressing a different application area, our denoiser is learned on-the-fly from a high-level feature representation which is itself learned.
1In the limit of infinite training data, with isotropic Gaussian noise of some standard deviation σ.
5 EXPERIMENTS
We evaluate denoising feature matching on learning synthesis models from three datasets of increasing diversity and size: CIFAR-10, STL-10, and ImageNet. Although several authors have described GAN-based image synthesis models operating at 128 × 128 (Salimans et al., 2016; Zhao et al., 2016) and 256× 256 (Zhao et al., 2016) resolution, we carry out our investigations at relatively low resolutions, both for computational ease and because we believe that the problem of unconditional modeling of diverse image collections is not well solved even at low resolutions; making progress in this regime is likely to yield insights that apply to the higher-resolution case.
In all experiments, we employ isotropic Gaussian corruption noise with σ = 1. Although we experimented with annealing σ towards 0 (as also performed in Sønderby et al. (2016)), an annealing schedule which consistently outperformed fixed noise remained elusive. We experimented with convolutional denoisers, but our best results to date were obtained with deep, fully-connected denoisers using the ReLU nonlinearity on the penultimate layer of the discriminator. The number of hidden units was fixed to the same value in all denoiser layers, and the procedure is apparently robust to this hyperparameter choice, as long as it is greater than or equal to the input dimensionality.
Our generator and discriminator architectures follow the methods outlined in Radford et al. (2015). Accordingly, batch normalization (Ioffe & Szegedy, 2015) was used in the generator and discriminator in the same manner as Radford et al. (2015), and in all layers of the denoiser except the output layer. In particular, as in Radford et al. (2015), we separately batch normalize data and generator samples for the discriminator and denoiser with respect to each source’s statistics. We calculate updates with respect to all losses with the parameters of all three networks fixed, and update all parameters simultaneously.
All networks were trained with the Adam optimizer Kingma & Ba (2014) with a learning rate of 10−4 and β1 = 0.5. The Adam optimizer is scale invariant, and so it suffices to e.g. tune λdenoise and fix λadv to 1. In our experiments, we set λdenoise to 0.03/nh, where nh is the number of discriminator hidden units fed as input to the denoiser; this division decouples the scale of the first term of (4) from the dimensionality of the representation used, reducing the need to adjust this hyperparameter simply because we altered the architecture of the discriminator.
5.1 CIFAR-10
CIFAR-10 (Krizhevsky & Hinton, 2009) is a small, well-studied dataset consisting of 50,000 32×32 pixel RGB training images and 10,000 test images from 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
Samples from our model trained on CIFAR-10 are shown in Figure 1, and Inception scores for several methods, including those reported in Salimans et al. (2016) and scores computed from samples generated from a model presented in Dumoulin et al. (2016), are presented in Table 1. We achieve a mean Inception score of 7.72, falling slightly short of Salimans et al. (2016), which employed a supervised discriminator network (the same work reports a score of 4.36 ± .04 when labels are omitted from their training procedure). Qualitatively, the samples include recognizable cars, boats and various animals. The best performing generator network consisted of the 32 × 32 ImageNet architecture from Radford et al. (2015) with half the number of parameters at each layer, and less than 40% of the parameters of the CIFAR-10 generator presented in Salimans et al. (2016).
5.2 STL-10
STL-10 (Coates et al., 2011) is a dataset consisting of a small labeled set and larger (100,000) unlabeled set of 96 × 96 RGB images. The unlabeled set is a subset of ImageNet that is more diverse than CIFAR-10 (or the labeled set of STL-10), but less diverse than full ImageNet. We downsample by a factor of 2 on each dimension and train our networks at 48× 48. Inception scores for our model and a baseline, consisting of the same architecture trained without denoising feature matching (both trained for 50 epochs), are shown in Table 2. Samples are displayed in Figure 2.
5.3 IMAGENET
The ImageNet database (Russakovsky et al., 2014) is a large-scale database of natural images. We train on the designated training set of the most widely used release, the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC2012), consisting of a highly unbalanced split among 1,000 object classes. We preprocess the dataset as rescaled central crops following the procedure of Krizhevsky et al. (2012), except at 32 × 32 resolution to facilitate comparison with Radford et al. (2015).
ImageNet poses a particular challenge for unsupervised GANs due to its high level of diversity and class skew. With a generator and discriminator architecture identical to that used for the same dataset in Radford et al. (2015), we achieve a higher Inception score using denoising feature matching, using denoiser with 10 hidden layers of 2,048 rectified linear units each. Both fall far short of the score assigned to real data at this resolution; there is still plenty of room for improvement. Samples are displayed in Figure 3.
6 DISCUSSION AND FUTURE DIRECTIONS
We have shown that training a denoising model on high-level discriminator activations in a GAN, and using the denoiser to propose high-level feature targets for the generator, can usefully improve
GAN image models. Higher Inception scores, as well as visual inspection, suggest that the procedure captures class-specific features of the training data in a manner superior to the original adversarial objective alone. That being said, we do not believe we are yet making optimal use of the paradigm. The non-stationarity of the feature distribution on which the denoiser is trained could be limiting the ability of the denoiser to obtain a good fit, and the information backpropagated to the generator is always slightly stale. Steps to reduce this non-stationarity may be fruitful; we experimented briefly with historical averaging as explored in Salimans et al. (2016) but did not observe a clear benefit thus far. Structured denoisers, including denoisers that learn an energy function for multiple hidden layers at once, could conceivably aid in obtaining a better fit. Learning a partially stochastic transition operator rather than a deterministic denoiser could conceivably capture interesting multimodalities that are “blurred” by a unimodal denoising function.
Our method is orthogonal and could conceivably be used in combination with several other GAN extensions. For example, methods incorporating an encoder component (Donahue et al., 2016; Dumoulin et al., 2016), various existing conditional architectures (Mirza & Osindero, 2014; Denton et al., 2015; Reed et al., 2016), or the semi-supervised variant employed in Salimans et al. (2016), could all be trained with an additional denoising feature matching objective.
We have proposed a useful heuristic, but a better theoretical grounding regarding how GANs are trained in practice is a necessary direction for future work, including grounded criteria for assessing mode coverage and mass misassignment, and principled criteria for assessing convergence or performing early stopping.
ACKNOWLEDGMENTS
We thank Ian Goodfellow, Laurent Dinh, Yaroslav Ganin and Kyle Kastner for helpful discussions. We thank Vincent Dumoulin and Ishmael Belghazi for making available code and model parameters used in comparison to ALI, as well as Alec Radford for making available the code and model parameters for his ImageNet model. We would like to thank Antonia Creswell and Hiroyuki Yamazaki for pointing out an error in the initial version of this manuscript, and anonymous reviewers for valuable feedback. We thank the University of Montreal and Compute Canada for the computational resources used for this investigation, as well as the authors of Theano (Al-Rfou et al., 2016), Blocks and Fuel (van Merriënboer et al., 2015). We thank CIFAR, NSERC, Google, Samsung and Canada Research Chairs for funding.

The authors present a way to complement the Gerative Adversarial Network traning procedure with an additional term based on denoising autoencoders. The use of denoising autoencoders is motivated by the observation that they implicitly capture the distribution of the data they were trained on. While sampling methods based denoising autoencoders alone don't amount to very interesting generative models (at least no-one could demonstrate otherwise), this paper shows that it can be combined successfully with generative adversarial networks.
My overall assessment of this paper is that it is well written, well reasoned, and presents a good idea motivated from first principles. I feel that the idea presented here is not revolutionary or a very radical departure from what has been done before, I would have liked a slightly more structured experiments section which focusses on and provides insights into the relative merits of different choices one could make (see pre-review questions for details), rather than focussing just on demonstrating that a chosen variant works.
In addition to this general review, I have already posted specific questions and criticism in the pre-review questions - thanks for the authors' responses. Based on those responses the area I am most uncomfortable about is whether the (Alain & Bengio, 2014) intuition about the denoising autoencoders is valid if it all happens in a nonlinear featurespace. If the denoiser function's behaviour ends up depending on the Jacobian of the nonlinear transformation Phi, another question is whether this dependence is exploitable by the optimization scheme.


CONTEXT-CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS
We introduce a simple semi-supervised learning approach for images based on in-painting using an adversarial loss. Images with random patches removed are presented to a generator whose task is to fill in the hole, based on the surrounding pixels. The in-painted images are then presented to a discriminator network that judges if they are real (unaltered training images) or not. This task acts as a regularizer for standard supervised training of the discriminator. Using our approach we are able to directly train large VGG-style networks in a semi-supervised fashion. We evaluate on STL-10 and PASCAL datasets, where our approach obtains performance comparable or superior to existing methods.
1 INTRODUCTION
Deep neural networks have yielded dramatic performance gains in recent years on tasks such as object classification (Krizhevsky et al., 2012), text classification (Zhang et al., 2015) and machine translation (Sutskever et al., 2014; Bahdanau et al., 2015). These successes are heavily dependent on large training sets of manually annotated data. In many settings however, such large collections of labels may not be readily available, motivating the need for methods that can learn from data where labels are rare.
We propose a method for harnessing unlabeled image data based on image in-painting. A generative model is trained to generate pixels within a missing hole, based on the context provided by surrounding parts of the image. These in-painted images are then used in an adversarial setting (Goodfellow et al., 2014) to train a large discriminator model whose task is to determine if the image was real (from the unlabeled training set) or fake (an in-painted image). The realistic looking fake examples provided by the generative model cause the discriminator to learn features that generalize to the related task of classifying objects. Thus adversarial training for the in-painting task can be used to regularize large discriminative models during supervised training on a handful of labeled images.
1.1 RELATED WORK
Learning From Context: The closest work to ours is the independently developed context-encoder approach of Pathak et al. (2016). This introduces an encoder-decoder framework, shown in Fig. 1(a), that is used to in-paint images where a patch has been randomly removed. After using this as a pre-training task, a classifier is added to the encoder and the model is fine-tuned using the labeled examples. Although both approaches use the concept of in-painting, they differ in several important ways. First, the architectures are different (see Fig. 1): in Pathak et al. (2016), the features for the classifier are taken from the encoder, whereas ours come from the discriminator network. In practice this makes an important difference as we are able to directly train large models such as VGG (Simonyan & Zisserman, 2015) using adversarial loss alone. By contrast, Pathak et al. (2016) report difficulties in training an AlexNet encoder with this loss. This leads to the second difference, namely that on account of these issues, they instead employ an `2 loss when training models for classification and detection (however they do use a joint `2 and adversarial loss to achieve impressive in-painting results). Finally, the unsupervised learning task differs between the two models. The context-encoder learns a feature representation suitable for in-painting whereas our model learns a feature representation suitable for differentiating real/fake in-paintings. Notably, while we also use a neural network to generate the in-paintings, this model is only used as an adversary for the
discriminator, rather than as a feature extractor. In section 4, we compare the performance of our model to the context-encoder on the PASCAL dataset.
Other forms of spatial context within images have recently been utilized for representation learning. Doersch et al. (2015) propose training a CNN to predict the spatial location of one image patch relative to another. Noroozi & Favaro (2016) propose a model that learns by unscrambling image patches, essentially solving a jigsaw puzzle to learn visual representations. In the text domain, context has been successfully leveraged as an unsupervised criterion for training useful word and sentence level representations (Collobert et al., 2011; Mikolov et al., 2015; Kiros et al., 2015).
Deep unsupervised and semi-supervised learning: A popular method of utilizing unlabeled data is to layer-wise train a deep autoencoder or restricted Botlzmann machine (Hinton et al., 2006) and then fine tune with labels on a discriminative task. More recently, several autoencoding variants have been proposed for unsupervised and semi-supervised learning, such as the ladder network (Rasmus et al., 2015), stacked what-where autoencoders (Zhao et al., 2016) and variational autoencoders (Kingma & Welling, 2014; Kingma et al., 2014).
Dosovitskiy et al. (2014) achieved state-of-the-art results by training a CNN with a different class for each training example and introducing a set of transformations to provide multiple examples per class. The pseudo-label approach (Lee, 2013) is a simple semi-supervised method that trains using the maximumly predicted class as a label when labels are unavailable. Springenberg (2015) propose a categorical generative adversarial network (CatGAN) which can be used for unsupervised and semi-supervised learning. The discriminator in a CatGAN outputs a distribution over classes and is trained to minimize the predicted entropy for real data and maximize the predicted entropy for fake data. Similar to our model, CatGANs use the feature space learned by the discriminator for the final supervised learning task. Salimans et al. (2016) recently proposed a semi-supervised GAN model in which the discriminator outputs a softmax over classes rather than a probability of real vs. fake. An additional ‘generated’ class is used as the target for generated samples. This method differs from our work in that it does not utilize context information and has only been applied to datasets of small resolution. However, the discriminator loss is similar to the one we propose and could be combined with our context-conditional approach.
More traditional semi-supervised methods include graph-based approaches (Zhou et al., 2004; Zhu, 2006) that show impressive performance when good image representations are available. However, the focus of our work is on learning such representations.
Generative models of images: Restricted Boltzmann machines (Salakhutdinov, 2015), de-noising autoencoders (Vincent et al., 2008) and variational autoencoders (Kingma & Welling, 2014) optimize a maximum likelihood criterion and thus learn decoders that map from latent space to image space. More recently, generative adversarial networks (Goodfellow et al., 2014) and generative mo-
ment matching networks (Li et al., 2015; Dziugaite et al., 2015) have been proposed. These methods ignore data likelihoods and instead directly train a generative model to produce realistic samples. Several extensions to the generative adversarial network framework have been proposed to scale the approach to larger images (Denton et al., 2015; Radford et al., 2016; Salimans et al., 2016). Our work draws on the insights of Radford et al. (2016) regarding adversarial training practices and architecture for the generator network, as well as the notion that the discriminator can produce useful features for classification tasks.
Other models used recurrent approaches to generate images (Gregor et al., 2015; Theis & Bethge, 2015; Mansimov et al., 2016; van den Oord et al., 2016). Dosovitskiy et al. (2015) trained a CNN to generate objects with different shapes, viewpoints and color. Sohl-Dickstein et al. (2015) propose a generative model based on a reverse diffusion process. While our model does involve image generation, it differs from these approaches in that the main focus is on learning a good representation for classification tasks.
Predictive generative models of videos aim to extrapolate from current frames to future ones and in doing so learn a feature representation that is useful for other tasks. In this vein, Ranzato et al. (2014) used an `2-loss in pixel-space. Mathieu et al. (2015) combined an adversarial loss with `2, giving models that generate crisper images. While our model is also predictive, it only considers interpolation within an image, rather than extrapolation in time.
2 APPROACH
We present a semi-supervised learning framework built on generative adversarial networks (GANs) of Goodfellow et al. (2014). We first review the generative adversarial network framework and then introduce context conditional generative adversarial networks (CC-GANs). Finally, we show how combining a classification objective and a CC-GAN objective provides a unified framework for semi-supervised learning.
2.1 GENERATIVE ADVERSARIAL NETWORKS
The generative adversarial network approach (Goodfellow et al., 2014) is a framework for training generative models, which we briefly review. It consists of two networks pitted against one another in a two player game: A generative model, G, is trained to synthesize images resembling the data distribution and a discriminative model, D, is trained to distinguish between samples drawn from G and images drawn from the training data.
More formally, let X = {x1, ...,xn} be a dataset of images of dimensionality d. Let D denote a discriminative function that takes as input an image x ∈ Rd and outputs a scalar representing the probability of input x being a real sample. Let G denote the generative function that takes as input a random vector z ∈ Rz sampled from a prior noise distribution pNoise and outputs a synthesized image x̃ = G(z) ∈ Rd. Ideally, D(x) = 1 when x ∈ X and D(x) = 0 when x was generated from G. The GAN objective is given by:
min G max D Ex∼X [logD(x)] + Ez∼pNoise(z)[log(1−D(G(z)))] (1)
The conditional generative adversarial network (Mirza & Osindero, 2014) is an extension of the GAN in which bothD andG receive an additional vector of information y as input. The conditional GAN objective is given by:
min G max D Ex,y∼X [logD(x,y)] + Ez∼pNoise(z)[log(1−D(G(z,y),x))] (2)
2.2 CONTEXT-CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS
We propose context-conditional generative adversarial networks (CC-GANs) which are conditional GANs where the generator is trained to fill in a missing image patch and the generator and discriminator are conditioned on the surrounding pixels.
In particular, the generator G receives as input an image with a randomly masked out patch. The generator outputs an entire image. We fill in the missing patch from the generated output and then pass the completed image into D. We pass the completed image into D rather than the context and the patch as two separate inputs so as to prevent D from simply learning to identify discontinuities along the edge of the missing patch.
More formally, let m ∈ Rd denote to a binary mask that will be used to drop out a specified portion of an image. The generator receives as input m x where denotes element-wise multiplication. The generator outputs xG = G(m x, z) ∈ Rd and the in-painted image xI is given by:
xI = (1−m) xG +m x (3)
The CC-GAN objective is given by:
min G max D
Ex∼X [logD(x)] + Ex∼X ,m∼M[log(1−D(xI))] (4)
2.3 COMBINED GAN AND CC-GAN
While the generator of the CC-GAN outputs a full image, only a portion of it (corresponding to the missing hole) is seen by the discriminator. In the combined model, which we denote by CC-GAN2, the fake examples for the discriminator include both the in-painted image xI and the full image xG produced by the generator (i.e. not just the missing patch). By combining the GAN and CC-GAN approaches, we introduce a wider array of negative examples to the discriminator. The CC-GAN2 objective given by:
min G max D
Ex∼X [logD(x)] (5)
+ Ex∼X ,m∼M[log(1−D(xI))] (6) + Ex∼X ,m∼M[log(1−D(xG))] (7)
2.4 SEMI-SUPERVISED LEARNING WITH CC-GANS
A common approach to semi-supervised learning is to combine a supervised and unsupervised objective during training. As a result unlabeled data can be leveraged to aid the supervised task.
Intuitively, a GAN discriminator must learn something about the structure of natural images in order to effectively distinguish real from generated images. Recently, Radford et al. (2016) showed that a GAN discriminator learns a hierarchical image representation that is useful for object classification. Such results suggest that combining an unsupervised GAN objective with a supervised classification objective would produce a simple and effective semi-supervised learning method. This approach, denoted by SSL-GAN, is illustrated in Fig. 1(b). The discriminator network receives a gradient from the real/fake loss for every real and generated image. The discriminator also receives a gradient from the classification loss on the subset of (real) images for which labels are available.
Generative adversarial networks have shown impressive performance on many diverse datasets. However, samples are most coherent when the set of images the network is trained on comes from a limited domain (eg. churches or faces). Additionally, it is difficult to train GANs on very large images. Both these issues suggest semi-supervised learning with vanilla GANs may not scale well to datasets of large diverse images. Rather than determining if a full image is real or fake, context conditional GANs address a different task: determining if a part of an image is real or fake given the surrounding context.
Formally, let XL = {(x1, y1), ..., (xn, yn)} denote a dataset of labeled images. Let Dc(x) denote the output of the classifier head on the discriminator (see Fig. 1(c) for details). Then the semisupervised CC-GAN objective is:
min G max D Ex∼X [logD(x)] + Ex∼X ,m∼M[log(1−D(xI))] + λcEx,y∼XL [log(Dc(y|x))] (8)
The hyperparameter λc balances the classification and adversarial losses. We only consider the CCGAN in the semi-supervised setting and thus drop the SSL notation when referring to this model.
2.5 MODEL ARCHITECTURE AND TRAINING DETAILS
The architecture of our generative model,G, is inspired by the generator architecture of the DCGAN (Radford et al., 2016). The model consists of a sequence of convolutional layers with subsampling (but no pooling) followed by a sequence of fractionally-strided convolutional layers. For the discriminator, D, we used the VGG-A network (Simonyan & Zisserman, 2015) without the fully connected layers (which we call the VGG-A’ architecture). Details of the generator and discriminator are given
in Fig. 2. The input to the generator is an image with a patch zeroed out. In preliminary experiments we also tried passing in a separate mask to the generator to make the missing area more explicit but found this did not effect performance.
Even with the context conditioning it is difficult to generate large image patches that look realistic, making it problematic to scale our approach to high resolution images. To address this, we propose conditioning the generator on both the high resolution image with a missing patch and a low resolution version of the whole image (with no missing region). In this setting, the generators task becomes one of super-resolution on a portion of an image. However, the discriminator does not receive the low resolution image and thus is still faced with the same problem of determining if a given in-painting is viable or not. Where indicated, we used this approach in our PASCAL VOC 2007 experiments, with the original image being downsampled by a factor of 4. This provided enough information for the generator to fill in larger holes but not so much that it made the task trivial. This optional low resolution image is illustrated in Fig. 2(left) with the dotted line.
We followed the training procedures of Radford et al. (2016). We used the Adam optimizer (Kingma & Ba, 2015) in all our experiments with learning rate of 0.0002, momentum term β1 of 0.5, and the remaining Adam hyperparameters set to their default values. We set λc = 1 for all experiments.
3 EXPERIMENTS
3.1 STL-10 CLASSIFICATION
STL-10 is a dataset of 96×96 color images with a 1:100 ratio of labeled to unlabeled examples, making it an ideal fit for our semi-supervised learning framework. The training set consists of 5000 labeled images, mapped to 10 pre-defined folds of 1000 images each, and 100,000 unlabeled images. The labeled images belong to 10 classes and were extracted from the ImageNet dataset and the unlabeled images come from a broader distribution of classes. We follow the standard testing protocol and train 10 different models on each of the 10 predefined folds of data. We then evaluate classification accuracy of each model on the test set and report the mean and standard deviation.
We trained our CC-GAN and CC-GAN2 models on 64×64 crops of the 96×96 image. The hole was 32×32 pixels and the location of the hole varied randomly (see Fig. 3(top)). We trained for 100 epochs and then fine-tuned the discriminator on the 96x96 labeled images, stopping when training accuracy reached 100%. As shown in Table 1, the CC-GAN model performs comparably to current state of the art (Dosovitskiy et al., 2014) and the CC-GAN2 model improves upon it.
We also trained two baseline models in an attempt to tease apart the contributions of adversarial training and context conditional adversarial training. The first is a purely supervised training of the VGG-A’ model (the same architecture as the discriminator in the CC-GAN framework). This was trained using a dropout of 0.5 on the final layer and weight decay of 0.001. The performance of this model is significantly worse than the CC-GAN model.
We also trained a semi-supervised GAN (SSL-GAN, see Fig. 1(b)) on STL-10. This consisted of the same discriminator as the CC-GAN (VGG-A’ architecture) and generator from the DCGAN model (Radford et al., 2016). The training setup in this case is identical to the CC-GAN model. The SSLGAN performs almost as well as the CC-GAN, confirming our hypothesis that the GAN objective is a useful unsupervised criterion.
3.2 PASCAL VOC CLASSIFICATION
In order to compare against other methods that utilize spatial context we ran the CC-GAN model on PASCAL VOC 2007 dataset. This dataset consists of natural images coming from 20 classes. The dataset contains a large amount of variability with objects varying in size, pose, and position. The training and validation sets combined contain 5,011 images, and the test set contains 4,952 images. The standard measure of performance is mean average precision (mAP).
We trained each model on the combined training and validation set for ∼5000 epochs and evaluated on the test set once1. Following Pathak et al. (2016), we train using random cropping, and then evaluate using the average prediction from 10 random crops.
Our best performing model was trained on images of resolution 128×128 with a hole size of 64×64 and a low resolution input of size 32×32. Table 2 compares our CC-GAN method to other feature learning approaches on the PASCAL test set. It outperforms them, beating the current state of the art (Wang & Gupta, 2015) by 3.8%. It is important to note that our feature extractor is the VGGA’ model which is larger than the AlexNet architecture (Krizhevsky et al., 2012) used by other approaches in Table 2. However, purely supervised training of the two models reveals that VGG-A’
1Hyperparameters were determined by initially training on the training set alone and measuring performance on the validation set.
is less than 2% better than AlexNet. Furthermore, our model outperforms the supervised VGG-A’ baseline by a 7% margin (62.2% vs. 55.2%). This suggests that our gains stem from the CC-GAN method rather than the use of a better architecture.
Table 3 shows the effect of training on different resolutions. The CC-GAN improves over the baseline CNN consistently regardless of image size. We found that conditioning on the low resolution image began to help when the hole size was largest (64×64). We hypothesize that the low resolution conditioning would be more important for larger images, potentially allowing the method to scale to larger image sizes than we explored in this work.
3.3 INPAINTING
We now show some sample in-paintings produced by our CC-GAN generators. In our semisupervised learning experiments on STL-10 we remove a single fixed size hole from the image. The top row of Fig. 3 shows in-paintings produced by this model. We can also explored different masking schemes as illustrated in the remaining rows of Fig. 3 (however these did not improve classification results). In all cases we see that training the generator with the adversarial loss produces sharp semantically plausible in-painting results.
Fig. 4 shows generated images and in-painted images from a model trained with the CC-GAN2 criterion. The output of a CC-GAN generator tends to be corrupted outside the patch used to inpaint the image (since gradients only flow back to the missing patch). However, in the CC-GAN2 model, we see that both the in-painted image and the generated image are coherent and semantically consistent with the masked input image.
Fig. 5 shows in-painted images from a generator trained on 128×128 PASCAL images. Fig. 6 shows the effect of adding a low resolution (32×32) image as input to the generator. For comparison we also show the result of in-painting by filling in with a bi-linearly upsampled image. Here we see the generator produces high-frequency structure rather than simply learning to copy the low resolution patch.
4 DISCUSSION
We have presented a simple semi-supervised learning framework based on in-painting with an adversarial loss. The generator in our CC-GAN model is capable of producing semantically meaningful
in-paintings and the discriminator performs comparable to or better than existing semi-supervised methods on two classification benchmarks.
Since discrimination of real/fake in-paintings is more closely related to the target task of object classification than extracting a feature representation suitable for in-filling, it is not surprising that we are able to exceed the performance of Pathak et al. (2016) on PASCAL classification. Furthermore, since our model operates on images half the resolution as those used by other approaches (128×128 vs. 224×244), there is potential for further gains if improvements in the generator resolution can be made. Our models and code are available at https://github.com/edenton/cc-gan.
Acknowledgements: Emily Denton is supported by a Google Fellowship. Rob Fergus is grateful for the support of CIFAR.

After rebuttal:
Thanks for reporting the AlexNet results. The fact that they are not great is not so bad by itself, and as the authors mention, it would be interesting to understand why this happens. But the fact that these results  were not in the paper (and in fact still are not there) is disturbing. Moreover, some claims in the paper look wrong in the light of these results, for example:
- "This suggests that our gains stem from the CC-GAN method rather than the use of a better architecture."
- "Since discrimination of real/fake in-paintings is more closely related to the target task of object classification than extracting a feature representation suitable for in-filling, it is not surprising that we are able to exceed the performance of Pathak et al. (2016) on PASCAL classification."
These statements, and possibly other parts of the paper, have to be updated. I think the paper cannot be published in its current form. Perhaps after a revision.
--------
Initial review:
The paper demonstrates an application of generative adversarial networks (GAN) to unsupervised feature learning. The authors show that the representation learned by the discriminator of a conditional GAN trained for image inpainting performs well on image classification. As a side-effect, fairly convincing inpaintings are produced.
The proposed method combines two existing ideas: using the discriminator of a GAN as a feature learner [Radford et al. 2015] and performing unsupervised feature learning with image inpainting [Pathak et al. 2016]. Therefore conceptual novelty of the paper is limited. On the plus side, the authors implement their idea well and demonstrate state-of-the-art results on STL-10 and good results on Pascal VOC (although Pascal experiments are incomplete, see below). Overall, I am in the borderline mode, and I will gladly raise the score if the authors address my concerns regarding the experiments.
1) Experimental evaluation on Pascal VOC is not quite satisfactory. Comparison with prior work is unfair because the network architecture used by the authors (VGG) is different from the architecture used by all existing methods (AlexNet). It is great that the authors do not try to hide this fact in the paper, but I do not understand why the authors are not willing to simply run their method with AlexNet architecture, although two commenters asked them to do so. Such an experiment would strongly support authors’ claims. Current reasoning that “we thought it reasonable to use more current models while making the difference clear” is not convincing. It is great that better architectures lead to better results, but it is also very important to properly compare to prior work. On a related topic, Doersch et al. also tried using VGG architecture, would it be possible to compare to that? Yet another question: why are you not comparing to [Noroozi&Favaro, ECCV 2016] ? I would also like the authors to address the comment by Richard Zhang.
2) Qualitative inpainting results are incomplete: comparison with previous methods (for instance, [Pathak et al 2016]) is missing, and it is impossible to compare different versions of the proposed method because different images are used for different variants. I realize there may be too little space in the main paper to show all the results, but many more results should be shown in the supplementary material. Quantitative results are missing. Currently the inpainting results are just interesting pictures to look at, but they do not add as much to the paper as they could.