We present LR-GAN: an adversarial image generation model which takes scene structure and context into account. Unlike previous generative adversarial networks (GANs), the proposed GAN learns to generate image background and foregrounds separately and recursively, and stitch the foregrounds on the background in a contextually relevant manner to produce a complete natural image. For each foreground, the model learns to generate its appearance, shape and pose. The whole model is unsupervised, and is trained in an end-to-end manner with gradient descent methods. The experiments demonstrate that LR-GAN can generate more natural images with objects that are more human recognizable than DCGAN.
We present LR-GAN: an adversarial image generation model which takes scene structure and context into account. Unlike previous generative adversarial networks (GANs), the proposed GAN learns to generate image background and foregrounds separately and recursively, and stitch the foregrounds on the background in a contextually relevant manner to produce a complete natural image. For each foreground, the model learns to generate its appearance, shape and pose. The whole model is unsupervised, and is trained in an end-to-end manner with gradient descent methods. The experiments demonstrate that LR-GAN can generate more natural images with objects that are more human recognizable than DCGAN.
1 INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) have shown significant promise as generative models for natural images. A flurry of recent work has proposed improvements over the original GAN work for image generation (Radford et al., 2015; Denton et al., 2015; Salimans et al., 2016; Chen et al., 2016; Zhu et al., 2016; Zhao et al., 2016), multi-stage image generation including part-based models (Im et al., 2016; Kwak & Zhang, 2016), image generation conditioned on input text or attributes (Mansimov et al., 2015; Reed et al., 2016b;a), image generation based on 3D structure (Wang & Gupta, 2016), and even video generation (Vondrick et al., 2016).
While the holistic ‘gist’ of images generated by these approaches is beginning to look natural, there is clearly a long way to go. For instance, the foreground objects in these images tend to be deformed, blended into the background, and not look realistic or recognizable.
One fundamental limitation of these methods is that they attempt to generate images without taking into account that images are 2D projections of a 3D visual world, which has a lot of structures in it. This manifests as structure in the 2D images that capture this world. One example of this structure is that images tend to have a background, and foreground objects are placed in this background in contextually relevant ways.
We develop a GAN model that explicitly encodes this structure. Our proposed model generates images in a recursive fashion: it first generates a background, and then conditioned on the background generates a foreground along with a shape (mask) and a pose (affine transformation) that together define how the background and foreground should be composed to obtain a complete image. Conditioned on this composite image, a second foreground and an associated shape and pose are generated, and so on. As a byproduct in the course of recursive image generation, our approach generates some object-shape foreground-background masks in a completely unsupervised way, without access to any object masks for training. Note that decomposing a scene into foreground-background layers is a classical ill-posed problem in computer vision. By explicitly factorizing appearance and transformation, LR-GAN encodes natural priors about the images that the same foreground can be ‘pasted’ to the different backgrounds, under different affine transformations. According to the experiments, the absence of these priors result in degenerate foreground-background decompositions, and thus also degenerate final composite images.
∗Work was done while visiting Facebook AI Research.
We mainly evaluate our approach on four datasets: MNIST-ONE (one digit) and MNIST-TWO (two digits) synthesized from MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky & Hinton, 2009) and CUB-200 (Welinder et al., 2010). We show qualitatively (via samples) and quantitatively (via evaluation metrics and human studies on Amazon Mechanical Turk) that LR-GAN generates images that globally look natural and contain clear background and object structures in them that are realistic and recognizable by humans as semantic entities. An experimental snapshot on CUB-200 is shown in Fig. 1. We also find that LR-GAN generates foreground objects that are contextually relevant to the backgrounds (e.g., horses on grass, airplanes in skies, ships in water, cars on streets, etc.). For quantitative comparison, besides existing metrics in the literature, we propose two new quantitative metrics to evaluate the quality of generated images. The proposed metrics are derived from the sufficient conditions for the closeness between generated image distribution and real image distribution, and thus supplement existing metrics.
2 RELATED WORK
Early work in parametric texture synthesis was based on a set of hand-crafted features (Portilla & Simoncelli, 2000). Recent improvements in image generation using deep neural networks mainly fall into one of the two stochastic models: variational autoencoders (VAEs) (Kingma et al., 2016) and generative adversarial networks (GANs) (Goodfellow et al., 2014). VAEs pair a top-down probabilistic generative network with a bottom up recognition network for amortized probabilistic inference. Two networks are jointly trained to maximize a variational lower bound on the data likelihood. GANs consist of a generator and a discriminator in a minmax game with the generator aiming to fool the discriminator with its samples with the latter aiming to not get fooled.
Sequential models have been pivotal for improved image generation using variational autoencoders: DRAW (Gregor et al., 2015) uses attention based recurrence conditioning on the canvas drawn so far. In Eslami et al. (2016), a recurrent generative model that draws one object at a time to the canvas was used as the decoder in VAE. These methods are yet to show scalability to natural images. Early compelling results using GANs used sequential coarse-to-fine multiscale generation and classconditioning (Denton et al., 2015). Since then, improved training schemes (Salimans et al., 2016) and better convolutional structure (Radford et al., 2015) have improved the generation results using
GANs. PixelRNN (van den Oord et al., 2016) is also recently proposed to sequentially generates a pixel at a time, along the two spatial dimensions.
In this paper, we combine the merits of sequential generation with the flexibility of GANs. Our model for sequential generation imbibes a recursive structure that more naturally mimics image composition by inferring three components: appearance, shape, and pose. One closely related work combining recursive structure with GAN is that of Im et al. (2016) but it does not explicitly model object composition and follows a similar paradigm as by Gregor et al. (2015). Another closely related work is that of Kwak & Zhang (2016). It combines recursive structure and alpha blending. However, our work differs in three main ways: (1) We explicitly use a generator for modeling the foreground poses. That provides significant advantage for natural images, as shown by our ablation studies; (2) Our shape generator is separate from the appearance generator. This factored representation allows more flexibility in the generated scenes; (3) Our recursive framework generates subsequent objects conditioned on the current and previous hidden vectors, and previously generated object. This allows for explicit contextual modeling among generated elements in the scene. See Fig. 17 for contextually relevant foregrounds generated for the same background, or Fig. 6 for meaningful placement of two MNIST digits relative to each.
Models that provide supervision to image generation using conditioning variables have also been proposed: Style/Structure GANs (Wang & Gupta, 2016) learns separate generative models for style and structure that are then composed to obtain final images. In Reed et al. (2016a), GAN based image generation is conditioned on text and the region in the image where the text manifests, specified during training via keypoints or bounding boxes. While not the focus of our work, the model proposed in this paper can be easily extended to take into account these forms of supervision.
3 PRELIMINARIES
3.1 GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Networks (GANs) consist of a generator G and a discriminator D that are simultaneously trained with competing goals: The generator G is trained to generate samples that can ‘fool’ a discriminator D, while the discriminator is trained to classify its inputs as either real (coming from the training dataset) or fake (coming from the samples of G). This competition leads to a minmax formulation with a value function:
min θG max θD
( Ex∼pdata(x)[log(D(x; θD))] + Ez∼pz(z)[log(1−D(G(z; θG); θD))] ) , (1)
where z is a random vector from a standard multivariate Gaussian or a uniform distribution pz(z), G(z; θG) maps z to the data space, D(x) is the probability that x is real estimated by D. The advantage of the GANs formulation is that it lacks an explicit loss function and instead uses the discriminator to optimize the generative model. The discriminator, in turn, only cares whether the sample it receives is on the data manifold, and not whether it exactly matches a particular training example (as opposed to losses such as MSE). Hence, the discriminator provides a gradient signal only when the generated samples do not lie on the data manifold so that the generator can readjust its parameters accordingly. This form of training enables learning the data manifold of the training set and not just optimizing to reconstruct the dataset, as in autoencoder and its variants.
While the GANs framework is largely agnostic to the choice of G and D, it is clear that generative models with the ‘right’ inductive biases will be more effective in learning from the gradient information (Denton et al., 2015; Im et al., 2016; Gregor et al., 2015; Reed et al., 2016a; Yan et al., 2015). With this motivation, we propose a generator that models image generation via a recurrent process – in each time step of the recurrence, an object with its own appearance and shape is generated and warped according to a generated pose to compose an image in layers.
3.2 LAYERED STRUCTURE OF IMAGE
An image taken of our 3D world typically contains a layered structure. One way of representing an image layer is by its appearance and shape. As an example, an image x with two layers, foreground f and background b may be factorized as:
x = f m+ b (1−m), (2)
where m is the mask depicting the shapes of image layers, and the element wise multiplication operator. Some existing methods assume the access to the shape of the object either during training (Isola & Liu, 2013) or both at train and test time (Reed et al., 2016a; Yan et al., 2015). Representing images in layered structure is even straightforward for video with moving objects (Darrell & Pentland, 1991; Wang & Adelson, 1994; Kannan et al., 2005). Vondrick et al. (2016) generates videos by separately generating a fixed background and moving foregrounds. A similar way of generating single image can be found in Kwak & Zhang (2016).
Another way is modeling the layered structure with object appearance and pose as:
x = ST (f ,a) + b, (3)
where f and b are foreground and background, respectively; a is the affine transformation; ST is the spatial transformation operator. Several works fall into this group (Roux et al., 2011; Huang & Murphy, 2015; Eslami et al., 2016). In Huang & Murphy (2015), images are decomposed into layers of objects with specific poses in a variational autoencoder framework, while the number of objects (i.e., layers) is adaptively estimated in Eslami et al. (2016).
To contrast with these works, LR-GAN uses a layered composition, and the foreground layers simultaneously model all three dominant factors of variation: appearance f , shape m and pose a. We will elaborate it in the following section.
4 LAYERED RECURSIVE GAN (LR-GAN)
The basic structure of LR-GAN is similar to GAN: it consists of a discriminator and a generator that are simultaneously trained using the minmax formulation of GAN, as described in §.3.1. The key innovation of our work is the layered recursive generator, which is what we describe in this section.
The generator in LR-GAN is recursive in that the image is constructed recursively using a recurrent network. Layered in that each recursive step composes an object layer that is ‘pasted’ on the image generated so far. Object layer at timestep t is parameterized by the following three constituents – ‘canonical’ appearance ft, shape (or mask) mt, and pose (or affine transformation) at for warping the object before pasting in the image composition.
Fig. 2 shows the architecture of the LR-GAN with the generator architecture unrolled for generating background x0 ( . = xb) and foreground x1 and x2. At each time step t, the generator composes the next image xt via the following recursive computation:
xt = ST (mt,at)︸ ︷︷ ︸ affine transformed mask ST (ft,at)︸ ︷︷ ︸ affine transformed appearance +(1− ST (mt,at)) xt−1︸ ︷︷ ︸ pasting on image composed so far , ∀t ∈ [1, T ] (4)
where ST ( ,at) is a spatial transformation operator that outputs the affine transformed version of with at indicating parameters of the affine transformation. Since our proposed model has an explicit transformation variable at that is used to warp the object, it can learn a canonical object representation that can be re-used to generate scenes where the object occurs as mere transformations of it, such as different scales or rotations. By factorizing the appearance, shape and pose, the object generator can focus on separately capturing regularities in these three factors that constitute an object. We will demonstrate in our experiments that removing these factorizations from the model leads to its spending capacity in variability that may not solely be about the object in Section 5.5 and 5.6.
4.1 DETAILS OF GENERATOR ARCHITECTURE
Fig. 2 shows our LR-GAN architecture in detail – we use different shapes to indicate different kinds of layers (convolutional, fractional convolutional, (non)linear, etc), as indicated by the legend. Our model consists of two main pieces – a background generator Gb and a foreground generator Gf . Gb and Gf do not share parameters with each other. Gb computation happens only once, while Gf is recurrent over time, i.e., all object generators share the same parameters. In the following, we will introduce each module and connections between them.
Temporal Connections. LR-GAN has two kinds of temporal connections – informally speaking, one on ‘top’ and one on ‘bottom’. The ‘top’ connections perform the act of sequentially ‘pasting’
object layers (Eqn. 4). The ‘bottom’ connections are constructed by a LSTM on the noise vectors z0, z1, z2. Intuitively, this noise-vector-LSTM provides information to the foreground generator about what else has been generated in past. Besides, when generating multiple objects, we use a pooling layer P cf and a fully-connected layer E c f to extract the information from previous generated object response map. By this way, the model is able to ‘see’ previously generated objects.
Background Generator. The background generatorGb is purposely kept simple. It takes the hidden state of noise-vector-LSTM h0l as the input and passes it to a number of fractional convolutional layers (also called ‘deconvolution’ layer in some papers) to generate images at its end. The output of background generator xb will be used as the canvas for the following generated foregrounds.
Foreground Generator. The foreground generatorGf is used to generate an object with appearance and shape. Correspondingly, Gf consists of three sub-modules, Gcf , which is a common ‘trunk’ whose outputs are shared by Gif and G m f . G i f is used to generate the foreground appearance ft, while Gmf generates the mask mt for the foreground. All three sub-modules consists of one or more fractional convolutional layers combined with batch-normalization and nonlinear layers. The generated foreground appearance and mask have the same spatial size as the background. The top of Gmf is a sigmoid layer in order to generate one channel mask whose values range in (0, 1).
Spatial Transformer. To spatially transform foreground objects, we need to estimate the transformation matrix. As in Jaderberg et al. (2015), we predict the affine transformation matrix with a linear layer Tf that has six-dimensional outputs. Then based on the predicted transformation matrix, we use a grid generator Gg to generate the corresponding sampling coordinates in the input for each location at the output. The generated foreground appearance and mask share the same transformation matrix, and thus the same sampling grid. Given the grid, the sampler S will simultaneously sample the ft and mt to obtain f̂t and m̂t, respectively. Different from Jaderberg et al. (2015), our sampler here normally performs downsampling, since the the foreground typically has smaller size than the background. Pixels in f̂t and m̂t that are from outside the extent of ft and mt are set to zero. Finally, f̂t and m̂t are sent to the compositor C which combines the canvas xt−1 and f̂t through layered composition with blending weights given by m̂t (Eqn. 4).
Pseudo-code for our approach and detailed model configuration are provided in the Appendix.
4.2 NEW EVALUATION METRICS
Several metrics have been proposed to evaluate GANs, such as Gaussian parzen window (Goodfellow et al., 2014), Generative Adversarial Metric (GAM) (Im et al., 2016) and Inception Score (Salimans et al., 2016). The common goal is to measure the similarity between the generated data distribution Pg(x) = G(z; θz) and the real data distribution P (x). Most recently, Inception Score has been used in several works (Salimans et al., 2016; Zhao et al., 2016). However, it is an assymetric metric and could be easily fooled by generating centers of data modes.
In addition to these metrics, we present two new metrics based on the following intuition – a sufficient (but not necessary) condition for closeness of Pg(x) and P (x) is closeness of Pg(x|y) and P (x|y), i.e., distributions of generated data and real data conditioned on all possible variables of interest y, e.g., category label. One way to obtain this variable of interest y is via human annotation. Specifically, given the data sampled from Pg(x) and P (x), we ask people to label the category of the samples according to some rules. Note that such human annotation is often easier than comparing samples from the two distributions (e.g., because there is no 1:1 correspondence between samples to conduct forced-choice tests).
After the annotations, we need to verify whether the two distributions are similar in each category. Clearly, directly comparing the distributions Pg(x|y) and P (x|y) may be as difficult as comparing Pg(x) and P (x). Fortunately, we can use Bayes rule and alternatively compare Pg(y|x) and P (y|x), which is a much easier task. In this case, we can simply train a discriminative model on the samples from Pg(x) and P (x) together with the human annotations about categories of these samples. With a slight abuse of notation, we use Pg(y|x) and P (y|x) to denote probability outputs from these two classifiers (trained on generated samples vs trained on real samples). We can then use these two classifiers to compute the following two evaluation metrics:
Adversarial Accuracy: Computes the classification accuracies achieved by these two classifiers on a validation set, which can be the training set or another set of real images sampled from P (x). If Pg(x) is close to P (x), we expect to see similar accuracies.
Adversarial Divergence: Computes the KL divergence between Pg(y|x) and P (y|x). The lower the adversarial divergence, the closer two distributions are. The low bound for this metric is exactly zero, which means Pg(y|x) = P (y|x) for all samples in the validation set. As discussed above, we need human efforts to label the real and generated samples. Fortunately, we can further simplify this. Based on the labels given on training data, we split the training data into categories, and train one generator for each category. With all these generators, we generate samples of all categories. This strategy will be used in our experiments on the datasets with labels given.
5 EXPERIMENT
We conduct qualitative and quantitative evaluations on three datasets: 1) MNIST (LeCun et al., 1998); 2) CIFAR-10 (Krizhevsky & Hinton, 2009); 3) CUB-200 (Welinder et al., 2010). To add variability to the MNIST images, we randomly scale (factor of 0.8 to 1.2) and rotate (−π4 to π 4 ) the digits and then stitch them to 48 × 48 uniform backgrounds with random grayscale value between [0, 200]. Images are then rescaled back to 32 × 32. Each image thus has a different background grayscale value and a different transformed digit as foreground. We rename this sythensized dataset as MNIST-ONE (single digit on a gray background). We also synthesize a dataset MNIST-TWO containing two digits on a grayscale background. We randomly select two images of digits and perform similar transformations as described above, and put one on the left and the other on the right side of a 78× 78 gray background. We resize the whole image to 64× 64. We develop LR-GAN based on open source code1. We assume the number of objects is known. Therefore, for MNIST-ONE, MNIST-TWO, CIFAR-10, and CUB-200, our model has two, three, two, and two timesteps, respectively. Since the size of foreground object should be smaller than that of canvas, we set the minimal allowed scale 2 in affine transforamtion to be 1.2 for all datasets except for MNIST-TWO, which is set to 2 (objects are smaller in MNIST-TWO). In LR-GAN, the
1https://github.com/soumith/dcgan.torch 2Scale corresponds to the size of the target canvas with respect to the object – the larger the scale, the larger
the canvas, and the smaller the relative size of the object in the canvas. 1 means the same size as the canvas.
background generator and foreground generator have similar architectures. One difference is that the number of channels in the background generator is half of the one in the foreground generator. We compare our results to that of DCGAN (Radford et al., 2015). Note that LR-GAN without LSTM at the first timestep corresponds exactly to the DCGAN. This allows us to run controlled experiments. In both generator and discriminator, all the (fractional) convolutional layers have 4 × 4 filter size with stride 2. As a result, the number of layers in the generator and discriminator automatically adapt to the size of training images. Please see the Appendix (Section 6.2) for details about the configurations. We use three metrics for quantitative evaluation, including Inception Score (Salimans et al., 2016) and the proposed Adversarial Accuracy, Adversarial Divergence. Note that we report two versions of Inception Score. One is based on the pre-trained Inception net, and the other one is based on the pre-trained classifier on the target datasets.
5.1 QUALITATIVE RESULTS
In Fig. 3 and 4, we show the generated samples for CIFAR-10 and CUB-200, respectively. MNIST results are shown in the next subsection. As we can see from the images, the compositional nature of our model results in the images being free of blending artifacts between backgrounds and foregrounds. For CIFAR-10, we can see the horses and cars with clear shapes. For CUB-200, the bird shapes tend to be even sharper.
5.2 MNIST-ONE AND MNIST-TWO
We now report the results on MNIST-ONE and MNIST-TWO. Fig. 5 shows the generation results of our model on MNIST-ONE. As we can see, our model generates the background and the foreground in separate timestep, and can disentagle the foreground digits from background nearly perfectly. Though initial values of the mask randomly distribute in the range of (0, 1), after training, the masks are nearly binary and accurately carve out the digits from the generated foreground. More results on MNIST-ONE (including human studies) can be found in the Appendix (Section 6.3).
Fig. 6 shows the generation results for MNIST-TWO. Similarly, the model is also able to generate background and the two foreground objects separately. The foreground generator tends to generate a single digit at each timestep. Meanwhile, it captures the context information from the previous time steps. When the first digit is placed to the left side, the second one tends to be placed on the right side, and vice versa.
5.3 CUB-200
We study the effectiveness of our model trained on the CUB-200 bird dataset. In Fig. 1, we have shown a random set of generated images, along with the intermediate generation results of the model. While being completely unsupervised, the model, for a large fraction of the samples, is able to
successfully disentangle the foreground and the background. This is evident from the generated bird-like masks.
We do a comparative study based on Amazon Mechanical Turk (AMT) between DCGAN and LRGAN to quantify relative visual quality of the generated images. We first generated 1000 samples from both the models. Then, we performed perfect matching between the two image sets using the Hungarian algorithm on L2 norm distance in the pixel space. This resulted in 1000 image pairs. Some examplar pairs are shown in Fig. 7. For each image pair, 9 judges are asked to choose the one that is more realistic. Based on majority voting, we find that our generated images are selected 68.4% times, compared with 31.6% times for DCGAN. This demonstrates that our model has generated more realistic images than DCGAN. We can attribute this difference to our model’s ability to generate foreground separately from the background, enabling stronger edge cues.
5.4 CIFAR-10
We now qualitatively and quantitatively evaluate our model on CIFAR-10, which contains multiple object categories and also various backgrounds.
Comparison of image generation quality: We conduct AMT studies to compare the fidelity of image generation. Towards this goal, we generate 1000 images from DCGAN and LR-GAN, respectively. We ask 5 judges to label each image to either belong to one of the 10 categories or as ‘non recognizable’ or ‘recognizable but not belonging to the listed categories’. We then assign each image a quality level between [0,5] that captures the number of judges that agree with the majority choice. Fig. 8 shows the images generated by both approaches, ordered by increasing quality level. We merge images at quality level 0 (all judges said non-recognizable) and 1 together, and similarly images at level 4 and 5. Visually, the generated samples by our model have clearer boundaries and object structures. We also computed the fraction of non-recognizable images: Our model had a 10% absolute drop in non-recognizability rate (67.3% for ours vs. 77.7% for DCGAN). For reference, 11.4% of real CIFAR images were categorized as non-recognizable. Fig. 9 shows more generated (intermediate) results of our model.
Quantitative evaluation on generators: We evaluate the generators based on three metrics: 1) Inception Score; 2) Adversarial Accuracy; 3) Adversarial Divergence. To obtain a classifier model for evaluation, we remove the top layer in the discriminator used in our model, and then append two fully connected layers on the top of it. We train this classifier using the training samples of CIFAR-10 based on the annotations. Following Salimans et al. (2016), we generated 50,000 images
based on DCGAN and LR-GAN, repsectively. We compute two types of Inception Scores. The standard Inception Score is based on the Inception net as in Salimans et al. (2016), and the contextual Inception Score is based on our trained classifier model. To distinguish, we denote the standard one as ‘Inception Score†’, and the contextual one as ‘Inception Score††’. To obtain the Adversarial Accuracy and Adversarial Divergence scores, we train one generator on each of 10 categories for DCGAN and LR-GAN, respectively. Then, we use these generators to generate samples of different categories. Given these generated samples, we train the classifiers for DCGAN and LR-GAN separately. Along with the classifier trained on the real samples, we compute the Adversarial Accuracy
and Adversarial Divergence on the real training samples. In Table 1, we report the Inception Scores, Adversarial Accuracy and Adversarial Divergence for comparison. We can see that our model outperforms DCGAN across the board. To point out, we obtan different Inception Scores based on different classifier models, which indicates that the Inception Score varies with different models.
Quantitative evaluation on discriminators: We evaluate the discriminator as an extractor for deep representations. Specifically, we use the output of the last convolutional layer in the discriminator as features. We perform a 1-NN classification on the test set given the full training set. Cosine similarity is used as the metric. On the test set, our model achieves 62.09%±0.01% compared to DCGAN’s 56.05%±0.02%. Contextual generation: We also show the efficacy of our approach to generate diverse foregrounds conditioned on fixed background. The results in Fig. 17 in Appendix showcase that the foreground generator generates objects that are compatible with the background. This indicates that the model has captured contextual dependencies between the image layers.
Category specific models: The objects in CIFAR-10 exhibit huge variability in shapes. That can partly explain why some of the generated shapes are not as compelling in Fig. 9. To test this hypothesis, we reuse the generators trained for each of 10 categories used in our metrics to obtain the generation results. Fig. 10 shows results for categories ‘horse’, ‘frog’ and ‘cat’. We can see that the model is now able to generate object-specific appearances and shapes, similar in vein to our results on the CUB-200 dataset.
5.5 IMPORTANCE OF TRANSFORMATIONS
Fig. 11 shows results from an ablated model without affine transformations in the foreground layers, and compares the results with the full model that does include these transformations. We note that one significant problem emerges that the decompositions are degenerate, in the sense that the model is unable to break the symmetry between foreground and background layers, often generating object appearances in the model’s background layer and vice versa. For CUB-200, the final generated images have some blendings between foregrounds and backgrounds. This is particularly the case for
those images without bird-shape masks. For CIFAR-10, a number of generated masks are inverted. In this case, the background images are carved out as the foreground objects. The foreground generator takes almost all the duty to generate the final images, which make it harder to generate images as clear as the model with transformation. From these comparisons, we qualitatively demonstrate the importance of modeling transformations in the foreground generation process. Another merit of using transformation is that the intermediate outputs of the model are more interpretable and faciliate to the downstreaming tasks, such as scene paring, which is demonstrated in Section 6.8.
5.6 IMPORTANCE OF SHAPES
We perform another ablation study by removing the mask generator to understand the importance of modeling object shapes. In this case, the generated foreground is simply pasted on top of the generated background after being transformed. There is no alpha blending between the foregrounds and backgrounds. The generation results for three datasets, MNIST-ONE, CUB-200, CIFAR-10 are shown in Fig. 12. As we can see, though the model works well for the generation of MNIST-ONE, it fails to generate reasonable images across the other datasets. Particularly, the training does not even converge for CUB-200. Based on these results, we qualitatively demonstrate that mask generator in our model is fairly important to obtain plausible results, especially for realistic images.
6 APPENDIX
6.1 ALGORITHM
Algo. 1 illustrates the generative process in our model. g(?) evaluates the function g at ?. ◦ is a composition operator that composes its operands so that f ◦ g(?) = f(g(?)).
Algorithm 1 Stochastic Layered Recursive Image Generation 1: z0 ∼ N (0, I) 2: x0 = Gb(z0) . background generator 3: h0l ← 0 4: c0l ← 0 5: for t ∈ [1 · · ·T ] do 6: zt ∼ N (0, I) 7: htl , c t l ← LSTM([zt, h t−1 l , c t−1 l ]) . pass through LSTM
8: if t = 1 then 9: yt← htl
10: else 11: yt← Elf ([htl h t−1 f ]) . pass through non-linear embedding layers E l f 12: end if 13: st← Gcf (yt) . predict shared cube for Gif and Gmf 14: at ← Tf (yt) . object transformation 15: ft← Gif (st) . generate object appearance 16: mt← Gmf (st) . generate object shape 17: htf ← Ecf ◦ P cf (st) . predict shared represenation embedding 18: xt← ST (mt,at) ST (ft,at) + (1− ST (mt,at)) xt−1 19: end for
6.2 MODEL CONFIGURATIONS
Table 2 lists the information and model configuration for different datasets. The dimensions of random vectors and hidden vectors are all set to 100. We also compare the number of parameters in DCGAN and LR-GAN. The numbers before ‘/’ are our model, after ‘/’ are DCGAN. Based on the same notation used in (Zhao et al., 2016), the architectures for the different datasets are:
6.3 RESULTS ON MNIST-ONE
We conduct human studies on generation results on MNIST-ONE. Specifically, we generate 1,000 images using both LR-GAN and DCGAN. As references, we also include 1000 real images. Then we ask the users on AMT to label each image to be one of the digits (0-9). We also provide them an option ‘non recognizable’ in case the generated image does not seem to contain a digit. Each image was judged by 5 unique workers. Similar to CIFAR-10, if an image is recognized to be the same digit by all 5 users, it is assigned to quality level 5. If it is not recognizable according to all users, it is assigned to quality level 0. Fig. 13 (left) shows the number of images assigned to all six quality levels. Compared to DCGAN, our model generated more samples with high quality levels. As expected, the real images have many samples with high quality levels. In Fig. 13 (right), we show the number of images that are recognized to each digit category (0-9). For qualitative comparison, we show examplar images at each quality level in Fig. 14. From left to right, the quality level increases from 0 to 5. As expected, the images with higher quality level are more clear.
For quantitative evaluation, we use the same way as for CIFAR-10. The classifier model used for contextual Inception Score is trained based on the training set. We generate 60,000 samples based on DCGAN and LR-GAN for evaluation, respectively. To obtain the Adversarial Accuracy and Adversarial Divergence, we first train 10 generators for 10 digit categories separately, and then use the generated samples to train the classifier. As shown in Table 3, our model has higher scores than DCGAN on both standard and contextual Inception Score. Also, our model has a slightly higher
adversarial accuracy, and lower adversarial divergence than DCGAN. We find that the all three image sets have low standard Inception Scores. This is mainly because the Inception net is trained on ImageNet, which has a very different data distribution from the MNIST dataset. Based on this, we argue that the standard Inception Score is not suitable for some image datasets.
6.4 MORE RESULTS ON CUB-200
In this experiment, we reduce the minimal allowed object scale to 1.1, which allows the model to generate larger foreground objects. The results are shown in Fig. 15. Similar to the results when the constraint is 1.2, the crisp bird-like masks are generated automatically by our model.
6.5 MORE RESULTS ON CIFAR-10
6.5.1 QUALITATIVE RESULTS
In Fig. 16, we show more results on CIFAR-10 when setting minimal allowed object scale to 1.1. The rightmost column block also shows the training images that are closest to the generated images (cosine similarity in pixel space). We can see our model does not memorize the training data.
6.5.2 WALKING IN THE LATENT SPACE
Similar to DCGAN, we also show results by walking in the latent space. Note that our model has two or more inputs. So we can walk along any of them or their combination. In Fig. 17, we generate multiple foregrounds for the same fixed generated background. We find that our model consistently generates contextually compatible foregrounds. For example, for the grass-like backgrounds, the foreground generator generates horses and deer, and airplane-like objects for the blue sky.
6.5.3 WORD CLOUD BASED ON HUMAN STUDY
As we mentioned above, we conducted human studies on CIFAR-10. Besides asking people to select a name from a list for an image, we also conducted another human study where we ask people to use one word (free-form) to describe the main object in the image. Each image was ‘named’ by 5 unique people. We generate word clouds for real images, images generated by DCGAN and LR-GAN, as shown in Fig. 18.
6.6 RESULTS ON LFW FACE DATASET
We conduct experiment on face images in LFW dataset (Huang et al., 2007). Different from previous works which work on cropped and aligned faces, we directly generate the original images which contains a large portion of backgrounds. This configuration helps to verify the efficiency of LR-GAN to model the object appearance, shape and pose. In Fig. 19, we show the (intermediate) generation results of LR-GAN. Surprisingly, without any supervisions, the model generated background and faces in separate steps, and the generated masks accurately depict face shapes. Moreover, the model
learns where to place the generated faces so that the whole image looks natural. For comparison, please refer to (Kwak & Zhang, 2016) which does not model the transformation. We can find the generation results degrade much.
6.7 STATISTICS ON TRANSFORMATION MATRICES
In this part, we analyze the statistics on the transformation matrices generated by our model for different datasets, including MNIST-ONE, CUB-200, CIFAR-10 and LFW. We used affine transformation in our model. So there are 6 parameters, scaling in the x coordinate (sx), scaling in the y coordinate (sy), translation in the x coordinate (tx), translation in the y coordinate (ty), rotation in the x coordinate (rx) and rotation in the y coordinate (ry). In Fig. 20, we show the histograms on different parameters for different datasets.These histograms show that the model produces non-trivial varied scaling, translation and rotation on all datasets. For different datasets, the learned transformation have different patterns. We hypothesize that this is mainly determined by the configurations of objects in the images. For example, on MNIST-ONE, all six parameters have some fluctuations since the synthetic dataset contains digits randomly placed at different locations. For the other three datasets, the scalings converge to single value since the object sizes do not vary much, and the variations on rotation and translation suffice to generate realistic images. Specifically, we can find the generator largely relies on the translation on x coordinate for generating CUB-200. This makes sense since birds in the images have similar scales, orientations but various horizontal locations. For CIFAR-10, since there are 10 different object categories, the configurations are more diverse, hence the generator uses all parameters for generation except for the scaling. For LFW, since faces have similar configurations, the learned transformations have less fluctuation as well. As a result, we can see that LR-GAN indeed models the transformations on the foreground to generate images.
6.8 CONDITIONAL IMAGE GENERATION
Considering our model can generate object-like masks (shapes) for images, we conducted an experiment to evaluate whether our model can be potentially used for image segmentation and object detection. We make some changes to the model. For the background generator, the input is a real image instead of a random vector. Then the image is passed through an encoder to extract the hidden features, which replaces the random vector z0 and are fed to the background generator. For the foreground generator, we subtract the image generated by the background generator from the input image to obtain a residual image. Then this residual image is fed to the same encoder to get the hidden features, which are used as the input for foreground generator. In our conditional model, we want to reconstruct the image, so we add a reconstruction loss along with the adversarial loss. We train this conditional model on CIFAR-10. The (intermediate) outputs of the model is shown in Fig. 21. Interestingly, the model successfully learned to decompose the input images into background and foreground. The background generator tends to do an image inpainting by generating a complete background without object, while the foreground generator works as a segmentation model to get object mask from the input image.
Similarly, we also run the conditional LR-GAN on LFW dataset. As we can see in Fig. 22, the foreground generator automatically and consistently learned to generate the face regions, even though there are large portion of background in the input images. In other words, the conditional LR-GAN successfully learned to detection faces in images. We suspect this success is due to that it has low cost for the generator to generate similar images, and thus converge to the case that the first generator generate background, and the second generator generate face images.
Based on these experiments, we argue that our model can be possibly used for image segmentation and object detection in a generative and unsupervised manner. One future work would be verifying this by applying it to high-resolution and more complicate datasets.

The authors propose a method that generates naturally looking images by first generating the background and then conditioned on the previous layer one or multiple foreground objects. Additionally they add a image transformer layer that allows the model to more easily model different appearances.
I would like to see some discussion about the choice of foreground+mask rather than just predicting foreground directly. For MNIST, for example the foreground seems completely irrelevant. For CUB and CIFAR of course the fg adds the texture and color while the masks ensures a crisp boundary.
- Is the mask a binary mask or a alpha blending mask?
- I find the fact that the model learns to decompose images this nicely and learns to produce crisp foreground masks w/o too much spurious elements (though there are some in CIFAR) pretty fascinating.
The proposed evaluation metric makes sense and seems reasonable. However, AFAICT, theoretically it would be possible to get a high score even though the GAN produces images not recognizable to humans, but only to the classifier network that produces P_g. E.g. if the Generator encodes the class in some subtle way (though this shouldn't happen given the training with an adversarial network).
Fig 3 shows indeed nicely that the decomposition is much nicer when spatial transformers are used. However, it also seems to indicate that the foreground prediction and the foreground mask are largely redundant. For the final results the "niceness" of the decomposition appears to be largely irrelevant.
Furthermore, the transformation layer seems to have a small effect, judging from the transformed masked foreground objects. They are mainly scaled down.
- What is the 3rd & 6th column in Fig 9? It is not clear if the final composed images are really as bad as "advertised".
Regarding the eval experiment using AMT it is not clear why it is better to provide the users with L2 minimized NN matches rather than random pairs.
I assume that Tab 1 Adversarial Divergence for Real images was not actually evaluated? It would be interesting to see how close to 0 multiple differently initialized networks actually are. Also please mention how the confidences/std where generated, i.e. different training sets, initialisations, eval sets, and how many runs.


END-TO-END OPTIMIZED IMAGE COMPRESSION
We describe an image compression method, consisting of a nonlinear analysis transformation, a uniform quantizer, and a nonlinear synthesis transformation. The transforms are constructed in three successive stages of convolutional linear filters and nonlinear activation functions. Unlike most convolutional neural networks, the joint nonlinearity is chosen to implement a form of local gain control, inspired by those used to model biological neurons. Using a variant of stochastic gradient descent, we jointly optimize the entire model for rate–distortion performance over a database of training images, introducing a continuous proxy for the discontinuous loss function arising from the quantizer. Under certain conditions, the relaxed loss function may be interpreted as the log likelihood of a generative model, as implemented by a variational autoencoder. Unlike these models, however, the compression model must operate at any given point along the rate– distortion curve, as specified by a trade-off parameter. Across an independent set of test images, we find that the optimized method generally exhibits better rate–distortion performance than the standard JPEG and JPEG 2000 compression methods. More importantly, we observe a dramatic improvement in visual quality for all images at all bit rates, which is supported by objective quality estimates using MS-SSIM.
1 INTRODUCTION
Data compression is a fundamental and well-studied problem in engineering, and is commonly formulated with the goal of designing codes for a given discrete data ensemble with minimal entropy (Shannon, 1948). The solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (such as vectors of image pixel intensities) must be quantized to a finite set of discrete values, which introduces error. In this context, known as the lossy compression problem, one must trade off two competing costs: the entropy of the discretized representation (rate) and the error arising from the quantization (distortion). Different compression applications, such as data storage or transmission over limited-capacity channels, demand different rate–distortion trade-offs.
Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is intractable (Gersho and Gray, 1992). For this reason, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-valued representation, quantizing its elements independently, and then encoding the resulting discrete representation using a lossless entropy code (Wintz, 1972; Netravali and Limb, 1980). This scheme is called transform coding due to the central role of the transforma-
∗JB and EPS are supported by the Howard Hughes Medical Institute.
tion. For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG 2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of transform coding methods – transform, quantizer, and entropy code – are separately optimized (often through manual parameter adjustment).
We have developed a framework for end-to-end optimization of an image compression model based on nonlinear transforms (figure 1). Previously, we demonstrated that a model consisting of linear– nonlinear block transformations, optimized for a measure of perceptual distortion, exhibited visually superior performance compared to a model optimized for mean squared error (MSE) (Ballé, Laparra, and Simoncelli, 2016). Here, we optimize for MSE, but use a more flexible transforms built from cascades of linear convolutions and nonlinearities. Specifically, we use a generalized divisive normalization (GDN) joint nonlinearity that is inspired by models of neurons in biological visual systems, and has proven effective in Gaussianizing image densities (Ballé, Laparra, and Simoncelli, 2015). This cascaded transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which effectively implements a parametric form of vector quantization on the original image space. The compressed image is reconstructed from these quantized values using an approximate parametric nonlinear inverse transform.
For any desired point along the rate–distortion curve, the parameters of both analysis and synthesis transforms are jointly optimized using stochastic gradient descent. To achieve this in the presence of quantization (which produces zero gradients almost everywhere), we use a proxy loss function based on a continuous relaxation of the probability model, replacing the quantization step with additive uniform noise. The relaxed rate–distortion optimization problem bears some resemblance to those used to fit generative image models, and in particular variational autoencoders (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014), but differs in the constraints we impose to ensure that it approximates the discrete problem all along the rate–distortion curve. Finally, rather than reporting differential or discrete entropy estimates, we implement an entropy code and report performance using actual bit rates, thus demonstrating the feasibility of our solution as a complete lossy compression method.
2 CHOICE OF FORWARD, INVERSE, AND PERCEPTUAL TRANSFORMS
Most compression methods are based on orthogonal linear transforms, chosen to reduce correlations in the data, and thus to simplify entropy coding. But the joint statistics of linear filter responses exhibit strong higher order dependencies. These may be significantly reduced through the use of joint local nonlinear gain control operations (Schwartz and Simoncelli, 2001; Lyu, 2010; Sinz and Bethge, 2013), inspired by models of visual neurons (Heeger, 1992; Carandini and Heeger, 2012). Cascaded versions of such models have been used to capture multiple stages of visual transformation
(Simoncelli and Heeger, 1998; Mante, Bonin, and Carandini, 2008). Some earlier results suggest that incorporating local normalization in linear block transform coding methods can improve coding performance (Malo et al., 2006), and can improve object recognition performance of cascaded convolutional neural networks (Jarrett et al., 2009). However, the normalization parameters in these cases were not optimized for the task. Here, we make use of a generalized divisive normalization (GDN) transform with optimized parameters, that we have previously shown to be highly efficient in Gaussianizing the local joint statistics of natural images, much more so than cascades of linear transforms followed by pointwise nonlinearities (Ballé, Laparra, and Simoncelli, 2015).
Note that some training algorithms for deep convolutional networks incorporate “batch normalization”, rescaling the responses of linear filters in the network so as to keep it in a reasonable operating range (Ioffe and Szegedy, 2015). This type of normalization is different from local gain control in that the rescaling factor is identical across all spatial locations. Moreover, once the training is completed, the scaling parameters are typically fixed, which turns the normalization into an affine transformation with respect to the data – unlike GDN, which is spatially adaptive and can be highly nonlinear.
Specifically, our analysis transform ga consists of three stages of convolution, subsampling, and divisive normalization. We represent the ith input channel of the kth stage at spatial location (m,n) as u(k)i (m,n). The input image vector x corresponds to u (0) i (m,n), and the output vector y is u (3) i (m,n). Each stage then begins with an affine convolution:
v (k) i (m,n) = ∑ j ( hk,ij ∗ u(k)j ) (m,n) + ck,i, (1)
where ∗ denotes 2D convolution. This is followed by downsampling:
w (k) i (m,n) = v (k) i (skm, skn), (2)
where sk is the downsampling factor for stage k. Each stage concludes with a GDN operation:
u (k+1) i (m,n) = w (k) i (m,n)( βk,i + ∑ j γk,ij ( w (k) j (m,n) )2) 12 . (3) The full set of h, c, β, and γ parameters (across all three stages) constitute the parameter vector φ to be optimized.
Analogously, the synthesis transform gs consists of three stages, with the order of operations reversed within each stage, downsampling replaced by upsampling, and GDN replaced by an approximate inverse we call IGDN (more details in the appendix). We define û(k)i (m,n) as the input to the kth synthesis stage, such that ŷ corresponds to û(0)i (m,n), and x̂ to û (3) i (m,n). Each stage then consists of the IGDN operation:
ŵ (k) i (m,n) = û (k) i (m,n) · ( β̂k,i + ∑ j γ̂k,ij ( û (k) j (m,n) )2) 12 , (4)
which is followed by upsampling:
v̂ (k) i (m,n) =
{ ŵ
(k) i (m/ŝk, n/ŝk) if m/ŝk and n/ŝk are integers,
0 otherwise, (5)
where ŝk is the upsampling factor for stage k. Finally, this is followed by an affine convolution:
û (k+1) i (m,n) = ∑ j ( ĥk,ij ∗ v̂(k)j ) (m,n) + ĉk,i. (6)
Analogously, the set of ĥ, ĉ, β̂, and γ̂ make up the parameter vector θ. Note that the down/upsampling operations can be implemented jointly with their adjacent convolution, improving computational efficiency.
In previous work, we used a perceptual transform gp, separately optimized to mimic human judgements of grayscale image distortions (Laparra et al., 2016), and showed that a set of one-stage transforms optimized for this distortion measure led to visually improved results (Ballé, Laparra, and Simoncelli, 2016). Here, we set the perceptual transform gp to the identity, and use mean squared error (MSE) as the metric (i.e., d(z, ẑ) = ‖z − ẑ‖22). This allows a more interpretable comparison to existing methods, which are generally optimized for MSE, and also allows optimization for color images, for which we do not currently have a reliable perceptual metric.
3 OPTIMIZATION OF NONLINEAR TRANSFORM CODING MODEL
Our objective is to minimize a weighted sum of the rate and distortion, R+λD, over the parameters of the analysis and synthesis transforms and the entropy code, where λ governs the trade-off between the two terms (figure 2, left panel). Rather than attempting optimal quantization directly in the image space, which is intractable due to the high dimensionality, we instead assume a fixed uniform scalar quantizer in the code space, and aim to have the nonlinear transformations warp the space in an appropriate way, effectively implementing a parametric form of vector quantization (figure 1). The actual rates achieved by a properly designed entropy code are only slightly larger than the entropy (Rissanen and Langdon, 1981), and thus we define the objective functional directly in terms of entropy:
L[ga, gs, Pq] = −E [ log2 Pq ] + λE [ d(z, ẑ) ] , (7)
where both expectations will be approximated by averages over a training set of images. Given a powerful enough set of transformations, we can assume without loss of generality that the quantization bin size is always one and the representing values are at the centers of the bins. That is,
ŷi = qi = round(yi), (8) where index i runs over all elements of the vectors, including channels and spatial locations. The marginal density of ŷi is then given by a train of discrete probability masses (Dirac delta functions, figure 2, right panel) with weights equal to the probability mass function of qi:
Pqi(n) = ∫ n+ 12 n− 12 pyi(t) dt, for all n ∈ Z . (9)
Note that both terms in (7) depend on the quantized values, and the derivatives of the quantization function (8) are zero almost everywhere, rendering gradient descent ineffective. To allow optimization via stochastic gradient descent, we replace the quantizer with an additive i.i.d. uniform noise
source ∆y, which has the same width as the quantization bins (one). This relaxed formulation has two desirable properties. First, the density function of ỹ = y+ ∆y is a continuous relaxation of the probability mass function of q (figure 2, right panel):
pỹ(n) = Pq(n), for all n ∈ ZM , (10) which implies that the differential entropy of ỹ can be used as an approximation of the entropy of q. Second, independent uniform noise approximates quantization error in terms of its marginal moments, and is frequently used as a model of quantization error (Gray and Neuhoff, 1998). We can thus use the same approximation for our measure of distortion. We examine the empirical quality of these rate and distortion approximations in section 4.
We assume independent marginals in the code space for both the relaxed probability model of ỹ and the entropy code, and model the marginals pỹi non-parametrically to reduce model error. Specifically, we use finely sampled piecewise linear functions which we update similarly to onedimensional histograms (see appendix). Since pỹi = pyi ∗ U(0, 1) is effectively smoothed by a box-car filter – the uniform density on the unit interval, U(0, 1) – the model error can be made arbitrarily small by decreasing the sampling interval.
Given this continuous approximation of the quantized coefficient distribution, the loss function for parameters θ and φ can be written as:
L(θ,φ) = Ex,∆y [ − ∑ i log2 pỹi(ga(x;φ) + ∆y;ψ (i))
+ λ d ( gp ( gs(ga(x;φ) + ∆y;θ) ) , gp(x) )] . (11)
where vector ψ(i) parameterizes the piecewise linear approximation of pỹi (trained jointly with θ and ϕ). This is continuous and differentiable, and thus well-suited for stochastic optimization.
3.1 RELATIONSHIP TO VARIATIONAL GENERATIVE IMAGE MODELS
We derived our formulation directly from the classical rate–distortion optimization problem. However, once the transition to a continuous loss function is made, the optimization problem resembles those encountered in fitting generative models of images, and can more specifically be cast in the context of variational autoencoders (Kingma and Welling, 2014; Rezende, Mohamed, and Wierstra, 2014). In Bayesian variational inference, we are given an ensemble of observations of a random variable x along with a generative model px|y(x|y). We seek to find a posterior py|x(y|x), which generally cannot be expressed in closed form. The approach followed by Kingma and Welling (2014) consists of approximating this posterior with a density q(y|x), by minimizing the Kullback– Leibler divergence between the two:
DKL [ q‖py|x ] = Ey∼q log q(y|x)− Ey∼q log py|x(y|x) = Ey∼q log q(y|x)− Ey∼q log px|y(x|y)− Ey∼q log py(y) + const. (12)
This objective function is equivalent to our relaxed rate–distortion optimization problem, with distortion measured as MSE, if we define the generative model as follows (figure 3):
px|ỹ(x|ỹ;λ,θ) = N ( x; gs(ỹ;θ), (2λ) −11 ) , (13)
pỹ(ỹ;ψ (0),ψ(1), . . . ) = ∏ i pỹi(ỹi;ψ (i)), (14)
and the approximate posterior as follows:
q(ỹ|x;φ) = ∏ i U(ỹi; yi, 1) with y = ga(x;φ), (15)
where U(ỹi; yi, 1) is the uniform density on the unit interval centered on yi. With this, the first term in the Kullback–Leibler divergence is constant; the second term corresponds to the distortion, and the third term corresponds to the rate (both up to additive constants). Note that if a perceptual transform gp is used, or the metric d is not Euclidean, px|ỹ is no longer Gaussian, and equivalence to variational autoencoders cannot be guaranteed, since the distortion term may not correspond to a normalizable density. For any affine and invertible perceptual transform and any translation-invariant metric, it can be shown to correspond to the density
px|ỹ(x|ỹ;λ,θ) = 1
Z(λ) exp
( −λ d ( gp ( gs(ỹ;θ) ) , gp(x) )) , (16)
where Z(λ) normalizes the density (but need not be computed to fit the model).
Despite the similarity between our nonlinear transform coding framework and that of variational autoencoders, it is worth noting several fundamental differences. First, variational autoencoders are continuous-valued, and digital compression operates in the discrete domain. Comparing differential entropy with (discrete) entropy, or entropy with an actual bit rate, can potentially lead to misleading results. In this paper, we use the continous domain strictly for optimization, and perform the evaluation on actual bit rates, which allows comparison to existing image coding methods. We assess the quality of the rate and distortion approximations empirically.
Second, generative models aim to minimize differential entropy of the data ensemble under the model, i.e., explaining fluctuations in the data. This often means minimizing the variance of a “slack” term like (13), which in turn maximizes λ. Transform coding methods, on the other hand, are optimized to achieve the best trade-off between having the model explain the data (which increases rate and decreases distortion), and having the slack term explain the data (which decreases rate and increases distortion). The overall performance of a compression model is determined by the shape of the convex hull of attainable model distortions and rates, over all possible values of the model parameters. Finding this convex hull is equivalent to optimizing the model for particular values of λ (see figure 2). In contrast, generative models operate in a regime where λ is inferred and ideally approaches infinity for noiseless data, which corresponds to the regime of lossless compression. Even so, lossless compression methods still need to operate in a discretized space, typically directly on quantized luminance values. For generative models, the discretization of luminance values is usually considered a nuisance (Theis, van den Oord, and Bethge, 2015), although there are examples of generative models that operate on quantized pixel values (van den Oord, Kalchbrenner, and Kavukcuoglu, 2016).
Finally, although correspondence between the typical slack term (13) of a generative model (figure 3, left panel) and the distortion metric in rate–distortion optimization holds for simple metrics (e.g., Euclidean distance), a more general perceptual measure would be considered a peculiar choice from a generative modeling perspective, if it corresponds to a density at all.
4 EXPERIMENTAL RESULTS
We jointly optimized the full set of parameters φ, θ, and all ψ over a subset of the ImageNet database (Deng et al., 2009) consisting of 6507 images using stochastic descent. This optimization was performed separately for each λ, yielding separate transforms and marginal probability models for each value.
For the grayscale analysis transform, we used 128 filters (size 9 × 9) in the first stage, each subsampled by a factor of 4 vertically and horizontally. The remaining two stages retain the number of channels, but use filters operating across all input channels (5× 5× 128), with outputs subsampled by a factor of 2 in each dimension. The net output thus has half the dimensionality of the input. The synthesis transform is structured analogously. For RGB images, we trained a separate set of models, with the first stage augmented to operate across three (color) input channels. For the two largest values of λ, and for RGB models, we increased the network capacity by increasing the number of channels in each stage to 256 and 192, respectively. Further details about the parameterization of the transforms and their training can be found in the appendix.
We first verified that the continuously-relaxed loss function given in section 3 provides a good approximation to the actual rate–distortion values obtained with quantization (figure 4). The relaxed distortion term appears to be mostly unbiased, and exhibits a relatively small variance. The relaxed (differential) entropy provides a somewhat positively biased estimate of the discrete entropy for the coarser quantization regime, but the bias disappears for finer quantization, as expected. Note that since the values of λ do not have any intrinsic meaning, but serve only to map out the convex hull of optimal points in the rate–distortion plane (figure 2, left panel), a constant bias in either of the terms would simply alter the effective value of λ, with no effect on the compression performance.
We compare the rate–distortion performance of our method to two standard methods: JPEG and JPEG 2000. For our method, all images were compressed using uniform quantization (the continuous relaxation using additive noise was used only for training purposes). To make the comparisons more fair, we implemented a simple entropy code based on the context-based adaptive binary arithmetic coding framework (CABAC; Marpe, Schwarz, and Wiegand, 2003). All sideband information needed by the decoder (size of images, value of λ, etc.) was included in the bit stream (see appendix). Note that although the computational costs for training our models are quite high, encoding or decoding an image with the trained models is efficient, requiring only execution of the optimized analysis transformation and quantizer, or the synthesis transformation, respectively. Evaluations were performed on the Kodak image dataset1, an uncompressed set of images commonly used to evaluate image compression methods. We also examined a set of relatively standard (if outdated) images used by the compression community (known by the names “Lena”, “Barbara”, “Peppers”, and “Mandrill”) as well as a set of our own digital photographs. None of these test images was included in the training set. All test images, compressed at a variety of bit rates using all three methods, along with their associated rate–distortion curves, are available online at http://www.cns.nyu.edu/˜lcv/iclr2017.
1Downloaded from http://www.cipr.rpi.edu/resource/stills/kodak.html
Although we used MSE as a distortion metric for training, the appearance of compressed images is both qualitatively different and substantially improved, compared to JPEG and JPEG 2000. As an example, figure 5 shows an image compressed using our method optimized for a low value of λ (and thus, a low bit rate), compared to JPEG/JPEG 2000 images compressed at equal or greater bit rates. The image compressed with our method has less detail than the original (not shown, but available online), with fine texture and other patterns often eliminated altogether, but this is accomplished in a way that preserves the smoothness of contours and sharpness of many of the edges, giving them a natural appearance. By comparison, the JPEG and JPEG 2000 images exhibit artifacts that are common to all linear transform coding methods: since local features (edges, contours, texture elements,
etc.) are represented using particular combinations of localized linear basis functions, independent scalar quantization of the transform coefficients causes imbalances in these combinations, and leads to visually disturbing blocking, aliasing, and ringing artifacts that reflect the underlying basis functions.
Remarkably, we find that the perceptual advantages of our method hold for all images tested, and at all bit rates. The progression from high to low bit rates is shown for an example image in figure 6 (additional examples provided in appendix and online). As bit rate is reduced, JPEG and JPEG 2000 degrade their approximation of the original image by coarsening the precision of the coefficients of linear basis functions, thus exposing the visual appearance of those basis functions. On the other hand, our method appears to progressively simplify contours and other image features, effectively concealing the underlying quantization of the representation. Consistent with the appearance of these example images, we find that distortion measured with a perceptual metric (MS-SSIM; Wang, Simoncelli, and Bovik, 2003), indicates substantial improvements across all tested images and bit rates (figure 7; additional examples provided in the appendix and online). Finally, when quantified with PSNR, we find that our method exhibits better rate–distortion performance than both JPEG and JPEG 2000 for most (but not all) test images, especially at the lower bit rates.
5 DISCUSSION
We have presented a complete image compression method based on nonlinear transform coding, and a framework to optimize it end-to-end for rate–distortion performance. Our compression method offers improvements in rate–distortion performance over JPEG and JPEG 2000 for most images and bit rates. More remarkably, although the method was optimized using mean squared error as a distortion metric, the compressed images are much more natural in appearance than those compressed with JPEG or JPEG 2000, both of which suffer from the severe artifacts commonly seen in linear transform coding methods. Consistent with this, perceptual quality (as estimated with the MS-SSIM index) exhibits substantial improvement across all test images and bit rates. We believe this visual improvement arises because the cascade of biologically-inspired nonlinear transformations in the model have been optimized to capture the features and attributes of images that are represented in the statistics of the data, parallel to the processes of evolution and development that are believed to have shaped visual representations within the human brain (Simoncelli and Olshausen, 2001). Nevertheless, additional visual improvements might be possible if the method were optimized using a perceptual metric in place of MSE (Ballé, Laparra, and Simoncelli, 2016).
For comparison to linear transform coding methods, we can interpret our analysis transform as a single-stage linear transform followed by a complex vector quantizer. As in many other optimized representations – e.g., sparse coding (Lewicki and Olshausen, 1998) – as well as many engineered representations – e.g., the steerable pyramid (Simoncelli, Freeman, et al., 1992), curvelets (Candès and Donoho, 2002), and dual-tree complex wavelets (Selesnick, Baraniuk, and Kingsbury, 2005) – the filters in this first stage are localized and oriented and the representation is overcomplete. Whereas most transform coding methods use complete (often orthogonal) linear transforms with spatially separable filters, the overcompleteness and orientation tuning of our initial transform may explain the ability of the model to better represent features and contours with continuously varying orientation, position and scale (Simoncelli, Freeman, et al., 1992).
Our work is related to two previous publications that optimize image representations with the goal of image compression. Gregor, Besse, et al. (2016) introduce an interesting hierarchical representation of images, in which degradations are more natural looking than those of linear representations. However, rather than optimizing directly for rate–distortion performance, their modeling is generative. Due to the differences between these approaches (as outlined in section 3.1), their procedure of obtaining coding representations from the generative model (scalar quantization, and elimination of hierarchical levels of the representation) is less systematic than our approach and unlikely to be optimal. Further, no entropy code is provided, and the authors therefore resort to comparing entropy estimates to bit rates of established compression methods, which can be unreliable. The model developed by Toderici et al. (2016) is optimized to provide various rate–distortion trade-offs and directly output a binary representation, making it more easily comparable to other image compression methods. Moreover, their formulation has the advantage over ours that a single representation is sought for all rate points. However, it is not clear whether their formulation necessarily leads to rate–distortion optimality (and their empirical results suggest that this is not the case).
We are currently testing models that use simpler rectified-linear or sigmoidal nonlinearities, to determine how much of the performance and visual quality of our results is due to use of biologicallyinspired joint nonlinearities. Preliminary results indicate that qualitatively similar results are achievable with other activation functions we tested, but that rectified linear units generally require a substantially larger number of model parameters/stages to achieve the same rate–distortion performance as the GDN/IGDN nonlinearities. This suggests that GDN/IGDN transforms are more efficient for compression, producing better models with fewer stages of processing (as we previously found for density estimation; Ballé, Laparra, and Simoncelli, 2015), which might be an advantage for deployment of our method, say, in embedded systems. However, such conclusions are based on a somewhat limited set of experiments and should at this point be considered provisional. More generally, GDN represents a multivariate generalization of a particular type of sigmoidal function. As such, the observed efficiency advantage relative to pointwise nonlinearities is expected, and a variant of a universal function approximation theorem (e.g., Leshno et al., 1993) should hold.
The rate–distortion objective can be seen as a particular instantiation of the general unsupervised learning or density estimation problems. Since the transformation to a discrete representation may be viewed as a form of classification, it is worth considering whether our framework offers any insights that might be transferred to more specific supervised learning problems, such as object recognition. For example, the additive noise used in the objective function as a relaxation of quantization might also serve the purpose of making supervised classification networks more robust to small perturbations, and thus allow them to avoid catastrophic “adversarial” failures that have been demonstrated in previous work (Szegedy et al., 2013). In any case, our results provide a strong example of the power of end-to-end optimization in achieving a new solution to a classical problem.
ACKNOWLEDGMENTS
We thank Olivier Hénaff and Matthias Bethge for fruitful discussions.
6 APPENDIX
6.1 NETWORK ARCHITECTURE AND OPTIMIZATION
As described in the main text, our analysis transform consists of three stages of convolution, downsampling, and GDN. The number and size of filters, downsampling factors, and connectivity from layer to layer are provided in figure 8 for the grayscale transforms. The transforms for RGB images and for high bit rates differ slightly in that they have an increased number of channels in each stage. These choices are somewhat ad-hoc, and a more thorough exploration of alternative architectures could potentially lead to significant performance improvements.
We have previously shown that GDN is highly efficient in Gaussianizing the local joint statistics of natural images (Ballé, Laparra, and Simoncelli, 2015). Even though Gaussianization is a quite different optimization problem than the rate–distortion objective with the set of constraints defined above, it is similar in that a marginally independent latent model is assumed in both cases. When optimizing for Gaussianization, the exponents in the parametric form of GDN control the tail behavior of the Gaussianized densities. Since tail behavior is less important here, we chose to simplify the functional form, fixing the exponents as well as forcing the weight matrix to be symmetric (i.e., γk,ij = γk,ji).
The synthesis transform is meant to function as an approximate inverse transformation, so we construct it by applying a principle known from the LISTA algorithm (Gregor and LeCun, 2010) to the fixed point iteration previously used to invert the GDN transform (Ballé, Laparra, and Simoncelli, 2015). The approximate inverse consists of one iteration, but with a separate set of parameters from the forward transform, which are constrained in the same way, but trained separately. We refer to this nonlinear transform as “inverse GDN” (IGDN).
The full model (analysis and synthesis filters, GDN and IGDN parameters) were optimized, for each λ, over a subset of the ImageNet database (Deng et al., 2009) consisting of 6507 images. We applied a number of preprocessing steps to the images in order to reduce artifacts and other unwanted contaminations: first, we eliminated images with excessive saturation. We added a small amount of uniform noise, corresponding to the quantization of pixel values, to the remaining images. Finally, we downsampled and cropped the images to a size of 256 × 256 pixels each, where the amount of downsampling and cropping was randomized, but depended on the size of the original image. In order to reduce high-frequency noise and compression artifacts, we only allowed resampling factors less than 0.75, discarding images that were too small to satisfy this constraint.
To ensure efficient and stable optimization, we used the following techniques:
• We used the Adam optimization algorithm (Kingma and Ba, 2014) to obtain values for the parameters φ and θ, starting with α = 10−4, and subsequently lowering it by a factor of 10 whenever the improvement of both rate and distortion stagnated, until α = 10−7.
• Linear filters were parameterized using their discrete cosine transform (DCT) coefficients. We found this to be slightly more effective in speeding up the convergence than discrete Fourier transform (DFT) parameterization (Rippel, Snoek, and Adams, 2015).
• We parameterized the GDN parameters in terms of the elementwise relationship
βk,i = (β ′ k,i) 2 − 2−10.
The squaring ensures that gradients are smaller around parameter values close to 0, a regime in which the optimization can otherwise become unstable. To obtain an unambiguous mapping, we projected each β′k,i onto the interval [2
−5,∞) after each gradient step. We applied the same treatment to γk,ij , and additionally averaged γ′k,ij with its transpose after each step in order to make it symmetric as explained above. The IGDN parameters were treated in the same way.
• To remove the scaling ambiguity between the each linear transform and its following nonlinearity (or preceding nonlinearity, in the case of the synthesis transform), we renormalized the linear filters after each gradient step, dividing each filter by the square root of the sum of its squared coefficients. For the analysis transform, the sum runs over space and all input channels, and for the synthesis transform, over space and all output channels.
We represented each of the marginals pỹi as a piecewise linear function (i.e., a linear spline), using 10 sampling points per unit interval. The parameter vector ψ(i) consists of the value of pỹi at these sampling points. We did not use Adam to update ψ(i); rather, we used ordinary stochastic gradient descent to minimize the negative expected likelihood:
Lψ(ψ (0),ψ(1), . . . ) = −Eỹ ∑ i pỹi(ỹi;ψ (i)). (17)
and renormalized the marginal densities after each step. After every 106 gradient steps, we used a heuristic to adapt the range of the spline approximation to cover the range of values of ỹi obtained on the training set.
6.2 ENTROPY CODE
We implemented an entropy code based on the context-adaptive binary arithmetic coding (CABAC) framework defined by Marpe, Schwarz, and Wiegand (2003). Arithmetic entropy codes are designed to compress discrete-valued data to bit rates closely approaching the entropy of the representation, assuming that the probability model used to design the code describes the data well. The following information was encoded into the bitstream:
• the size of the image (two 16-bit integers, bypassing arithmetic coding), • whether the image is grayscale or RGB (one bit, bypassing arithmetic coding), • the value of λ (one 16-bit integer, bypassing arithmetic coding), which provides an index
for the parameters of the analysis and synthesis transforms as well as the initial probability models for the entropy codes (these are fixed after optimization, and assumed to be available to encoder and decoder).
• the value of each element of q, iterating over channels, and over space in raster-scan order, using the arithmetic coding engine.
Since CABAC operates on binary values, the quantized values in q need to be converted to binary decisions. We follow a simple scheme inspired by the encoding of H.264/AVC transform coefficients as detailed by Marpe, Schwarz, and Wiegand (2003). For each qi, we start by testing if the encoded value is equal to the mode of the distribution. If this is the case, the encoding of qi is completed. If not, another binary decision determines whether it is smaller or larger than the mode. Following
that, each possible integer value is tested in turn, which yields a bifurcated chain of decisions as illustrated in figure 9. This process is carried out until either one of the binary decisions determines qi, or some minimum (qi,min) or maximum (qi,max) value is reached. In case qi is outside of that range, the difference between it and the range bound is encoded using an exponential Golomb code, bypassing the arithmetic coding engine.
Adaptive codes, such as CABAC, can potentially further improve bit rates, and to some extent correct model error, by adapting the probability model on-line to the statistics of the data. In our code, this is achieved by sharing the marginal probability model Pqi of each element in q across space within each channel. We derived the initial probability models by subsampling the continuous densities pỹi determined during optimization, as in (10). However, note that due to the simple raster-scan ordering, the coding scheme presented above only crudely exploits spatial adaptation of the probability model compared to existing coding methods such as JPEG 2000 and H.264/AVC. Thus, the performance gains compared to a well-designed non-adaptive entropy code are relatively small (figure 10), and likely smaller than those achieved by the entropy code in JPEG 2000, to which we compare.
6.3 EVALUATION DETAILS AND ADDITIONAL EXAMPLE IMAGES
Although it is desirable to summarize and compare the rate–distortion behavior of JPEG, JPEG 2000, and our method across an image set, it is difficult to do this in a way that is fair and interpretable. First, rate–distortion behavior varies substantially across bit rates for different images. For example, for the image in figure 12, our method achieves the same MSE with roughly 50% of the bits needed by JPEG 2000 for low rates, and about 30% for high rates. For the image in figure 17, the gains are more modest, although still significant through the range. But for the image in figure 15, our method only slightly outperforms JPEG 2000 at low rates, and under-performs it at high rates. Note that these behaviors are again different for MS-SSIM, which shows a significant improvement for all images and bit rates (consistent with their visual appearance).
Second, there is no obvious or agreed-upon method for combining rate–distortion curves across images. More specifically, one must decide which points in the curves to combine. For our method, it is natural to average the MSE and entropy values across images compressed using the same choice of λ, since these are all coded and decoded using exactly the same representation and quantization scheme. For JPEG, it seems natural to average over images coded at the same “quality” setting, which appear to be coded using the same quantization choices. The OpenJPEG implementation of JPEG 2000 we use allows selection of points on the rate–distortion curve either through specification of a target bit rate, or a target quality. This choice has no effect on rate–distortion plots for individual images (verified, but not shown), but has a substantial effect when averaging over images, since the two choices lead one to average over a different set of R-D points. This is illustrated in figure 11. Even if points were selected in exactly the same fashion for each of the methods (say, matched to a given set of target rates), summary plots can still over- or underemphasize high rate vs. low rate performance.
We conclude that summaries of rate–distortion are of limited use. Instead, we encourage the reader to browse our extensive collection of test images, with individual rate–distortion plots for each image, available at http://www.cns.nyu.edu/˜lcv/iclr2017 in both grayscale and RGB.
In the following pages, we show additional example images, compressed at relatively low bit rates, in order to visualize the qualitative nature of compression artifacts. On each page, the JPEG 2000 image is selected to have the lowest possible bit rate that is equal or greater than the bit rate of the proposed method. In all experiments, we compare to JPEG with 4:2:0 chroma subsampling, and the OpenJPEG implementation of JPEG 2000 with the default “multiple component transform”. For evaluating PSNR, we use the JPEG-defined conversion matrix to convert between RGB and Y’CbCr. For evaluating MS-SSIM (Wang, Simoncelli, and Bovik, 2003), we used only the resulting luma component. Original images are not shown, but are available online, along with compressed images at a variety of other bit rates, at http://www.cns.nyu.edu/˜lcv/iclr2017.

This paper extends an approach to rate-distortion optimization to deep encoders and decoders, and from a simple entropy encoding scheme to adaptive entropy coding. In addition, the paper discusses the approach’s relationship to variational autoencoders.
Given that the approach to rate-distortion optimization has already been published, the novelty of this submission is arguably not very high (correct me if I missed a new trick). In some ways, this paper even represents a step backward, since earlier work optimized for a perceptual metric where here MSE is used. However, the results are a visible improvement over JPEG 2000, and I don’t know of any other learned encoding which has been shown to achieve this level of performance. The paper is very well written.
Equation 10 appears to be wrong and I believe the partition function should depend on g_s(y; theta). This would mean that the approach is not equivalent to a VAE for non-Euclidean metrics.
What was the reason for optimizing MSE rather than a perceptual metric as in previous work? Given the author’s backgrounds, it is surprising that even the evaluation was only performed in terms of PSNR.
What is the contribution of adaptive entropy coding versus the effect of deeper encoders and decoders? This seems like an important piece of information, so it would be interesting to see the performance without adaptation as in the previous paper. More detail on the adaptive coder and its effects should be provided, and I will be happy to give a higher score when the authors do.