GENERATIVE ADVERSARIAL PARALLELIZATION
Generative Adversarial Networks (GAN) have become one of the most studied frameworks for unsupervised learning due to their intuitive formulation. They have also been shown to be capable of generating convincing examples in limited domains, such as low-resolution images. However, they still prove difficult to train in practice and tend to ignore modes of the data generating distribution. Quantitatively capturing effects such as mode coverage and more generally the quality of the generative model still remain elusive. We propose Generative Adversarial Parallelization (GAP), a framework in which many GANs or their variants are trained simultaneously, exchanging their discriminators. This eliminates the tight coupling between a generator and discriminator, leading to improved convergence and improved coverage of modes. We also propose an improved variant of the recently proposed Generative Adversarial Metric and show how it can score individual GANs or their collections under the GAP model.
1 INTRODUCTION
The growing popularity Generative Adversarial Networks (GAN) and their variants stems from their success in producing realistic samples (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016; Dumoulin et al., 2016) as well as the intuitive nature of the adversarial training framework (Goodfellow et al., 2014). Compared to other unsupervised learning paradigms, GANs have several merits:
• The objective function is not restricted to distances in input (e.g. pixel) space, for example, reconstruction error. Moreover, there is no restriction to certain type of functional forms such as having a Bernoulli or Gaussian output distribution.
• Compared to undirected probabilistic graphical models (Hinton et al., 2006; Salakhutdinov & Hinton, 2009), samples are generated in a single pass rather than iteratively. Moreover, the time to generate a sample is much less than recurrent models like PixelRNN (Oord et al., 2016).
• Unlike inverse transformation sampling models, the latent variable size is not restricted (Hyvarinen & Pajunen, 1999; Dinh et al., 2014).
In contrast, GANs are known to be difficult to train, especially as the data generating distribution becomes more complex. There have been some attempts to address this issue. For example, Salimans et al. (2016) propose several tricks such as feature matching and minibatch discrimination. In this work, we attempt to address training difficulty in a different way: extending two player generative adversarial games into a multi-player game. This amounts to training many GAN-like variants in parallel, periodically swapping their discriminators such that generator-discriminator coupling is reduced. Figure 1 provides a graphical depiction of our method.
Besides the training dilemma, from the point of view of density estimation, GANs possess very different characteristics compared to other probabilistic generative models. Most probabilistic models distribute the probability mass over the entire domain, whereas GAN by nature puts point-wise probability mass near the data. The question of whether this is desirable property or not is still an open question1. However, the primary concern of this property is that GAN may fail to allocate mass to
1Noting that the general view in ML is that there is nothing wrong with sampling from a degenerate distribution (Neal, 1998).
some important modes of the data generating distribution. We argue that our proposed model could alleviate this problem.
That our solution involves training many pairs of generators and discriminators together is a product of the fact that deep learning algorithms and distributed systems have been co-evolving for some time. Hardware accelerators, specifically Graphics Processing Units, (GPUs) have played a fundamental role in advancing deep learning, in particular because deep architectures are so well suited to parallelism (Coates et al., 2013). Data-based parallelism distributes large datasets over disparate nodes. Model-based parallelism allows complex models to be split over nodes. In both cases, learning must account for the coordination and communication among processors. Our work leverages recent advances along these lines (Ma et al., 2016).
2 BACKGROUND
The concept of a two player zero-sum game is borrowed from game theory in order to train a generative adversarial network (Goodfellow et al., 2014). A GAN consists of a generator G and discriminator D, both parameterized as feed-forward neural networks. The goal of the generator is to generate samples that fool the discriminator from thinking that those samples are from the data distribution p(x), ad interim the discriminative network’s goal is to not get tricked by the generator.
This view is formalized into a minimax objective such that the discriminator maximizes the expectation of its predictions while the generator minimizes the expectation of the discriminator’s predictions,
min θG max θD V (D,G) = min θG max θD
[ Ex∼pD [ logD(x) ] + Ez∼pG [ log ( 1−D(G(z)) )]] . (1)
where θG and θD are the parameters (weights) of the neural networks, pD is the data distribution, and pG is the prior distribution of the generative network.
Proposition 2 in (Goodfellow et al., 2014) illustrates the ideal concept of the solution. For two player game, each network’s gain of the utility (loss of the cost) ought to balance out the gain (loss) of the other network. In this scenario, the generator’s distribution becomes the data distribution. Remark that when the objective function is convex, gradient-based training is guaranteed to converge to a saddle point.
2.1 EMPIRICAL OBSERVATIONS
The reality of training GANs is quite different from the ideal case due to the following reasons:
1. The discriminative and generative networks are bounded by a finite number of parameters, which limits their modeling capacity.
2. Practically speaking, the second term of the objective function in Equation 1 is a bottleneck early on in training, where the discriminator can perfectly distinguish the noisy samples coming from the generator. The argument of the log saturates and gradient will not flow to the generator.
3. The GAN objective function is known to be non-convex and it is defined over a high-dimensional space. This often results in failure of gradient-based training to converge.
The first issue comes from the nature of the modelling problem. Nevertheless, due to the expressiveness of deep neural networks, they have been shown empirically to be capable of generating natural images (Radford et al., 2015; Im et al., 2016) by adopting parameter-efficient convolutional architectures. The second issue is typically addressed by inverting the generator’s minimization into the maximization formulation in Equation 1 accordingly,
min θG log(1−D(G(z)))→ max θG log(D(G(z))). (2)
This provides better gradient flow in the earlier stages of training (Goodfellow et al., 2014).
Although there have been cascades of success in image generation tasks using advanced GANs (Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), all of them mention the problem of difficulty in training. For example, Radford et al. (2015) state that the generator ... collapsing all samples to a single point ... is a common failure mode observed in GANs. This scenario can occur when the generator allocates most of its probability mass to a single sample that the discriminator has difficulty learning. Empirically, convergence of the learning curve does not correspond to improved quality of samples coming from the GAN and vice-versa. This is primarily caused by the third issue mentioned above. Gradient-based optimization methods are only guaranteed to converge to a Nash Equilibrium for convex functions, whereas the loss surface of the neural networks used in GANs are highly non-convex and there is no guarantee that a Nash Equilibrium even exists.
3 PARALLELIZING GENERATIVE ADVERSARIAL NETWORKS
Algorithm 1 Training procedure of GAP. Let T be total number of weight updates. Let N be the total number of GANs. Let K be the swapping frequency. LetM = {(G1, D1), (G2, D2), · · · , (GN , DN )}. while t < T do
UpdateMit = (Git , Dit) ∀i = 1 · · ·N .
if t % K == 0 then Randomly select N2 pairs with indices (i, j) w/o replacement. Swap Di and Dj (Gi and Gj) ∀i 6= j.
end if end while Select the best GAN based on GAM evaluation.
The subject of generative modeling with GANs has undergone intensive study, and model evaluation between various types of GANs is topic of increased interest and debate (Theis et al., 2015). Our work is inspired by the Generative Adversarial Metric (Im et al., 2016). The GAM enables us to quantitatively evaluate any pair of GANs. The core concept of the GAM is to swap one discriminator (generator) with the other discriminator (generator) during the test phase (see the pictorial example in Figure 8). The GAM concept can easily be extended from evaluation to the training phase.
Our proposal trains multiple GANs simultaneously. However, unlike the popular method of data parallelism, we do not train them independently with shared parameters, rather we try to produce synergy effects among different GANs during the training phase. This can be achieved simply by randomly swapping different discriminators (generators) every K updates. After training multiple GANS with our proposed method, we can select the best one based on the GAM. The pseudocode is shown in Algorithm 1.
We call our proposed method generative adversarial parallelization (GAP). Note that our method is not model-specific in a sense that GAP can be applied to any extension of GANs. For example, GAP can be applied to DCGAN or GRAN, or we can even apply GAP on several types of GANs simultaneously. Say, we have four GPUs available on which to parallelize models. We can allocate two GPUs for DCGANs and the remaining two GPUs for GRANs. Therefore, we view GAP as an operator rather than a model topology/architecture.
3.1 GAP AS REGULARIZATION
In a two player generative adversarial game, the concept of overfitting still exists. However, the realization of overfitting can be hard to notice. This is mainly due to not having a reconstructive error function. For models with a reconstruction-based objective, samples will simply become identical to the training data as the error approaches zero. On the other hand, with the GAN objective, even when the error approaches zero, it does not imply that the samples will look like the data. So, how can we characterize overfitting in a GAN?
We argue that overfitting in GANs manifests itself differently than in reconstructive models. Let us explain using two analogies to describe this phenomenon. Consider a generator as a judo fighter and discriminator as a sparring partner. When a judo fighter is only trained with the same sparring partner, his/her fighting strategy will naturally adapt to the style of his/her sparring partner. Thus, when the fighter is exposed to a new fighter with a different style, this fighter may suffer. Similarly, if a student learns from a single teacher, his/her learning experience will not only be limited but even overfitted to the teacher’s style of exams (see Figure 2). Equivalently, a paired generator and discriminator are likely to be adapted to their own strategy. Here, GAP intrinsically prevents this problem as the generator (discriminator) periodically gets paired with different discriminator (generator). Thus, GAP can be viewed as a regularizer.
3.2 MODE COVERAGE
The kind of overfitting problem mentioned above further relates to the problem of assigning probability mass to different modes of the data generating distribution – what we call mode coverage.
Let us re-consider the example introduced in Section 2.1. Say, the generator was able to figure out a single mode from which samples are drawn that confuse the discriminator. As long as the discriminator does not learn to fix this problem, the generator is not motivated to consider any other modes. This kind of scenario allows the generator to cheat by staying within a single, or small set of modes rather than exploring alternatives.
The story is not exactly the same when there are several different discriminators interacting with each generator. Since different discriminators may be good at distinguishing samples from different modes, each generator must put some effort into fooling all of the discriminators by generating samples from different modes. The situation where samples from a single mode fool all of the discriminators grows much less likely as the number and diversity of discriminators and generators increases (see Figure 3 and 4). Full details of this visualization are provided in Section 4.1.
4 EXPERIMENTS
We conduct an empirical investigation of GAP using two recently proposed GAN-variants as starting points: DCGAN (Radford et al., 2015) and GRAN (Im et al., 2016)2. In each case, we compare individual GAN-style models to GAP-style ensembles trained in parallel.
As it is difficult to quantitatively assess mode coverage, first we aim to visualize samples from GAP vs. other GAN variants on low-dimensional (toy) datasets as well as low-dimensional projections on real data. Then to evaluate each model quantitatively, we apply the GAM-II metric which is a re-formulation of GAM (Im et al., 2016) which can be used to compare different GAN architectures. Its motivation and use is described in Section 4.1. We consider, in total, five GAP variants which are summarized in Table 1.
2 The Theano-based DCGAN and GRAN implementations were based on https://github.com/Newmu/dcgan and https://github.com/jiwoongim/GRAN, respectively.
4.1 EXPERIMENTAL SETUP
All of our models are implemented in Theano (Bergstra et al., 2010) – a Python library that facilitates deep learning research. Because every update of each model is implemented as a separate process during training, swapping their parameters among different GANs necessitates interprocess communication3. Similar to the Theano-MPI framework, we chose to do inter-GPU memory transfer instead of passing through host memory in order to reduce communication overhead. Random swapping of the two discriminators’ parameters is achieved with an in-place MPI_SendRecv operation as DCGAN and GRAN share the same architecture and therefore the same parameterization.
Throughout the experiments, all datasets were normalized between [0, 1]. We used the same hyperparameters reported in (Radford et al., 2015) and (Im et al., 2016) for DCGAN and GRAN, respectively. The only additional hyper-parameter introduced by GAP is the frequency of swapping discriminators during training. We also made deliberate fine-grained distinctions among each GAN trained under GAP. These were: i) the generator’s prior distribution was selected as either uniform or Gaussian; ii) the order of mini-batches was permuted during learning; and iii) noise was injected at the input during learning and the amount of noise was decayed over time. The point of introducing these distinctions was to avoid multiple GANs converging to the same or very similar solutions. Lastly, we used gradient clipping (Pascanu et al., 2013) on both discriminators and generators.
To measure the performance of GANs, our first attempt was to apply GAM to evaluate our model. Unfortunately, we realized that GAM is not applicable when comparing GAP vs. non-GAP models. This is because GAM requires the discriminator the GANs under comparison to have similar error rates on a held-out test set. However, as shown in Figure 6, GAP boosts the generalization of the discriminators, which causes it to have different test error rates compared to the error rate from non-GAP models. Hence, we propose a new metric that omits the GAM’s constraints which we call GAM-II. It simply measures the average (or worst case) error rate among a collection of discriminators. A detailed description of GAM-II is provided in Appendix A.1.
4.2 RESULTS
We report our experimental results by answering a few core questions.
Q: Do GAP-trained models cover more modes of the data generating distribution?
Determining whether applying GAP achieves broader mode coverage is difficult to validate in high-dimensional spaces. Therefore, we initially verified GAP and non-GAP models on two lowdimensional synthetic datasets. The R15 dataset4 contains 500 two-dimensional data points with 15 clusters as shown in Figure 3a. The Mixture of Gaussians dataset5 contains 2,500 two-dimensional data points with 25 clusters as shown in Figure 4a.
Both discriminator and generator had four fully-connected batch-normalized layers with ReLU activation units. We first optimized the hyper-parameters of a single GAN based on visually inspecting the samples that it generated (i.e. Figure 3 shows samples from the best performing single GAN that we trained). We then trained four parallelized GANs using the same hyper-parameters of the best single GAN.
The samples generated from both models are shown in Figure 3 and 4. We observe that GAP(GAN×4) produces samples that look more similar to the original dataset compared to a single
3We used openMPI for implementing GAP – see https://www.open-mpi.org/ 4The R15 dataset can be found at https://cs.joensuu.fi/sipu/datasets/ 5The Mixture of Gaussians dataset can be found at https://github.com/IshmaelBelghazi/ALI/
GAN. The overlap of samples generated by four GANs are consistent with Figure 3c. Note that as we decrease the number of training points, the overlap of GAN samples deviates from the original dataset while GAP seems not to suffer from this phenomenon. For example, when we used all 600 examples of R15, both GAN and GAP samples matched the distribution of data in Figure 3a. However, as we use less training examples, GAN failed to accurately model the data distribution by dropping modes. The samples plotted in Figure 3c are based on training each model with a random subset of 100 examples drawn from the original 600. Based on the synthetic experiments we confirm that GAP can improve mode coverage when a limited number of training samples are available.
In order to gain a qualitative sense of models trained using a high dimensional dataset, we considered two experiments: i) we examined the class label predictions made on samples from each model to check how uniformly they were distributed. The histogram of the predicted classes is provided in Figure 14. ii) we created a t-SNE visualization of generated samples overlaid on top of the true data (see Appendix A.2). We find that the intersection of data points and samples generated by GAP is slightly better than samples generated by individual GANs. In addition to the synthetic data results, these visualizations suggest some favourable properties of GAP, but we hesitate to draw any strong conclusions from them.
Q: Does GAP enhance generalization?
To answer this question, we considered the MNIST, CIFAR-10, and LSUN church datasets which are often used to evaluate GAN variants. MNIST and CIFAR-10 consist of 50,000 training and 10,000 test images of size 27×28 and 32×32×3 pixels, respectively. Each contains 10 different classes of objects. The LSUN church dataset contains various outdoor church images. These high resolution images were downsampled to 64×64 pixels. The training set consists of 126,227 examples. One implicit but imperfect way to measure the generalization of a GAN is to observe generalization of the discriminator alone. This is because the generator is influenced by the discriminator and vice versa. If the discriminator is overfitting the training data, then the generator must be biased towards the training data as well. Here, we plot the learning curve of the discriminator during training for both GAP(DCGAN) and GAP(GRAN).
Figure 6 shows the learning curve for a single model versus groups of two and four models parallelized under GAP. We observe that more parallelization leads to less of a spread between the train and validation curves indicating the ability of GAP to improve generalization. Note that in order to plot a single representative learning curve while training multiple models under GAP, we averaged the learning curves of the individual models. To demonstrate that our observations are not merely attributable to smoothing by averaging, we show individual learning curves of the parallelized GANs (see Figure 13 in Appendix A.3). From now on, we will work with GAPD4 and GAPG4.
Q: How does the rate at which discriminators are swapped affect training?
As noted earlier, the swapping frequency is the only additional hyper-parameter introduced by GAP. We conduct a simple sensitivity analysis by plotting the validation cost of each GAN during training along with its standard deviation in Figure 7. We observe that GAP(DCGAN) varies the least at a swapping frequency of 0.5 – swapping twice per epoch. Meanwhile, GAP(GRANs) are not too sensitive to swapping frequencies above 0.1. Figure 12 in Appendix A.3 plots learning curves at different swapping frequencies. Across all rates, we still see that the spread between the training and validation costs decreases with the number of GANs trained in parallel.
Q: Does GAP(·) improve the quality of generative models?
We used GAM-II to evaluate GAP (see Appendix A.1). We first looked at the performance over four models: DCGAN, GRAN, GAPD4, and GAPG4. We also considered combining multiple GANvariants in a GAP model (hybrid GAP). We denote this model as GAPC4. GAPC4 consists of two DCGANs and two GRANs trained with GAP. Overall, we have ten generators and ten discriminators for DCGAN and GRAN: four discriminators from the individually-trained models, and four discriminators from GAP, and two discriminators from the GAP combination, GAPC4. We used the collection of all ten discriminators to evaluate the generators. Table 3 presents the results. Note that we report the minimum and maximum of average and worst error rates among four GANs. Looking
at the average errors, GAPD4 strongly outperforms DCGAN on all datasets. GAPG4 outperforms GRAN on CIFAR-10 and MNIST and strongly outperforms it on LSUN. For the case of the maximum worst-case error, GAP outperforms both DCGAN and GRAN across all datasets. However, we did not find an improvement on GAPC4 based on the GAM-II metric.
Additionally, we estimated the log-likelihood assigned by each model based on a recently proposed evaluation scheme that uses Annealed Importance Sampling (Wu et al., 2016). With the code provided by (Wu et al., 2016), we were able to evaluate DCGANs trained by GAPD4 and GAPcomb6. The results are shown in Table 4. Again, these results show that GAPD4 improves on DCGAN’s performance, but there is no advantage using combined GAPC4.
Samples from each CIFAR-10 and LSUN model for visual inspection are reproduced in Figures 16, 17, 18, and 19.
5 DISCUSSION
We have proposed Generative Adversarial Parallelization, a framework in which several adversarially-trained models are trained together, exchanging discriminators. We argue that this reduces the tight coupling between generator and discriminator and show empirically that this has a beneficial effect on mode coverage, convergence, and quality of the model under the GAM-II metric. Several directions of future investigation are possible. This includes applying GAP to the evolving variety of adversarial models, like improvedGAN (Salimans et al., 2016). We still view stability as an issue and partially address it by tricks such as clipping the gradient of the discriminator. In this work, we only explored synchronous training of GANs under GAP, however, asynchronous training may provide more stability. Recent work has explored the connection between GANs and actorcritic methods in reinforcement learning (Pfau & Vinyals, 2016). Under this view, we believe that GAP may have interesting implications for multi-agent RL. Although we have assessed mode coverage qualitatively either directly or indirectly via projections, quantitatively assessing mode coverage for generative models is still an open research problem.
6Unfortunately, we did not get the code provided by (Wu et al., 2016) to work on GRAN.
A SUPPLEMENTARY MATERIAL FOR GENERATIVE ADVERSARIAL PARALLELIZATION
A.1 GENERATIVE ADVERSARIAL METRIC II
Although GAM is a valid metric as it measures the likelihood ratio of two generative models, it is hard to apply in practice. This is due to having a test ratio constraint, which imposes the condition that the ratio between test error rates be approximately unity. However, because GAP improves the generalization of GANs as shown in Figure 6, the test ratio often does not equal one (see Section 4). We introduce a new generative adversarial metric, and call it GAM-II.
GAM-II evaluates a model based on either the average error rate or worst error rate of a collection of discriminators given a set of samples from each model to be evaluated:
argmax {Gj |Sj∼pGj } ̂(Sj) = argmax {Gj |Sj∼pGj }
1
Nj Nj∑ i=1 (Sj |Di), (3)
argmax {Gj |Sj∼pGj } ̄(Sj) = argmax {Gj |Sj∼pGj } min i=1···Nj
(Sj |Di) (4)
where outputs the classification error rate, and Nj is all discriminators except for the ones that the generator j saw during training. For example, the comparison of DCGAN and GAP applied to four DCGANs is shown in Figure 9.
Definition. We say that GAP helps if at least one of the models trained with GAP performs better than a single model. Moreover, GAP strongly-helps if all models trained with GAP perform better than a single model.
In our experiments, we assess GAP based on the definition above.
A.2 EXPERIMENTS WITH T-SNE
In order to get a qualitative sense of models trained using a high dimensional dataset, we consider a t-SNE map of generated samples overlaid on top of the true data. Normally, a t-SNE map is used to visualize clusters of embedded high-dimensional data. Here, we are more interested in the overlap between true data and generated samples by visualizing clusters which we interpret as modes of the data generating distribution.
Figure 10 and 11 present the t-SNE map of data and samples from single- and multiple- trained GANs under GAP. We find that the intersection of data points and samples generated by GAP is slightly better than samples generated by individual GANs. This provides an incomplete view but is nevertheless a helpful visualization.
A.3 SUPPORTING FIGURES
The supporting figures as in Figure 6 is presented for LSUN dataset in Figure 12. There are total of four plots with different swapping frequencies.
Figure 13 presents an instance of an individual learning curve in the case when multiple GANs are trained under GAP. The difference from from Figure 6 and Figure 12 is that GAP curves are represented by the learning curve of a single GAN within GAP rather than an average. Fortunately, the behaviour remains the same, where the spread between training and validation cost decreases as parallelization scales up (i.e. more models in a GAP).
We observed the distribution of class predictions on samples from each model in order to check how closely they match the training set distribution (which is uniform for MNIST). We trained a simple logistic regression on MNIST that resulted in a '99% test accuracy rate. The histogram of the predicted classes is provided in Figure 14. We looked at the exponentiated expected KL divergence between the predicted distribution and the (uniform) prior distribution, also known as the “Inception Score” (Salimans et al., 2016). The results are shown in Table 5.
Table 5: Inception Score of GAP
MODELS DCGAN GAPD4 GRAN GAPG4 MNIST 6.36± 0.09 6.592± 0.08 6.77± 0.08 6.85± 0.24
0 2 4 6 8 10 0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
(a) DCGAN (b) GAP(DCGANx4)
0 2 4 6 8 10 0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
(c) GRAN
0 2 4 6 8 10 0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
(d) GAP(GRANx4)
Figure 14: The distribution of the predicted class labels of samples from various models made by a separately trained logistic regression.
A.3.1 FINE-TUNING GANS USING GAP
We also tried fine-tuning individually-trained GANs using GAP, which we denote as GAPF4. GAPF4 consists of two trained DCGANs and two trained GRANs. They are then fine-tuned using GAP for five epochs. Samples from the fine-tuned models are shown in Figure 20.

This paper proposes Generative Adversarial Parallelization (GAP), one schedule to train N Generative Adversarial Networks (GANs) in parallel. GAP proceeds by shuffling the assignments between the N generators and the N discriminators at play every few epochs. Therefore, GAP forces each generator to compete with multiple discriminators at random. The authors claim that such randomization reduces undesired "mode collapsing behaviour", typical of GANs.
I have three concerns with this submission.
1) After training the N GANs for a sufficient amount of time, the authors propose to choose the best generator using the GAM metric. I oppose to this because of two reasons. First, a single GAN will most likely be unable to express the full richness of the true data begin modeled. Said differently, a single generator with limited power will either describe a mode well, or describe many modes poorly. Second, GAM relies on the scores given by the discriminators, which can be ill-posed (focus on artifacts). Since there is There is nothing wrong with mode collapsing when this happens under control. Thus, I believe that a better strategy would be to not choose and combine all generators into a mixture. Of course, this would require a way to decide on mixture weights. This can be done, for instance, using rejection sampling based on discriminator scores.
2) The authors should provide a theoretical (or at least conceptual) comparison to dropout. In essence, this paper has a very similar flavour: every generator is competing against all N discriminators, but at each epoch we drop N-1 for every generator. Related to the previous point, after training dropout keeps all the neurons, effectively approximating a large ensemble of neural networks.
3) The qualitative results are not convincing. Most of the figures show only results about GAP. How do the baseline samples look like? The GAN and LAPGAN papers show very similar samples. On the other hand, I do not find Figures 3 and 4 convincing: for instance, the generator in Figure 3 was most likely under-parametrized.
As a minor comment, I would remove Figure 2. This is because of three reasons: it may be protected by copyright, it occupies a lot of space, and it does not add much value to the explanation. Also, the indices (i_t) are undefined in Algorithm 1.
Overall, this paper shows good ideas, but it needs further work in terms of conceptual development and experimental evaluation.


Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image regions. For example, our model is able to detect spoken instances of the words “lighthouse” within an utterance and associate them with image regions containing lighthouses. We do not use any form of conventional automatic speech recognition, nor do we use any text transcriptions or conventional linguistic annotations. Our model effectively implements a form of spoken language acquisition, in which the computer learns not only to recognize word categories by sound, but also to enrich the words it learns with semantics by grounding them in images.
Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image regions. For example, our model is able to detect spoken instances of the words “lighthouse” within an utterance and associate them with image regions containing lighthouses. We do not use any form of conventional automatic speech recognition, nor do we use any text transcriptions or conventional linguistic annotations. Our model effectively implements a form of spoken language acquisition, in which the computer learns not only to recognize word categories by sound, but also to enrich the words it learns with semantics by grounding them in images.
1 INTRODUCTION
1.1 PROBLEM STATEMENT AND MOTIVATION
Automatically discovering words and other elements of linguistic structure from continuous speech has been a longstanding goal in computational linguists, cognitive science, and other speech processing fields. Practically all humans acquire language at a very early age, but this task has proven to be an incredibly difficult problem for computers. While conventional automatic speech recognition (ASR) systems have a long history and have recently made great strides thanks to the revival of deep neural networks (DNNs), their reliance on highly supervised training paradigms has essentially restricted their application to the major languages of the world, accounting for a small fraction of the more than 7,000 human languages spoken worldwide (Lewis et al., 2016). The main reason for this limitation is the fact that these supervised approaches require enormous amounts of very expensive human transcripts. Moreover, the use of the written word is a convenient but limiting convention, since there are many oral languages which do not even employ a writing system. In constrast, infants learn to communicate verbally before they are capable of reading and writing - so there is no inherent reason why spoken language systems need to be inseparably tied to text.
The key contribution of this paper has two facets. First, we introduce a methodology capable of not only discovering word-like units from continuous speech at the waveform level with no additional text transcriptions or conventional speech recognition apparatus. Instead, we jointly learn the semantics of those units via visual associations. Although we evaluate our algorithm on an English corpus, it could conceivably run on any language without requiring any text or associated ASR capability. Second, from a computational perspective, our method of speech pattern discovery runs in linear time. Previous work has presented algorithms for performing acoustic pattern discovery in continuous speech (Park & Glass, 2008; Jansen et al., 2010; Jansen & Van Durme, 2011) without the use of transcriptions or another modality, but those algorithms are limited in their ability to scale by their inherent O(n2) complexity, since they do an exhaustive comparison of the data against itself. Our method leverages correlated information from a second modality - the visual domain - to guide the discovery of words and phrases. This enables our method to run in O(n) time, and we demonstrate it scalability by discovering acoustic patterns in over 522 hours of audio data.
1.2 PREVIOUS WORK
A sub-field within speech processing that has garnered much attention recently is unsupervised speech pattern discovery. Segmental Dynamic Time Warping (S-DTW) was introduced by Park & Glass (2008), which discovers repetitions of the same words and phrases in a collection of untranscribed acoustic data. Many subsequent efforts extended these ideas(Jansen et al., 2010; Jansen & Van Durme, 2011; Dredze et al., 2010; Harwath et al., 2012; Zhang & Glass, 2009). Alternative approaches based on Bayesian nonparametric modeling (Lee & Glass, 2012; Ondel et al., 2016) employed a generative model to cluster acoustic segments into phoneme-like categories, and related works aimed to segment and cluster either reference or learned phoneme-like tokens into word-like and higher-level units (Johnson, 2008; Goldwater et al., 2009; Lee et al., 2015).
In parallel, the computer vision and NLP communities have begun to leverage deep learning to create multimodal models of images and text. Many works have focused on generating annotations or text captions for images (Socher & Li, 2010; Frome et al., 2013; Socher et al., 2014; Karpathy et al., 2014; Karpathy & Li, 2015; Vinyals et al., 2015; Fang et al., 2015; Johnson et al., 2016). One interesting intersection between word induction from phoneme strings and multimodal modeling of images and text is that of Gelderloos & Chrupaa (2016), who uses images to segment words within captions at the phoneme string level. Several recent papers have taken these ideas beyond text, and attempted to relate images to spoken audio captions directly at the waveform level (Harwath & Glass, 2015; Harwath et al., 2016).
While supervised object detection is a standard problem in the vision community, several recent works have tackled the problem of weakly-supervised or unsupervised object localization (Bergamo et al., 2014; Cho et al., 2015; Zhou et al., 2015; Cinbis et al., 2016). Although the focus of this work is discovering acoustic patterns, in the process we jointly associate the acoustic patterns with clusters of image crops, which we demonstrate capture visual patterns as well.
2 EXPERIMENTAL DATA
We employ a corpus of over 200,000 spoken captions for images taken from the Places205 dataset (Zhou et al., 2014), corresponding to over 522 hours of speech data. The captions were collected using Amazon’s Mechanical Turk service, in which workers were shown images and asked to describe them verbally in a free-form manner. Our data collection scheme is described in detail in Harwath et al. (2016), but the experiments in this paper leverage nearly twice the amount of data. For training our multimodal neural network as well as the pattern discovery experiments, we use a subset of 214,585 image/caption pairs, and we hold out a set of 1,000 pairs for evaluating the performance of the multimodal network’s retrieval ability. Because we lack ground truth text transcripts for the data, we used Google’s Speech Recognition public API to generate proxy transcripts which we use when analyzing our system. Note that the ASR was only used for analysis of the results, and was not involved in any of the learning.
3 AUDIO-VISUAL EMBEDDING NEURAL NETWORKS
We first train a deep multimodal embedding network similar in spirit to the one described in Harwath et al. (2016), but with a more sophisticated architecture. The model is trained to map entire image frames and entire spoken captions into a shared embedding space; however, as we will show, the trained network can then be used to localize patterns corresponding to words and phrases within the spectrogram, as well as visual objects within the image by applying it to small sub-regions of the image and spectrogram. The model is comprised of two branches, one which takes as input images, and the other which takes as input spectrograms. The image network is formed by taking the off-the-shelf VGG 16 layer network (Simonyan & Zisserman, 2014) and replacing the softmax classification layer with a linear transform which maps the 4096-dimensional activations of the second fully connected layer into our 1024-dimensional multimodal embedding space. In our experiments, the weights of this projection layer are trained, but the layers taken from the VGG network below it are kept fixed. The second branch of our network analyzes speech spectrograms as if they were black and white images. Our spectrograms are computed using 40 log Mel filterbanks with a 25ms Hamming window and a 10ms shift. Therefore, the input to this branch always has 1 color channel
and is always 40 pixels high (corresponding to the 40 Mel filterbanks), but the width of the spectrogram varies depending upon the duration of the spoken caption, with each pixel corresponding to approximately 10 milliseconds worth of audio. The specific network architecture we use is shown below, where C denotes the number of convolutional channels, W is filter width, H is filter height, and S is pooling stride.
1. Convolution with C=128, W=1, H=40, ReLU 2. Convolution with C=256, W=11, H=1, ReLU, maxpool with W=3, H=1, S=2 3. Convolution with C=512, W=17, H=1, ReLU, maxpool with W=3, H=1, S=2 4. Convolution with C=512, W=17, H=1, ReLU, maxpool with W=3, H=1, S=2 5. Convolution with C=1024, W=17, H=1, ReLU 6. Meanpool over entire caption width followed by L2 normalization
In practice during training, we restrict the caption spectrograms to all be 1024 frames wide (i.e., 10sec of speech) by applying truncation or zero padding; this introduces computational savings and was shown in Harwath et al. (2016) to only slightly degrade the performance. Additionally, both the images and spectrograms are mean normalized before training. The overall multimodal network is formed by tying together the image and audio branches with a layer which takes both of their output vectors and computes an inner product between them, representing the similarity score between a given image/caption pair. We train the network to assign high scores to matching image/caption pairs, and lower scores to mismatched pairs. The objective function and training procedure we use is identical to that described in Harwath et al. (2016), but we briefly describe it here.
Within a minibatch of B image/caption pairs, let Spj , j = 1, . . . , B denote the similarity score of the jth image/caption pair as output by the neural network. Next, for each pair we randomly sample one impostor caption and one impostor image from the same minibatch. Let Sij denote the similarity score between the jth caption and its impostor image, and Scj be the similarity score between the jth image and its impostor caption. The total loss for the entire minibatch is then computed as
L(θ) = B∑ j=1 max(0, Scj − S p j + 1) + max(0, S i j − S p j + 1). (1)
We train the neural network with 50 epochs of stochastic gradient descent using a batch size B = 128, a momentum of 0.9, and a learning rate of 1e-5 which is set to geometrically decay by a factor between 2 and 5 every 5 to 10 epochs.
4 FINDING AND CLUSTERING AUDIO-VISUAL CAPTION GROUNDINGS
Although we have trained our multimodal network to compute embeddings at the granularity of entire images and entire caption spectrograms, we can easily apply it in a more localized fashion. In the case of images, we can simply take any arbitrary crop of an original image and resize it to 224x224 pixels. The audio network is even more trivial to apply locally, because it is entirely convolutional and the final mean pooling layer ensures that the output will be a 1024-dim vector no matter the extent of the input. The bigger question is where to locally apply the networks in order to discover meaningful acoustic and visual patterns.
Given an image and its corresponding spoken audio caption, we use the term grounding to refer to extracting meaningful segments from the caption and associating them with an appropriate subregion of the image. For example, if an image depicted a person eating ice cream and its caption contained the spoken words “A person is enjoying some ice cream,” an ideal set of groundings would entail the acoustic segment containing the word “person” linked to a bounding box around the person, and the segment containing the word “ice cream” linked to a box around the ice cream. We use a constrained brute force ranking scheme to evaluate all possible groundings (with a restricted granularity) between an image and its caption. Specifically, we divide the image into a grid, and extract all of the image crops whose boundaries sit on the grid lines. Because we are mainly interested in extracting regions of interest and not high precision object detection boxes, to keep the number of proposal regions under control we impose several restrictions. First, we use a 10x10 grid on each image regardless of its original size. Second, we define minimum and maximum aspect ratios as 2:3
and 3:2 so as not to introduce too much distortion and also to reduce the number of proposal boxes. Third, we define a minimum bounding width as 30% of the original image width, and similarly a minimum height as 30% of the original image height. In practice, this results in a few thousand proposal regions per image.
To extract proposal segments from the audio caption spectrogram, we similarly define a 1-dim grid along the time axis, and consider all possible start/end points at 10 frame (pixel) intervals. We impose minimum and maximum segment length constraints at 50 and 100 frames (pixels), implying that our discovered acoustic patterns are restricted to fall between 0.5 and 1 second in duration. The number of proposal segments will vary depending on the caption length, and typically number in the several thousands. Note that when learning groundings we consider the entire audio sequence, and do not incorporate the 10sec duration constraint imposed during the first stage of learning.
Once we have extracted a set of proposed visual bounding boxes and acoustic segments for a given image/caption pair, we use our multimodal network to compute a similarity score between each unique image crop/acoustic segment pair. Each triplet of an image crop, acoustic segment, and similarity score constitutes a proposed grounding. A naive approach would be to simply keep the top N groundings from this list, but in practice we ran into two problems with this strategy. First, many proposed acoustic segments capture mostly silence due to pauses present in natural speech. We solve this issue by using a simple voice activity detector (VAD) which was trained on the TIMIT corpus(Garofolo et al., 1993). If the VAD estimates that 40% or more of any proposed acoustic segment is silence, we discard that entire grounding. The second problem we ran into is the fact that the top of the sorted grounding list is dominated by highly overlapping acoustic segments. This makes sense, because highly informative content words will show up in many different groundings with slightly perturbed start or end times. To alleviate this issue, when evaluating a grounding from the top of the proposal list we compare the interval intersection over union (IOU) of its acoustic segment against all acoustic segments already accepted for further consideration. If the IOU exceeds a threshold of 0.1, we discard the new grounding and continue moving down the list. We stop accumulating groundings once the scores fall to below 50% of the top score in the “keep” list, or when 10 groundings have been added to the “keep” list, whichever comes first. Figure 1 displays a pictorial example of our grounding procedure.
Once we have completed the grounding procedure, we are left with a small set of regions of interest in each image and caption spectrogram. We use the respective branches of our multimodal network to compute embedding vectors for each grounding’s image crop and acoustic segment. We then employ k-means clustering separately on the collection of image embedding vectors as well as the collection of acoustic embedding vectors. The last step is to establish an affinity score between each image cluster I and each acoustic cluster A; we do so using the equation
Affinity(I,A) = ∑ i∈I ∑ a∈A i>a · Pair(i,a) (2)
where i is an image crop embedding vector, a is an acoustic segment embedding vector, and Pair(i,a) is equal to 1 when i and a belong to the same grounding pair, and 0 otherwise. After clustering, we are left with a set of acoustic pattern clusters, a set of visual pattern clusters, and a set of linkages describing which acoustic clusters are associated with which image clusters. In the next section, we investigate the properties of these clusters in more detail.
5 EXPERIMENTS AND ANALYSIS
We trained our multimodal network on a set of 214,585 image/caption pairs, and vetted it with an image search (given caption, find image) and annotation (given image, find caption) task similar to the one used in Harwath et al. (2016); Karpathy et al. (2014); Karpathy & Li (2015). The image annotation and search recall scores on a 1,000 image/caption pair held-out test set are shown in Table 1, and are compared against the model architecture used in Harwath et al. (2016). We then performed the grounding and pattern clustering steps on the entire training dataset. This resulted in a total of 1,161,305 unique grounding pairs.
In order to evaluate the acoustic pattern discovery and clustering, we wish to assign a label to each cluster and cluster member, but this is not completely straightforward since each acoustic segment may capture part of a word, a whole word, multiple words, etc. Our strategy is to force-align the Google recognition hypothesis text to the audio, and then assign a label string to each acoustic segment based upon which words it overlaps in time. The alignments are created with the help of a Kaldi (Povey et al., 2011) speech recognizer based on the standard WSJ recipe and trained using the Google ASR hypothesis as a proxy for the transcriptions. Any word whose duration is overlapped 30% or more by the acoustic segment is included in the label string for the segment. We then employ a majority vote scheme to derive the overall cluster labels. When computing the purity of a cluster, we count a cluster member as matching the cluster label as long as the overall cluster label appears in the member’s label string. In other words, an acoustic segment overlapping the words “the lighthouse” would receive credit for matching the overall cluster label “lighthouse”. Several example clusters and a breakdown of the labels of their members are shown in Table 2. We investigated some simple schemes for predicting highly pure clusters, and found that the empirical variance of the cluster members (average squared distance to the cluster centroid) was a good indicator. Figure 2 displays a scatter plot of cluster purity weighted by the natural log of the cluster size against the empirical variance. Large, pure clusters are easily predicted by their low empirical variance, while a high empirical variance is indicative of a garbage cluster.
Ranking a set of k = 500 acoustic clusters by their variance, Table 3 displays some statistics for the 50 lowest-variance clusters. We see that most of the clusters are very large and highly pure, and their labels reflect interesting object categories being identified by the neural network. We additionally compute the coverage of each cluster by counting the total number of instances of the cluster label anywhere in the training data, and then compute what fraction of those instances were captured by the cluster. We notice many examples of high coverage clusters, e.g. the “skyscraper” cluster captures 84% of all occurrences of the word “skyscraper” anywhere in the training data, while the “baseball” cluster captures 86% of all occurrences of the word “baseball”. This is quite impressive given the fact that no conventional speech recognition was employed, and neither the multimodal neural network nor the grounding algorithm had access to the text transcripts of the captions.
To get an idea of the impact of the k parameter as well as a variance-based cluster pruning threshold based on Figure 2, we swept k from 250 to 2000 and computed a set of statistics shown in Table 4. We compute the standard overall cluster purity evaluation metric in addition to the average coverage across clusters. The table shows the natural tradeoff between cluster purity and redundancy
(indicated by the average cluster coverage) as k is increased. In all cases, the variance-based cluster pruning greatly increases both the overall purity and average cluster coverage metrics. We also notice that more unique cluster labels are discovered with a larger k.
Next, we examine the image clusters. Figure 3 displays the 9 most central image crops for a set of 10 different image clusters, along with the majority-vote label of each image cluster’s associated audio cluster. In all cases, we see that the image crops are highly relevant to their audio cluster label. We include many more example image clusters in Appendix A.
Finally, we wish to examine the semantic embedding space in more depth. We took the top 150 clusters from the same k = 500 clustering run described in Table 3 and performed t-SNE (van der Maaten & Hinton, 2008) analysis on the cluster centroid vectors. We projected each centroid down to 2 dimensions and plotted their majority-vote labels in Figure 4. Immediately we see that different clusters which capture the same label closely neighbor one another, indicating that distances in the embedding space do indeed carry information discriminative across word types (and suggesting that a more sophisticated clustering algorithm than k-means would perform better). More interestingly, we see that semantic information is also reflected in these distances. The cluster centroids for “lake,” “river,” “body,” “water,” “waterfall,” “pond,” and “pool” all form a tight meta-cluster, as do “restaurant,” “store,” “shop,” and “shelves,” as well as “children,” “girl,” “woman,” and “man.” Many other semantic meta-clusters can be seen in Figure 4, suggesting that the embedding space is capturing information that is highly discriminative both acoustically and semantically.
Table 1: Results for image search and annotation on the Places audio caption data (214k training pairs, 1k testing pairs). Recall is shown for the top 1, 5, and 10 hits. The model we use in this paper is compared against the meanpool variant of the model architecture presented in Harwath et al. (2016). For both training and testing, the captions were truncated/zero-padded to 10 seconds.
Search Annotation Model R@1 R@5 R@10 R@1 R@5 R@10
(Harwath et al., 2016) 0.090 0.261 0.372 0.098 0.266 0.352 This work 0.112 0.312 0.431 0.120 0.307 0.438
Figure 2: Scatter plot of audio cluster purity weighted by log cluster size against cluster variance for k = 500 (least-squares line superimposed).
Word Count Word Count
ocean 2150 castle 766 (silence) 127 (silence) 70 the ocean 72 capital 39
blue ocean 29 large castle 24 body ocean 22 castles 23
oceans 16 (noise) 21 ocean water 16 council 13
(noise) 15 stone castle 12 of ocean 14 capitol 10
oceanside 14 old castle 10
Table 2: Examples of the breakdown of word/phrase identities of several acoustic clusters
6 CONCLUSIONS AND FUTURE WORK
In this paper, we have demonstrated that a neural network trained to associate images with the waveforms representing their spoken audio captions can successfully be applied to discover and cluster acoustic patterns representing words or short phrases in untranscribed audio data. An analogous procedure can be applied to visual images to discover visual patterns, and then the two modali-
ties can be linked, allowing the network to learn e.g. that spoken instances of the word “train” are associated with image regions containing trains. This is done without the use of a conventional automatic speech recognition system and zero text transcriptions, and therefore is completely agnostic to the language in which the captions are spoken. Further, this is done in O(n) time with respect to the number of image/caption pairs, whereas previous state-of-the-art acoustic pattern discovery algorithms which leveraged acoustic data alone run in O(n2) time. We demonstrate the success of our methodology on a large-scale dataset of over 214,000 image/caption pairs, comprising over 522 hours of spoken audio data. We have shown that the shared multimodal embedding space learned by our model is discriminative not only across visual object categories, but also acoustically and semantically across spoken words. To the best of our knowledge, this paper contains by far the largest scale speech pattern discovery experiment ever performed, as well as the first ever successful effort
to learn the semantics of the discovered acoustic patterns by grounding them to patterns which are jointly discovered in another modality (images).
The future directions in which this research could be taken are incredibly fertile. Because our method creates a segmentation as well as an alignment between images and their spoken captions, a generative model could be trained using these alignments. The model could provide a spoken caption for an arbitrary image, or even synthesize an image given a spoken description. Modeling improvements are also possible, aimed at the goal of incorporating both visual and acoustic localization into the neural network itself. Additionally, by collecting a second dataset of captions for our images in a different language, such as Spanish, our model could be extended to learn the acoustic correspondences for a given object category in both languages. This paves the way for creating a speech-to-speech translation model not only with absolutely zero need for any sort of text transcriptions, but also with zero need for directly parallel linguistic data or manual human translations.
A APPENDIX: ADDITIONAL VISUALIZATIONS OF IMAGE PATTERN CLUSTERS
beach cliff pool desert field
chair table staircase statue stone
church forest mountain skyscraper trees
waterfall windmills window city bridge
flowers man wall archway baseball
boat shelves cockpit girl children
building rock kitchen plant hallway

CONTRIBUTIONS
This paper introduces a method for learning semantic "word-like" units jointly from audio and visual data. The authors use a multimodal neural network architecture which accepts both image and audio (as spectrograms) inputs. Joint training allows one to embed both image and spoken language captions into a shared representation space. Audio-visual groundings are generated by measuring affinity between image patches and audio clips. This allows the model to relate specific visual regions to specific audio segments. Experiments cover image search (audio to image) and annotation (image to audio) tasks and acoustic word discovery.
NOVELTY+SIGNIFICANCE
As correctly mentioned in Section 1.2, the computer vision and natural language communities have studied multimodal learning for use in image captioning and retrieval. With regards to multimodal learning, this paper offers incremental advancements since it primarily uses a novel combination of input modalities (audio and images).
However, bidirectional image/audio retrieval has already been explored by the authors in prior work (Harwath et al, NIPS 2016). Apart from minor differences in data and CNN architecture, the training procedure in this submission is identical to this prior work. The novelty in this submission is therefore the procedure for using the trained model for associating image regions with audio subsequences.
The methods employed for this association are relatively straightforward combination of standard techniques with limited novelty. The trained model is used to compute alignment scores between densely sampled image regions and audio subsequences; from these alignment scores a number of heuristics are applied to associate clusters of image regions with clusters of audio subsequences.
MISSING CITATION
There is a lot of work in this area spanning computer vision, natural language, and speech recognition. One key missing reference:
Ngiam, et al. "Multimodal deep learning." ICML 2011
POSITIVE POINTS
- Using more data and an improved CNN architecture, this paper improves on prior work for bidirectional image/audio retrieval
- The presented method performs efficient acoustic pattern discovery
- The audio-visual grounding combined with the image and acoustic cluster analysis is successful at discovering audio-visual cluster pairs
NEGATIVE POINTS
- Limited novelty, especially compared with Harwath et al, NIPS 2016
- Although it gives good results, the clustering method has limited novelty and feels heuristic
- The proposed method includes many hyperparameters (patch size, acoustic duration, VAD threshold, IoU threshold, number of k-means clusters, etc) and there is no discussion of how these were set or the sensitivity of the method to these choices


Generative adversarial networks (GANs) are successful deep generative models. They are based on a two-player minimax game. However, the objective function derived in the original motivation is changed to obtain stronger gradients when learning the generator. We propose a novel algorithm that repeats density ratio estimation and f-divergence minimization. Our algorithm offers a new unified perspective toward understanding GANs and is able to make use of multiple viewpoints obtained from the density ratio estimation research, e.g. what divergence is stable and relative density ratio is useful.
1 INTRODUCTION
There have been many recent studies about deep generative models. Generative adversarial networks (GAN) (Goodfellow et al., 2014) is the variant of these models that has attracted the most attention. It has been demonstrated that generating vivid, realistic images from a uniform distribution is possible (Radford et al., 2015; Denton et al., 2015). GANs are formulated as a two-player minimax game. However, the objective function derived in the original motivation is modified to obtain stronger gradients when learning the generator. GANs have been applied in various studies; however, few studies have attempted to reveal their mechanism (Goodfellow, 2014; Huszar, 2015).
Recently, f-GAN, which minimizes the variational estimate of f-divergence, has been proposed (Nowozin et al., 2016). The original GAN is a special case of f-GAN.
In this study, we propose a novel algorithm inspired by GANs from the perspective of density ratio estimation based on the Bregman divergence, which we refer to as b-GAN. The proposed algorithm iterates density ratio estimation and f-divergence minimization based on the obtained density ratio. This study make the following two primary contributions:
1. We derive a novel unified algorithm that employs well-studied results regarding density ratio estimation (Kanamori et al., 2012; Sugiyama et al., 2012; Menon & Ong, 2016).
2. In the original GANs, the value function derived from the two-player minimax game does not match the objective function that is actually used for learning the generative model. In our algorithm, the objective function derived from the original motivation is not changed for learning the generative model.
The remainder of this study is organized as follows. Section 2 describes related work. Section 3 introduces and analyzes the proposed algorithm in detail. Section 4 explains the proposed algorithm for specific cases. Section 5 reports experimental results. Section 6 summarizes our findings and discusses future work.
2 RELATED WORK
In this study, we denote an input space as X and a hidden space as Z. Let p(x) be the distribution of training data over X and q(x) be the generated distribution over X .
GANs (Goodfellow et al., 2014) were developed based on a game theory scenario, where two model, i.e., a generator network and a discriminator network, are simultaneously trained. The generator network GθG(z) produces samples with a probability density function of q(x; θG). The discriminator network TθD (x) attempts to distinguish the samples from the training samples and that from the generator. GANs are described as a zero-sum game, where the function v(G,T ) determines the pay-off of the discriminator and the function −v(G,T ) determines the pay-off of the generator. The discriminator TθD (x) and generator GθG(z) play the following two-player minimax game minθG maxθD v(G,T ), where v(G,T ) can be expressed as follows:
Ex∼p(x)[log TθD (x)] + Ex∼q(x;θG)[log(1− TθD (x))].
The discriminator and generator are iteratively trained by turns. For fixed G, the optimal T(x) is p(x)p(x)+q(x) . This suggests that training the discriminator can be formulated as a density ratio estimation. The generator is trained to minimize v(G,T ) adversarially. In fact, maximizing Ex∼q(x;θG)[log TθD (x)] is preferable instead of minimizing Ex∼q(x;θG)[log(1− TθD (x))]. Although this does not match the theoretical motivation, this heuristic is the key to successful learning. We analyze this heuristic in Section 3.4.
f-GAN (Nowozin et al., 2016) generalizes the GAN concept. First, we introduce f-divergence (Ali & Silvey, 1966). The f-divergence measures the difference between two probability distributions p and q and is defined as
Df (p||q) = ∫ q(x)f ( p(x)
q(x)
) dx = ∫ q(x)f (r(x)) dx, (1)
where f(x) is a convex function satisfying f(1) = 0. Note that in the space of positive measures, i.e., not satisfying normalized conditions, f-divergence must satisfy f ′(1) = 0 due to its invariance (Amari & Cichoki, 2010).
The function v(G,T ) of f-GAN is given by
Ex∼p(x)[TθD (x)]− Ex∼q(x;θG)[f ∗ (TθD (x))], (2)
where f∗ is a Fenchel conjugate of f (Nguyen et al., 2010). In Eq. 2, v(G,T ) comes from,
∫ q(x)f ( p(x)
q(x)
) dx = sup
T Ex∼p[T (x)]− Ex∼q[f∗(T (x))]. (3)
Following GANs, θD is trained to maximize Eq. 2 in order to estimate the f-divergence. In contrast, θG is trained to adversarially minimize Eq. 2 to minimize the f-divergence estimate. However, as in GANs, maximizing Ex∼q[T (x)] is used rather than minimizing Ex∼q[−f∗(T (x))]. The latter optimization is theoretically valid in their formulation; however, they used the former heuristically. Similar to GANs, f-GAN also formulates the training discriminator as a density ratio estimation. For a fixed G, the optimal T (x) is f ′(pq ), where f ′ denotes the first-order derivative of f . When f(x) is
x log x− (x+ 1) log(1 +x), f-GANs are equivalent to GANs. Table 1 summarizes GAN and f-GAN. We denote the step for updating θD as D-step and the step for updating θG as G-step.
3 METHOD
As described in Section 2, training the discriminators in the D-step of GANs and f-GANs is regarded as density ratio estimation. In this section, we further extend this idea. We first review the density ratio estimation method based on the Bregman divergence. Then, we explain and analyze a novel proposed b-GAN algorithm. See appendix E for recent research related to density ratio estimation.
3.1 DENSITY RATIO MATCHING UNDER THE BREGMAN DIVERGENCE
There have been many studies on direct density ratio estimation, where a density ratio model is fitted to a true density ratio model under the Bregman divergence (Sugiyama et al., 2012). We briefly review this method.
Assume there are two distributions p(x) and q(x). Our aim is to directly estimate the true density ratio r(x) = p(x)q(x) without estimating p(x) and q(x) independently . Let rθ(x) be a density ratio model. The integration of the Bregman divergence Bf [r(x)‖rθ(x)] between the density ratio model and the true density ratio with respect to measure q(x)dx is
BDf (r||rθ) = ∫ Bf [r(x)‖rθ(x)]q(x)dx
= ∫ (f(r(x))− f (rθ(x))− f ′ (rθ(x)) (r(x)− rθ(x))) q(x)dx. (4)
We define the terms related to rθ in BDf (r||rθ) as
BRf (rθ) = ∫ (f ′(rθ(x))rθ(x)− f(rθ(x))) q(x)dx− ∫ f ′ (rθ(x)) p(x)dx (5)
= ∫ f ′(rθ(x)) (rθ(x)q(x)− p(x)) dx−Df (qrθ‖q). (6)
Thus, estimating the density ratio problem turns out to be the minimization of Eq. 5 with respect to θ.
3.2 MOTIVATION
In this section, we introduce important propositions required to derive b-GAN. Proofs of propositions are given in Appendix C. The following proposition suggests that the supremum of the negative of Eq. 5 is equal to the f-divergence between p(x) and q(x).
Prop 3.1. The following equation holds:
Eq
[ f ( p(x)
q(x)
)] = sup
rθ
Ex∼p[f ′ (rθ(x))]− Ex∼q[(f ′(rθ(x))rθ(x)− f(rθ(x)))]. (7)
The right side of Eq. 7 reaches the supremum when rθ(x) = r(x) is satisfied.
It has been shown that the supremum of negative of Eq. 5 is equivalent to the supremum of Eq. 2. Interestingly, the negative of Eq. 5 has a dual relation with the objective function of f-GAN, i.e., Eq. 2.
Prop 3.2. Introducing dual coordinates TθD = f ′(rθ)(Amari & Cichoki, 2010) yields the right side of Eq. 5 from Eq. 2.
Prop 3.2 shows that the D-step of f-GAN can be regarded as the density ratio estimation because Eq. 5 expresses the density ratio estimation and Eq. 2 is a value function of f-GAN.
3.3 B-GAN
Our objective is to minimize the f-divergence between the distribution of the training data p(x) and the generated distribution q(x). We introduce two functions constructed using neural networks: rθD (x) : X → R parameterized by θD, and GθG(z) : Z → X parameterized by θG. Measure q(x; θG)dx is a probability measure induced from the uniform distribution by GθG(z). In this case, rθD (x) is regarded as a density ratio estimation network and GθG(z) is regarded as a generator network for minimizing the f-divergence between p(x) and q(x).
Motivated by Section 3.2, we construct a b-GAN using the following two steps.
1. Update θD to estimate the density ratio between p(x) and q(x; θG). To achieve this, we minimize Eq. 5 with respect to rθ(x). In this step, the density ratio model rθ(x) in Eq. 5 can be considered as rθD (x) in this step.
2. Update θG to minimize the f-divergence Df (p||q) between p(x) and q(x; θG) using the obtained density-ratio. We are able to suppose that q(x; θG)rθ(x) is close to p(x). Instead of Df (p||q), we update θG to minimize the empirical approximation of Df (qrθ||q).
The b-GAN algorithm is summarized in Algorithm 1, where B is the batch size. In this study, a single-step gradient method (Goodfellow et al., 2014; Nowozin et al., 2016) is adopted.
Algorithm 1: b-GAN for number of training iterations do
sample X̂ = {x1, ..., xB} from p(x) and Ẑ = {z1, ..., zB} from an uniform distribution. D-step: Update θD:
θt+1D = θ t D −∇θD
( 1
B B∑ i=1 f ′ ( rθD (G(zi)) ) rθD (G(zi))− f ( rθD (G(zi)) ) − f ′ ( rθD (xi)
)) .
G-step: Update θG:
θt+1G = θ t G −∇θG
( 1
B B∑ i=1 f ( r(GθG(zi))
)) .
end for
In the D-step, the proposed algorithm estimates pq toward any divergence; thus, it differs slightly from the D-step of f-GAN because the estimated values, i.e., f ′(pq ), are dependent on the divergences. We also introduce an f-GAN-like update as follows. As mentioned in Section 2, we have two options in the G step.
1. D-step: minimize Ex∼p(x)[−f ′(rθD (x))] +Ex∼q(x)[f ′(rθD (x))rθD (x)− f(rθD (x))] w.r.t θD.
2. G-step: minimize Ex∼q(x;θG)[−f ′(r(x))] or Ex∼q(x;θG)[−f ′(r(x))r(x) + f(r(x))] w.r.t θG.
3.4 ANALYSIS
Following Goodfellow et al. [2014], we explain the validity of the G-step and D-step. We then explain the meaning of b-GAN. Finally, we analyze differences between b-GAN and f-GAN.
The density ratio is estimated in the D-step. The estimator of r(x) is an M-estimator and is asymptotically consistent under the proper normal conditions (Appendix E.1).
In the G-step, we update the generator as minimizing Df (p||q) by replacing p(x) with rθ(x)q(x). We assume that q(x; θG) is equivalent to p(x) when θG = θ∗, q(x; θG) is identifiable, and the optimal r(x) is obtained in the D-step. By our assumption, the acquired value in the G-step is θ̂, which minimizes the empirical approximation of Df (r(x)q(x; θG)‖q(x; θG)) = Ex∼q(x;θG)[r(x)].
When θG is equal to θ∗, this equation is equal to 0. The estimator θ̂ can be considered as a kind of Z-estimator.
Usually, we cannot perform only a G-step because we do not know the form of p(x) and q(x). In b-GAN, Df (p||q) can be minimized by estimating the density ratio r(x) without estimating the densities directly.
In fact, the r(x) obtained at each iteration is different and not optimal because we adopt a single-step gradient method (Nowozin et al., 2016). Thus, b-GAN dynamically updates the generator to minimize the f-divergence between p(x) and q(x). As mentioned previously, f(x) must satisfy f ′(1) = 0 in this case because we cannot guarantee that rθ(x)q(x) is normalized.
Similar to GANs, the D-step and G-step work adversarially. In the D-step, rθ(x) is updated to fit the ratio between p(x) and q(x). In the G-step, q(x) changes, which means rθ(x) becomes inaccurate in terms of the density ratio estimator. Next, rθ(x) is updated in the D-step so that it fits the density ratio of p(x) and the new q(x). This learning situation is derived from Eq. 6 , which shows that θD is updated to increase Df (qrθ‖q) in the D-step. In contrast, θG is updated to decrease Df (qrθ‖q) in the G-step.
In Section 3.3, we also introduced a f-GAN-like update. Three choices can be considered for the G-step:
(1)Ex∼q(x:θG)[f(r(x))], (2)Ex∼q(x;θG)[−f ′(r(x))], (3)Ex∼q(x;θG)[−f ′(r(x))r(x) + f(r(x))].
Note that f is a convex function, f(1) = 0, and f ′(1) = 0. It is noted in (Nowozin et al., 2016) that case (2) works better than case (3) in practice. We also confirm this. The complete reason for this is unclear. However, we can find a partial reason by differentiating objective functions with respect to r. The derivatives of the objective functions are
(1)f ′(r), (2)− f ′′(r), (3)− rf ′′(r).
All signs are negative when r(x) is less than 1. Usually, when x is sampled from q(x), r(x) is less than 1. Therefore, we speculate that r(x) is less than 1 during most of the learning process when x is sampled from
q(x). When r(x) is small, the derivative is also small in (3) because the term r(x) is multiplied. Therefore, the derivative tends to be small in (3). The mechanism pulling r(x) to 1 does not work when r(x) is small. Thus, the case of (3) does not work well. A similar argument was proposed by Goodfellow et al. (2014) and Nowozin et al. (2016).
In our experimental case of (1) and (2) work properly (Section 5). The reason case (2) works is that function−f ′(r) behaves like an f-divergence and the derivative is large when r(x) is small. However, we cannot guarantee that −f ′(r) satisfies the conditions of f-divergence between positive measures, i.e,−f ′(r) is a convex function and−f ′′(1) = 0. If the derivatives in case (2) are negative when r(x) is greater than 1, there is a possibility that the mechanism pulling r(x) to 1 does not occur. In contrast, in case (1), when r(x) is greater than 1, the derivatives are positive, therefore, the mechanism pulling r(x) to 1 occurs. This prevents generators from emitting the same points. We can expect the same effects as -minibatch discrimination- (Salimans et al., 2016).
Throughout the analysis, we can easily extend the algorithm of b-GAN by using different divergences in the G-step and D-step. The original GAN can be regarded as one of such algorithms.
4 ALGORITHMS FOR SPECIFIC CASES
We adopt α-divergence in the positive measure space as the f-divergence (Amari & Cichoki, 2010). Here, we explain specific algorithms when f is α-divergence. Then, we explain some heuristics used in b-GAN.
4.1 ALGORITHMS WITH α-DIVERGENCE
In α-divergence, f(r) in Eq. 1 is
fα(r) =  4 1−α2 (1− r 1+α 2 ) + 21−α (r − 1) (α 6= ±1)
r log r − r + 1 (α = 1) − log r + r − 1 (α = −1).
(8)
The objective function derived from α-divergence is summarized as follows.
4.2 HEURISTICS
We describe some heuristic methods that work for our experiments. The heuristics introduced here are justified theoretically in Appendix C.
In the initial learning process, empirical distribution p and generated distribution q are completely different. Therefore, the estimated density ratio r(x) = p(x)q(x) is enormous when x is taken from p and tiny when x is taken from q. It seems that the learning does not succeed in this case. In fact, in our setting, when the final activation function of rθD (x) is taken from functions in the range (0,∞), b-GAN does not properly work. Therefore, we use a scaled sigmoid function such as a two-times sigmoid function. A similar idea has also been used in (Cortes et al., 2010).
As mentioned, density ratio p(x)q(x) is extremely sensitive. To avoid this problem, in the D-step of the KL-divergence, we also conducted experiments wherein we estimated pαp+(1−α)q (where α is small.) rather than p(x)q(x) . The same idea is introduced in the covariate shift situation (Sugiyama et al., 2013). A similar idea has also been used for GAN learning (Salimans et al., 2016) and class probability estimation (Reid & Williamson, 2010).
5 EXPERIMENTS
We conducted experiments to establish that the proposed algorithm works properly and can successfully generate natural images. The proposed algorithm is based on density ratio estimation; therefore, knowledge regarding the density ratio estimation can be utilized. In the experiments, using the Pearson divergence and estimating the relative density ratio is shown to be useful for stable learning. We also empirically confirm our statement in Section 3.4, i.e., f-divergence is increased when learning θD and decreased when learning θG.
5.1 SETTINGS
We applied the proposed algorithm to the CIFAR-10 data set (Krizhevsky, 2009) and Celeb A data set (Liu et al., 2015) because they are often used in GAN research (Salimans et al., 2016; Goodfellow et al., 2014). The images size are 32× 32 pixels. All results in this section are analyzed based on the results of the CIFAR-10 data set. The results for the Celeb A data set are presented in Appendix B. Our network architecture is nearly equivalent to that of previous study (Radford et al., 2015) (refer to the appendix A,B for details). Note that unless stated otherwise, the last layer function of rθD (x) is a sigmoid function multiplied by two. We used the TensorFlow for automatic differentiation (Abadi et al., 2015). For stochastic optimization, Adam was adopted (Kingma & Ba, 2014).
5.2 RESULTS
Figure 2 shows the density ratio estimate rθD (x) and loss values of the generators. For each divergence, we conducted four experiments with 40,000 epochs, where the initial learning rate value was fixed (5 × 10−5) with the exception of reversed KL divergence. These results show that the b-GANs using Pearson divergence are stable because the learning did not stop. The same results have been reported in the research into density ratio estimation (Yamada et al., 2011). In contrast, b-GANs using the KL divergence are unstable. In fact, the learning stopped between the 20,000th and 37,000th epoch when the learning rate was not as small. When we use a heuristic method, i.e.,
estimating the relative density ratio as described in Section 4.2, this problem is solved. For reversed KL divergence, the learning stopped too soon if the initial learning rate value was 5× 10−5. If the learning rate was 1× 10−6, the learning did not stop; however, it was still unstable. In Figure 2, the last layer activation function of the b-GANs is a twofold sigmoid function. In Figure 3, we use a sigmoid function multiplied by five. The results indicate that the estimated density ratio values approach one. We also confirm that the proposed algorithm works with sigmoid functions at other scales.
Figure 4 shows the estimated f-divergence Df (qrθ||q) before the G-step subtracted by Df (qrθ||q) after the G-step. Most of the values are greater than zero, which suggests f-divergence decreases at every G-step iteration. This observation is consistent with our analysis in Section 3.4.
Note that learning is successful with an f-GAN-like update when minimizing Eq[−f ′(r)]. However, the learning f-GAN-like update when minimizing Eq[f(r) − rf ′(r)] did not work well for our network architecture and data set.
6 CONCLUSIONS AND FUTURE WORK
We have proposed a novel unified algorithm to learn a deep generative model from a density ratio estimation perspective. Our algorithm provides the experimental insights that Pearson divergence and estimating relative density ratio are useful to improve the stability of GAN learning. Other insights regarding density ratio estimation would also be also useful. GANs are sensitive to data sets, the form of the network and hyper-parameters. Therefore, providing methods to improve GAN learning is meaningful.
Related research to our study that focuses on linking density ratio and a GAN, has been performed by explaining specific algorithms independently (Mohamed & Lakshminarayanan, 2016). In contrast, our framework is more unified.
In future, the following things should be considered.
• What is the optimal divergence? In research regarding density ratio estimation, the Pearson divergence (α = 3) is considered robust (Nam & Sugiyama, 2015). We empirically and theoretically confirmed the same property when learning deep generative models. It is also reported that using KL-divergence and reversed KL-divergence is not robust as scoring rules (Dawid et al., 2015). For generating realistic images, the reversed KL divergence (α = −1) is preferred because it is mode seeking (Huszar, 2015). However, if α is small, the density ratio estimation becomes inaccurate. For a robust density ratio, using power divergence has also been proposed (Sugiyama et al., 2012). The determination of the optimal divergence is a persistent problem (Appendix E).
• What should be estimates in D-step? In the D-step of b-GAN, r(x) is estimated. However, in the original GAN, r(x)/(1 + r(x)) is estimated. As unnormalized models, the latter is more robust than estimating r(x) (Pihlaja et al., 2010) (Appendix E.7). • We can consider algorithms that use different divergences in the G-step and D-step. In that
case, choice of the divergences are more diverse. Original GANs are described in such algorithms as mentioned in Section 3.5. • We can consider algorithms that use multiple divergences. This may improve the stability of
learning. • When sampling from q(x), if the objective is sampling from real data p(x), r(x) should be
multiplied. Hence, the density ratio is also useful when using samples from q(x). How to use samples obtained from generators meaningfully is an remaining important problem.
ACKNOWLEDGEMENTS
The authors would like to thank Masanori Misono for technical assistance with the experiments. We are grateful to Masashi Sugiyama, Makoto Yamada, and the members of the Preferred Networks team.
A CIFAR-10 DATASET
Figure 5 shows samples generated randomly using b-GANs. These results indicate that b-GANs can create natural images successfully. We did not conduct a Parzen window density estimation for the evaluations because of Theis et al., [2016].
Here, we describe the network architecture of rθD (x) and GθG(z) used in the b-GAN. BN is the batch normalization layer (Sergey & Christian, 2015).
A.1 rθD (x)
x→ Conv(3, 64)→ lRelu→ Conv(64, 256)→ BN→ lRelu→ Conv(256, 512)→ BN→ lRelu→ Reshape(4× 4× 512)→ Linear(4× 4× 512, 1)→ 2× Sigmoid
A.2 GθG(z)
z→ Linear(100,4× 4× 512)→ BN→ Relu→ Reshape(4, 4, 512)→ Conv(512, 256)→ BN→ Relu→ Conv(256, 64)→ BN→ Relu→ Conv(64, 3)→ tanh
B CELEB A DATASET
We also applied our algorithm to the Celeb A data set. The images are resized and cropped to 64× 64 pixcels. Figures 6 and 7 show samples randomly generated using b-GANs. The Network architecture is as follows.
B.1 rθD (x)
x→ Conv(3, 64)→ lRelu→ Conv(64, 128)→ BN→ lRelu→ Conv(128, 256)→ BN→ lRelu→ Conv(256, 512)→ BN→ lRelu→ Reshape(4× 4× 512)→ Linear(4× 4× 512, 1)→ 2×Sigmoid
B.2 GθG(z)
z→ Linear(64, 4 × 4 × 512)→ BN→ Relu→ Reshape(4, 4, 512)→ Conv(512, 256)→ BN→ Relu→ Conv(256, 128)→ BN→ Relu→ Conv(128, 64)→ BN→ Relu→ Conv(64, 3)→ tanh
C PROOF OF PROPOSITIONS IN SECTION 3
C.1 PROOF OF PROP 3.1
From Eq. 4, the following equation holds,
Ex∼p [f ′ (rθ(x))]− Ex∼q[(f ′(rθ(x))rθ(x)− f(rθ(x)))] = −BDf (r||rθ) + Eq [ f ( p(x)
q(x)
)] . (9)
Using BDf (r||rθ) ≥ 0 yields Eq. 7. We have BDf (r||rθ) = 0 when r is equal to rθ. Thus, the equality holds if and only if r is equal to rθ.
C.2 PROOF OF PROP3.2
r.h.s of Eq.2 = Ex∼p(x)[f ′(rθ(x))]− Ex∼q(x)[f∗ (f ′(rθ(x)))] (10)
= Ex∼p(x)[f ′(rθ(x))]− Ex∼q(x)[f(rθ(x))rθ(x)− f(rθ(x))] = Ex∼p(x)[f ′ (rθ(x))]− Ex∼q(x)[(f ′(rθ(x))rθ(x)− f(rθ(x)))]
= the negative of Eq. 5 In the derivation, we have used the following equation f∗ ( f ′(rθ(x)) ) = f(rθ(x))rθ(x)− f(rθ(x)) .
D EXPLANATION OF HEURISTICS USING RADEMACHER COMPLEXITY
Here, we justify three points using Rademacher complexity. All techniques used are described in the literature (Mohri et al., 2012).
• Pearson divergence is preferable to KL divergence in density ratio estimation.
• Relative density ratio is useful (introduced in Sec 4.2).
• The meaning of bounding the last output functions of discriminators (introduced in Sec 4.2).
Our objective is estimating r(x) = p(x)q(x) . We define hypothesis sets as H. In Section 3.4, we assume H as parametric models for simplicity. In this section, we do not restrict H to parametric models. We define r0 and rs as
r0 = argminh∈HBRf (h)
rs = argminh∈HB̂Rf (h),
where B̂R is an empirical approximation of BR. Note that BRf (h) reaches minimum values if and only if r = h holds. We want to analyze BRf (rs)−BRf (r). The regret, i.e., BRf (rs)−BRf (r) can be bounded as follows:
BRf (rs)−BRf (r) = BRf (rs)−BRf (r0) +BRf (r0)−BRf (r) ≤ 2 sup
h∈H |BRf (h)− B̂Rf (h)|+BRf (r0)−BRf (r). (11)
The first term of Eq. 11 is bounded by uniform law of large numbers. To do that, assume that all elements in H are bounded by constant C. First, we consider the case when f is the Pearson divergence. In that case, BRf (h) is
Ex∼p(x)[0.5h(x) 2]− Ex∼q(x)[h(x)]. (12)
We denote the Rademacher complexity of H as Rm,q when x is sampled from q(x) and the sample size is m. The first term of Eq. 12 is upper bounded by Talagrand’s lemma. For any δ with probability as least 1− δ, we have
sup h∈H |Ex∼p(x)[0.5h(x)2]−
1
m
∑ 0.5h(xi) 2| ≤ CRm,p(H) + 0.5C2 √
log 1δ 2m , (13)
where samples are taken from p(x) independently. the fact that 0.5h(x)2 is C-Lipchitz over the interval (0, C) is used (C is a constant real number). The first term of Eq. 12 is upper-bounded by Talagrand’s lemma. For any δ with probability at least 1− δ, we have
sup h∈H |Ex∼q(x)[h(x)]−
1
m
∑ h(xi)| ≤ Rm,q(H) + C √ log 1δ 2m , (14)
where samples are taken from q(x) independently. By combining Eq. 13 and Eq. 14, for any δ with probability as least 1− δ, the first term of Eq. 11 is bounded by
2CRm,p(H) + 2Rm,q(H) + (C2 + 2C) √ log 12δ 2m . (15)
When f is the KL-divergence, BRf (h) is Ex∼q(x)[h(x)]− Ex∼p(x)[log h(x)]. (16)
The second term of Eq. 16 cannot be bounded because log(x) is no longer Lipchitz continuous over (0, C). This explains why Pearson divergence is preferable to KL divergence in density ratio estimation. However, if h(x) is lower-bounded by the constant real number C, the problem is solved. Using the relative density ratio has the same effect.
In our setting, r(x) is a sigmoid function multiplied by C. If C is large, the approximation error, i.e., BRf (r0)−BRf (r) is small. However, there is a possibility that the estimation error increases according to Eq. 15, which may lead to the learning instability.
E SUMMARY OF DENSITY RATIO ESTIMATION RESEARCH
In this section, we review researches related to density ratio estimation, entangling them to b-GAN. The Eq. 4 has also been used in class probability density estimation and unnormalized models research. We briefly describe research that is closely connected to density ratio estimation and extract ideas that can also be applied to b-GAN.
E.1 NOTATION
We summarize notations frequently used in this section.
• r(x) = p(x)/q(x) Density ratios between p(x) and q(x). In b-GAN, p(x) is the probability density function of ”real data” and q(x) is the probability density function of ”generated data”.
• x(n)i Samples taken from p(x).
• x(d)i Samples taken from q(x).
• D = (p, q, π) The joint distribution of X and Y . The variable X has the the mixture distribution of πp + (1 − π)q and Y is a binary label taking values in {−1,+1} which satisfies π = P (Y = 1); thus, (p, q) = (P (X|Y = 1), P (X|Y = −1)) holds.
• η The Bayes optimal estimator P (Y = 1|X) for binary classification.
• l : {−1, 1} × [0, 1]→ R Loss function
• h1 : X → [0, 1] An element of hypothesis sets. In this case, the objective is estimating P (Y = 1|x).
• hg : X → V An element of hypothesis sets. V is a subset of R. Note that h1 is a special case of hg .
• Ψ : [0, 1]→ V A link function.
• k(x, ·) A positive definite and characteristic (Fukumizu et al., 2004) kernel on the measurable space ofR.
• Hk A reproducing kernel Hilbert space with a kernel k (Steinwart, 2011). An inner product in Hk is 〈·, ·〉.
• Φ : R → Hk A characteristic function:x→ k(x, ·).
• Xp A random variable with probability density function p.
• Bf [p‖q] Bregman divergence with f between p and q.
E.2 THEORETICAL ASPECTS OF DENSITY RATIO ESTIMATION
Following Kanamori et al. (2012), we expand the explanation of density ratio estimation. As noted in Section 3, the objective function of density ratio estimation is
B̂Rf (rθ) = 1
n n∑ i=1 ( f ′(rθ(x (d) i ))rθ(x (d) i )− f(rθ(x (d) i )) ) − 1 n n∑ i=1 f ′ ( rθ(x (n) i ) ) , (17)
which is an empirical approximation of Eq. 5. The estimator θ̂n is obtained by minimizing Eq. 17. The estimator θ̂n is an M-estimator because Eq. 17 is a form of 1n ∑n i=1m(x), where m is a function of x and x ∼ (xdi , xni ) holds. The estimator θ̂n satisfies consistency under mild conditions (see Prop 7.4. (Hayashi, 1997)). The essential condition is the expectation of Eq. 17 is uniquely minimized when rθ is equivalent to the true density ratio r. We have proved that proposition in Appendix D. Asymptotic normality also holds under suitable conditions (see Prop 7.8. (Hayashi, 1997)).
By differentiating Eq. 17 with respect to θ, the above method can be regarded as a type of moment matching as follows:
∂BRf (rθ)
∂θ =
1
n n∑ i=1 f ′′(rθ(x (d) i )r ′ θ(x (d) i )rθ(x (d) i )− 1 n n∑ i=1 f ′′(rθ(x (n) i ))r ′ θ(x (n) i ). (18)
The estimator θ̂n is attained when Eq.18 is equal to 0. By substituting f ′′(rθ(xi)r′θ(xi) with s(xi), Eq. 18 is reduced to be
1
n n∑ i=1 s(x (d) i )rθ(x (d) i )− 1 n n∑ i=1 s(x (n) i ). (19)
The above Eq. 18 is a form of moment matching.1 What is the optimal s(x)? Here, we focus on the variance of the estimator (efficiency). It is known that the Eq. 17 derived from the logistic model is known to be optimal (Qin, 1998). It is a natural consequence because the logistic model can be considered as a maximum likelihood and maximum likelihood reaches the Cramer-Rao bound asymptotically. Typically, general moment matching (GMM) achieves the lower bound when the estimating equation is the score of observations, i.e., when GMM is identical to maximum likelihood.
Efficiency is not the absolute criterion for choosing losses. For example, when there are many outliers in the data, robustness is more important than efficiency. In reality, an M estimator was introduced in the contest of robust statistics (Huber & Ronchetti, 2009).
E.3 CLASS PROBABILITY ESTIMATION
We have explained density ratio estimation, starting with the Bregman divergence. Importantly, density-ratio estimation is equivalent to class probability estimation. For details, see (Reid & Williamson, 2011; 2010; Dawid & Musio, 2014; Menon & Ong, 2016).
As in Section E.1, we introduce the variable Y . The joint distribution of X and Y is denoted as D = (p, q, π). Class probability estimation can be regraded as a minimization problem of the empirical estimation of the full risk E(X,Y )∼D[l(Y, h1)], which is denoted L(h1;D, l) (h1 is a hypothesis element). When the hypothesis h1, which minimizes E(X,Y )∼D[l(Y, h1)], is the Bayes optimal estimator η(x) uniquely, such a loss is called a proper loss.
A proper loss is naturally extended to a composite proper loss by introducing a link function Φ. In this case, the objective is estimating Φ(η(x)) correctly. The estimator is obtained by the empirical minimization of full risk E(X,Y )∼D[lΦ(Y, hg)] when lΦ(Y, hg) means l(Y,Φ−1(hg(x))) and Φ(h1(x)) is the same as hg(x). When the hypothesis hg(x), which minimizes E(X,Y )∼D[lΦ(Y, hg)], is the Bayes optimal estimator Φ(η(x)) uniquely, such a loss is called a proper composite loss with a link function Φ.
The conditional risk (conditioned on x) EY∼η[lΦ(Y, hg)] can be decomposed to
EY∼η[l Φ(Y, hg)] = ηL1(h g) + (1− η)L−1(hg). (20)
The conditional Bayes risk is
ηL1(Φ(η(x))) + (1− η)L−1(Φ(η(x))) = ηλ+1(η) + (1− η)λ−1(η), (21)
where λ+1(x) = L1(Φ(x)) and λ−1(x) = L−1(Φ(x)). We set Eq. 21, i.e., xλ+1(x)+(1−x)λ−1(x) as c(x). The regret of the composite proper loss can be written using Bregman divergence
L(hg;D, l)− L(Φ ◦ η;D, l) = EX [ Bc [ η(X)‖Φ−1(hg(X)) ]] . (22)
1In usual moment matching, s(x) is not be dependent on the form of probability density function. However, it depends on the form of rθ in this case.
The problem of minimizing the composite proper loss turns out be the density ratio estimation problem by setting Φ(x) as u/(1− u) and π = 0.5 (Menon & Ong, 2016). In this case, the LHS of Eq. 22 can be written as EX∼q [Bc⊗ [Φ(η(X))‖hg(X)]] (Φ(η(X)) = r(x)), where c⊗ is given by
c⊗ : x→ (1 + x)c ( x
1 + x
) .
The equation EX∼q [Bc⊗ [r(X)‖hg(X)]] corresponds to Eq. 4 by substituting c⊗ with f and hg with rθ.
What loss is the optimal loss? The above loss can be written in another form using weight by transforming Eq. 22 further. Determining what loss is better has been analyzed from the perspective of weight. For example, Reid & Williamson (2010) proposed the ”minimal symmetric convex proper loss” for surrogate loss. Regarding density ratio, Menon & Ong (2016) suggest that Pearson divergence is robust because the weight of Pearson divergence is uniform. However, according to their covariate shift experiment, Pearson divergence was not significantly superior to other divergence.
What is the difference between density ratio estimation (class probability estimation) and classification? The objective and assumption differ. As for assumption, in density ratio, the situation where p(x) and q(x) are overlapping would be preferable. However, in classification, the situation where p(x) and q(x) separate would be preferable. In addition, the objective of classification is slightly different from estimating a Bayes rule correctly. Margin loss is widely used in classification rather than zero-one loss. The theoretical guarantee of using margin loss for classification is that it is included in class calibrated loss (Bartlett et al., 2006). However, the margin loss is not equivalent to a proper loss, i.e, it is often not suitable for estimating η. The condition whereby margin loss is a proper loss is explained by Reid & Williamson (2010). A GAN using margin loss has been proposed (Zhao et al., 2016). Note that using margin loss is not supposed in b-GAN. They succeeded in generating high resolution images.
E.4 ROBUST LOSS AND DIVERGENCE
What is robust loss and divergence? Basu et al. (1998) proposed a robust divergence called power divergence (Basu et al., 1998), which is given as Bνβ [p‖q], where νβ is
νβ(x) = 1
β(β + 1) (xβ+1 − (β + 1)x+ β). (23)
This is also called β-divergence (Amari & Cichoki, 2010). The robust estimation equation is derived from power divergence compared to maximum likelihood. By setting f as νβ , robust density ratio estimation has been proposed (Sugiyama et al., 2012). In this case, the objective function is Ex∼q(x)[Bνβ [r‖rθ]]. Dawid et al. (2015) analyze robust proper loss from the perspective of influence function. They proposed a concept of B-robustness from the perspective of influence function. It is stated that using KL divergence and reversed KL divergence is not robust because the second derivative of f is not bounded at 0. That is a similar conclusion to our analysis in Appendix D.
E.5 KERNEL METHODS
We assume that k(x, ·) is a positive definite kernel. When X is a random variable taking values inR and Ψ(X) is a random variable taking values inHk with a characteristic map Ψ : x→ k(·, x), we can think of the mean of random variable Φ(X) denoted as mkX taking values inHk, which satisfy 〈f,mkX〉 = E[〈f,Ψ(X)〉] = E[f(x)](∀f ∈ Hk) and mkX(y) = 〈mkX , k(·, y)〉 = E[k(X, y)]. If the kernel is characteristic, the bijective from all measures onR to Hk exists such that a measure p(x)dx corresponds to the mean mkXp (Fukumizu et al., 2004). When there are random variables Xp and Xq , we can measure the distance between Xp and Xq by calculating ‖mkXp −m k Xq ‖2Hk .
As the density ratio estimation methods using kernels, the objective function is the empirical approximation of ‖mkXqrθ −m k Xp ‖2Hk . As generative moment matching networks (GMMN), the objective
function is ‖mkXq −m k Xp ‖2Hk (Dziugaite et al., 2015; Li et al., 2015). GMMNs seem to be superior to b-GAN because they can be trained without density ratio. However, the choice of kernels is difficult. In addition, an autoencoder appears to be required for generating complex data.
E.6 F-DIVERGENCE ESTIMATION AND TWO SAMPLE TEST
We consider the problem of f-divergence estimation between p(x) and q(x). This is applied straightforwardly to a two-sample test. Variational f-divergence estimation using Eq. 3 is proposed (Nguyen et al., 2010). In addition, the two step method, i.e., first estimating density ratio and then estimating f-divergence, is proposed (Kanamori et al., 2012). This method is also applied to a two-sample test. The latter method is similar to b-GAN. A kernel two sample test is also introduced calculating ‖mkXq −m k Xp ‖2Hk in (Gretton et al., 2012).
E.7 UNNORMALIZED MODELS
When the model p0(xm;φ) is unnormalized, the unified method including noise contrastive estimation was proposed in (Pihlaja et al., 2010; Gutmann & Hirayama, 2011; Gutmann & Hyvarinen, 2010).
In this case, the objective is estimating θ = {φ,C} when the log-likelihood of normalized model log p(x; θ) is equal to log p0(x;φ) + C and C is a normalizing constant. Compared to b-GAN, the auxiliary distribution q(x) is known. The parameter θ can be estimated as a minimization problem of Eq. 17 by replacing rθ(x) with p(x; θ)/q(x) (Pihlaja et al., 2010). As similar algorithm, the method estimating rθ first, then estimating p(x; θ) as rθ(x)q(x) is suggested by Gutmann & Hirayama (2011). Note that the latter method is similar to b-GAN.
Pihlaja et al. (2011) analyzed what loss is better by differentiating loss. They experimentally confirmed that noise contrastive estimation is robust with respect to the choice of the auxiliary distribution.

In this paper, the authors extend the f-GAN by using Bregman divergences for density ratio matching. The argument against f-GAN (which is a generalization of the regular GAN) is that the actual objective optimized by the generator during training is different from the theoretically motivated objective due to gradient issues with the theoretically motivated objective. In b-GANS, the discriminator is a density ratio estimator (r(x) = p(x) / q(x)), and the generator tries to minimize the f-divergence between p and q by writing p(x) = r(x)q(x).
My main problem with this paper is that it is unclear why any of this is useful. The connection to density estimation is interesting, but any derived conclusions between the two seem questionable. For example, in previous density estimation literature, the Pearson divergence is more stable. The authors claim that the same holds for GANS and try to show this in their experiments. Unfortunately, the experiments section is very confusing with unilluminating figures. Looking at the graph of density ratios is not particularly illuminating. They claim that for the Pearson divergence and modified KL-divergence, "the learning did not stop" by looking at the graph of density ratios. This is completely hand-wavey and no further evidence is given to back this claim. Also, why was the normal GAN objective not tried in light of this analysis? Furthermore, it seems that despite criticizing normal GANs for using a heuristic objective for the generator, multiple heuristics objectives and tricks are used to make b-GAN work.
I think this paper would be much improved if it was rewritten in a clear fashion. As it stands, it is difficult to understand the motivation or intuition behind this work.