TOPICRNN: A RECURRENT NEURAL NETWORK WITH LONG-RANGE SEMANTIC DEPENDENCY
In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence – both semantic and syntactic – but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned endto-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis on the IMDB movie review dataset and report an error rate of 6.28%. This is comparable to the state-of-the-art 5.91% resulting from a semi-supervised approach. Finally, TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.
1 INTRODUCTION
When reading a document, short or long, humans have a mechanism that somehow allows them to remember the gist of what they have read so far. Consider the following example:
“The U.S.presidential race isn’t only drawing attention and controversy in the United States – it’s being closely watched across the globe. But what does the rest of the world think about a campaign that has already thrown up one surprise after another? CNN asked 10 journalists for their take on the race so far, and what their country might be hoping for in America’s next —”
The missing word in the text above is easily predicted by any human to be either President or Commander in Chief or their synonyms. There have been various language models – from simple ngrams to the most recent RNN-based language models – that aim to solve this problem of predicting correctly the subsequent word in an observed sequence of words.
A good language model should capture at least two important properties of natural language. The first one is correct syntax. In order to do prediction that enjoys this property, we often only need to consider a few preceding words. Therefore, correct syntax is more of a local property. Word order matters in this case. The second property is the semantic coherence of the prediction. To achieve
∗Work was done while at Microsoft Research.
this, we often need to consider many preceding words to understand the global semantic meaning of the sentence or document. The ordering of the words usually matters much less in this case.
Because they only consider a fixed-size context window of preceding words, traditional n-gram and neural probabilistic language models (Bengio et al., 2003) have difficulties in capturing global semantic information. To overcome this, RNN-based language models (Mikolov et al., 2010; 2011) use hidden states to “remember” the history of a word sequence. However, none of these approaches explicitly model the two main properties of language mentioned above, correct syntax and semantic coherence. Previous work by Chelba and Jelinek (2000) and Gao et al. (2004) exploit syntactic or semantic parsers to capture long-range dependencies in language.
In this paper, we propose TopicRNN, a RNN-based language model that is designed to directly capture long-range semantic dependencies via latent topics. These topics provide context to the RNN. Contextual RNNs have received a lot of attention (Mikolov and Zweig, 2012; Mikolov et al., 2014; Ji et al., 2015; Lin et al., 2015; Ji et al., 2016; Ghosh et al., 2016). However, the models closest to ours are the contextual RNN model proposed by Mikolov and Zweig (2012) and its most recent extension to the long-short term memory (LSTM) architecture (Ghosh et al., 2016). These models use pre-trained topic model features as an additional input to the hidden states and/or the output of the RNN. In contrast, TopicRNN does not require pre-trained topic model features and can be learned in an end-to-end fashion. We introduce an automatic way for handling stop words that topic models usually have difficulty dealing with. Under a comparable model size set up, TopicRNN achieves better perplexity scores than the contextual RNN model of Mikolov and Zweig (2012) on the Penn TreeBank dataset 1. Moreover, TopicRNN can be used as an unsupervised feature extractor for downstream applications. For example, we derive document features of the IMDB movie review dataset using TopicRNN for sentiment classification. We reported an error rate of 6.28%. This is close to the state-of-the-art 5.91% (Miyato et al., 2016) despite that we do not use the labels and adversarial training in the feature extraction stage.
The remainder of the paper is organized as follows: Section 2 provides background on RNN-based language models and probabilistic topic models. Section 3 describes the TopicRNN network architecture, its generative process and how to perform inference for it. Section 4 presents per-word perplexity results on the Penn TreeBank dataset and the classification error rate on the IMDB 100K dataset. Finally, we conclude and provide future research directions in Section 5.
2 BACKGROUND
We present the background necessary for building the TopicRNN model. We first review RNN-based language modeling, followed by a discussion on the construction of latent topic models.
2.1 RECURRENT NEURAL NETWORK-BASED LANGUAGE MODELS
Language modeling is fundamental to many applications. Examples include speech recognition and machine translation. A language model is a probability distribution over a sequence of words in a predefined vocabulary. More formally, let V be a vocabulary set and y1, ..., yT a sequence of T words with each yt ∈ V . A language model measures the likelihood of a sequence through a joint probability distribution,
p(y1, ..., yT ) = p(y1) T∏ t=2 p(yt|y1:t−1).
Traditional n-gram and feed-forward neural network language models (Bengio et al., 2003) typically make Markov assumptions about the sequential dependencies between words, where the chain rule shown above limits conditioning to a fixed-size context window.
RNN-based language models (Mikolov et al., 2011) sidestep this Markov assumption by defining the conditional probability of each word yt given all the previous words y1:t−1 through a hidden
1Ghosh et al. (2016) did not publish results on the PTB and we did not find the code online.
state ht (typically via a softmax function):
p(yt|y1:t−1) , p(yt|ht), ht = f(ht−1, xt).
The function f(·) can either be a standard RNN cell or a more complex cell such as GRU (Cho et al., 2014) or LSTM (Hochreiter and Schmidhuber, 1997). The input and target words are related via the relation xt ≡ yt−1. These RNN-based language models have been quite successful (Mikolov et al., 2011; Chelba et al., 2013; Jozefowicz et al., 2016).
While in principle RNN-based models can “remember” arbitrarily long histories if provided enough capacity, in practice such large-scale neural networks can easily encounter difficulties during optimization (Bengio et al., 1994; Pascanu et al., 2013; Sutskever, 2013) or overfitting issues (Srivastava et al., 2014). Finding better ways to model long-range dependencies in language modeling is therefore an open research challenge. As motivated in the introduction, much of the long-range dependency in language comes from semantic coherence, not from syntactic structure which is more of a local phenomenon. Therefore, models that can capture long-range semantic dependencies in language are complementary to RNNs. In the following section, we describe a family of such models called probabilistic topic models.
2.2 PROBABILISTIC TOPIC MODELS
Probabilistic topic models are a family of models that can be used to capture global semantic coherency (Blei and Lafferty, 2009). They provide a powerful tool for summarizing, organizing, and navigating document collections. One basic goal of such models is to find groups of words that tend to co-occur together in the same document. These groups of words are called topics and represent a probability distribution that puts most of its mass on this subset of the vocabulary. Documents are then represented as mixtures over these latent topics. Through posterior inference, the learned topics capture the semantic coherence of the words they cluster together (Mimno et al., 2011).
The simplest topic model is latent Dirichlet allocation (LDA) (Blei et al., 2003). It assumes K underlying topics β = {β1, . . . , βK} , each of which is a distribution over a fixed vocabulary. The generative process of LDA is as follows: First generate the K topics, βk ∼iid Dirichlet(τ). Then for each document containing words y1:T , independently generate document-level variables and data:
1. Draw a document-specific topic proportion vector θ ∼ Dirichlet(α).
2. For the tth word in the document,
(a) Draw topic assignment zt ∼ Discrete(θ). (b) Draw word yt ∼ Discrete(βzt).
Marginalizing each zt, we obtain the probability of y1:T via a matrix factorization followed by an integration over the latent variable θ,
p(y1:T |β) = ∫ p(θ) T∏ t=1 ∑ zt p(zt|θ)p(yt|zt, β)dθ = ∫ p(θ) T∏ t=1 (βθ)ytdθ. (1)
In LDA the prior distribution on the topic proportions is a Dirichlet distribution; it can be replaced by many other distributions. For example, the correlated topic model (Blei and Lafferty, 2006) uses a log-normal distribution. Most topic models are “bag of words” models in that word order is ignored. This makes it easier for topic models to capture global semantic information. However, this is also one of the reasons why topic models do not perform well on general-purpose language modeling applications such as word prediction. While bi-gram topic models have been proposed (Wallach, 2006), higher order models quickly become intractable.
Another issue encountered by topic models is that they do not model stop words well. This is because stop words usually do not carry semantic meaning; their appearance is mainly to make the sentence more readable according to the grammar of the language. They also appear frequently in
almost every document and can co-occur with almost any word2. In practice, these stop words are chosen using tf-idf (Blei and Lafferty, 2009).
3 THE TOPICRNN MODEL
We next describe the proposed TopicRNN model. In TopicRNN, latent topic models are used to capture global semantic dependencies so that the RNN can focus its modeling capacity on the local dynamics of the sequences. With this joint modeling, we hope to achieve better overall performance on downstream applications.
The model. TopicRNN is a generative model. For a document containing the words y1:T ,
1. Draw a topic vector3 θ ∼ N(0, I). 2. Given word y1:t−1, for the tth word yt in the document,
(a) Compute hidden state ht = fW (xt, ht−1), where we let xt , yt−1. (b) Draw stop word indicator lt ∼ Bernoulli(σ(Γ>ht)), with σ the sigmoid function. (c) Draw word yt ∼ p(yt|ht, θ, lt, B), where
p(yt = i|ht, θ, lt, B) ∝ exp ( v>i ht + (1− lt)b>i θ ) .
The stop word indicator lt controls how the topic vector θ affects the output. If lt = 1 (indicating yt is a stop word), the topic vector θ has no contribution to the output. Otherwise, we add a bias to favor those words that are more likely to appear when mixing with θ, as measured by the dot product between θ and the latent word vector bi for the ith vocabulary word. As we can see, the longrange semantic information captured by θ directly affects the output through an additive procedure. Unlike Mikolov and Zweig (2012), the contextual information is not passed to the hidden layer of the RNN. The main reason behind our choice of using the topic vector as bias instead of passing it into the hidden states of the RNN is because it enables us to have a clear separation of the contributions of global semantics and those of local dynamics. The global semantics come from the topics which are meaningful when stop words are excluded. However these stop words are needed for the local dynamics of the language model. We hence achieve this separation of global vs local via a binary decision model for the stop words. It is unclear how to achieve this if we pass the topics to the
2Wallach et al. (2009) described using asymmetric priors to alleviate this issue. Although it is not clear how to use this idea in TopicRNN, we plan to investigate such priors in future work.
3Instead of using the Dirichlet distribution, we choose the Gaussian distribution. This allows for more flexibility in the sequence prediction problem and also has advantages during inference.
hidden states of the RNN. This is because the hidden states of the RNN will account for all words (including stop words) whereas the topics exclude stop words.
We show the unrolled graphical representation of TopicRNN in Figure 1(a). We denote all model parameters as Θ = {Γ, V, B,W,Wc} (see Appendix A.1 for more details). Parameter Wc is for the inference network, which we will introduce below. The observations are the word sequences y1:T and stop word indicators l1:T .4 The log marginal likelihood of the sequence y1:T is
log p(y1:T , l1:T |ht) = log ∫ p(θ) T∏ t=1 p(yt|ht, lt, θ)p(lt|ht)dθ. (2)
Model inference. Direct optimization of Equation 2 is intractable so we use variational inference for approximating this marginal (Jordan et al., 1999). Let q(θ) be the variational distribution on the marginalized variable θ. We construct the variational objective function, also called the evidence lower bound (ELBO), as follows:
L(y1:T , l1:T |q(θ),Θ) , Eq(θ) [ T∑ t=1 log p(yt|ht, lt, θ) + log p(lt|ht) + log p(θ)− log q(θ) ] ≤ log p(y1:T , l1:T |ht,Θ).
Following the proposed variational autoencoder technique, we choose the form of q(θ) to be an inference network using a feed-forward neural network (Kingma and Welling, 2013; Miao et al., 2015). Let Xc ∈ N |Vc|+ be the term-frequency representation of y1:T excluding stop words (with Vc the vocabulary size without the stop words). The variational autoencoder inference network q(θ|Xc,Wc) with parameter Wc is a feed-forward neural network with ReLU activation units that projects Xc into a K-dimensional latent space. Specifically, we have
q(θ|Xc,Wc) = N(θ;µ(Xc), diag(σ2(Xc))), µ(Xc) = W1g(Xc) + a1,
log σ(Xc) = W2g(Xc) + a2,
where g(·) denotes the feed-forward neural network. The weight matrices W1, W2 and biases a1, a2 are shared across documents. Each document has its own µ(Xc) and σ(Xc) resulting in a unique distribution q(θ|Xc) for each document. The output of the inference network is a distribution on θ, which we regard as the summarization of the semantic information, similar to the topic proportions in latent topic models. We show the role of the inference network in Figure 1(b). During training, the parameters of the inference network and the model are jointly learned and updated via truncated backpropagation through time using the Adam algorithm (Kingma and Ba, 2014). We use stochastic samples from q(θ|Xc) and the reparameterization trick towards this end (Kingma and Welling, 2013; Rezende et al., 2014).
Generating sequential text and computing perplexity. Suppose we are given a word sequence y1:t−1, from which we have an initial estimation of q(θ|Xc). To generate the next word yt, we compute the probability distribution of yt given y1:t−1 in an online fashion. We choose θ to be a point estimate θ̂, the mean of its current distribution q(θ|Xc). Marginalizing over the stop word indicator lt which is unknown prior to observing yt, the approximate distribution of yt is
p(yt|y1:t−1) ≈ ∑ lt p(yt|ht, θ̂, lt)p(lt|ht).
The predicted word yt is a sample from this predictive distribution. We update q(θ|Xc) by including yt toXc if yt is not a stop word. However, updating q(θ|Xc) after each word prediction is expensive, so we use a sliding window as was done in Mikolov and Zweig (2012). To compute the perplexity, we use the approximate predictive distribution above.
Model Complexity. TopicRNN has a complexity of O(H × H + H × (C + K) + Wc), where H is the size of the hidden layer of the RNN, C is the vocabulary size, K is the dimension of the topic vector, and Wc is the number of parameters of the inference network. The contextual RNN of Mikolov and Zweig (2012) accounts forO(H×H+H×(C+K)), not including the pre-training process, which might require more parameters than the additional Wc in our complexity.
4Stop words can be determined using one of the several lists available online. For example, http://www. lextek.com/manuals/onix/stopwords2.html
4 EXPERIMENTS
We assess the performance of our proposed TopicRNN model on word prediction and sentiment analysis5. For word prediction we use the Penn TreeBank dataset, a standard benchmark for assessing new language models (Marcus et al., 1993). For sentiment analysis we use the IMDB 100k dataset (Maas et al., 2011), also a common benchmark dataset for this application6. We use RNN, LSTM, and GRU cells in our experiments leading to TopicRNN, TopicLSTM, and TopicGRU.
4.1 WORD PREDICTION
We first tested TopicRNN on the word prediction task using the Penn Treebank (PTB) portion of the Wall Street Journal. We use the standard split, where sections 0-20 (930K tokens) are used for training, sections 21-22 (74K tokens) for validation, and sections 23-24 (82K tokens) for testing (Mikolov et al., 2010). We use a vocabulary of size 10K that includes the special token unk for rare words and eos that indicates the end of a sentence. TopicRNN takes documents as inputs. We split the PTB data into blocks of 10 sentences to constitute documents as done by (Mikolov and Zweig, 2012). The inference network takes as input the bag-of-words representation of the input document. For that reason, the vocabulary size of the inference network is reduced to 9551 after excluding 449 pre-defined stop words.
In order to compare with previous work on contextual RNNs we trained TopicRNN using different network sizes. We performed word prediction using a recurrent neural network with 10 neurons,
5Our code will be made publicly available for reproducibility. 6These datasets are publicly available at http://www.fit.vutbr.cz/~imikolov/rnnlm/
simple-examples.tgz and http://ai.stanford.edu/~amaas/data/sentiment/.
100 neurons and 300 neurons. For these experiments, we used a multilayer perceptron with 2 hidden layers and 200 hidden units per layer for the inference network. The number of topics was tuned depending on the size of the RNN. For 10 neurons we used 18 topics. For 100 and 300 neurons we found 50 topics to be optimal. We used the validation set to tune the hyperparameters of the model. We used a maximum of 15 epochs for the experiments and performed early stopping using the validation set. For comparison purposes we did not apply dropout and used 1 layer for the RNN and its counterparts in all the word prediction experiments as reported in Table 2. One epoch for 10 neurons takes 2.5 minutes. For 100 neurons, one epoch is completed in less than 4 minutes. Finally, for 300 neurons one epoch takes less than 6 minutes. These experiments were ran on Microsoft Azure NC12 that has 12 cores, 2 Tesla K80 GPUs, and 112 GB memory. First, we show five randomly drawn topics in Table 1. These results correspond to a network with 100 neurons. We also illustrate some inferred topic distributions for several documents from TopicGRU in Figure 2. Similar to standard topic models, these distributions are also relatively peaky.
Next, we compare the performance of TopicRNN to our baseline contextual RNN using perplexity. Perplexity can be thought of as a measure of surprise for a language model. It is defined as the exponential of the average negative log likelihood. Table 2 summarizes the results for different network sizes. We learn three things from these tables. First, the perplexity is reduced the larger the network size. Second, RNNs with context features perform better than RNNs without context features. Third, we see that TopicRNN gives lower perplexity than the previous baseline result reported by Mikolov and Zweig (2012). Note that to compute these perplexity scores for word prediction we use a sliding window to compute θ as we move along the sequences. The topic vector θ that is used from the current batch of words is estimated from the previous batch of words. This enables fair comparison to previously reported results (Mikolov and Zweig, 2012).7
Another aspect of the TopicRNN model we studied is its capacity to generate coherent text. To do this, we randomly drew a document from the test set and used this document as seed input to the inference network to compute θ. Our expectation is that the topics contained in this seed document are reflected in the generated text. Table 3 shows generated text from models learned on the PTB and IMDB datasets. See Appendix A.3 for more examples.
7We adjusted the scores in Table 2 from what was previously reported after correcting a bug in the computation of the ELBO.
4.2 SENTIMENT ANALYSIS
We performed sentiment analysis using TopicRNN as a feature extractor on the IMDB 100K dataset. This data consists of 100,000 movie reviews from the Internet Movie Database (IMDB) website. The data is split into 75% for training and 25% for testing. Among the 75K training reviews, 50K are unlabelled and 25K are labelled as carrying either a positive or a negative sentiment. All 25K test reviews are labelled. We trained TopicRNN on 65K random training reviews and used the remaining 10K reviews for validation. To learn a classifier, we passed the 25K labelled training reviews through the learned TopicRNN model. We then concatenated the output of the inference network and the last state of the RNN for each of these 25K reviews to compute the feature vectors. We then used these feature vectors to train a neural network with one hidden layer, 50 hidden units, and a sigmoid activation function to predict sentiment, exactly as done in Le and Mikolov (2014).
To train the TopicRNN model, we used a vocabulary of size 5,000 and mapped all other words to the unk token. We took out 439 stop words to create the input of the inference network. We used 500 units and 2 layers for the inference network, and used 2 layers and 300 units per-layer for the
RNN. We chose a step size of 5 and defined 200 topics. We did not use any regularization such as dropout. We trained the model for 13 epochs and used the validation set to tune the hyperparameters of the model and track perplexity for early stopping. This experiment took close to 78 hours on a MacBook pro quad-core with 16GHz of RAM. See Appendix A.4 for the visualization of some of the topics learned from this data.
Table 4 summarizes sentiment classification results from TopicRNN and other methods. Our error rate is 6.28%.8 This is close to the state-of-the-art 5.91% (Miyato et al., 2016) despite that we do not use the labels and adversarial training in the feature extraction stage. Our approach is most similar to Le and Mikolov (2014), where the features were extracted in a unsupervised way and then a one-layer neural net was trained for classification.
Figure 3 shows the ability of TopicRNN to cluster documents using the feature vectors as created during the sentiment analysis task. Reviews with positive sentiment are coloured in green while reviews carrying negative sentiment are shown in red. This shows that TopicRNN can be used as an unsupervised feature extractor for downstream applications. Table 3 shows generated text from models learned on the PTB and IMDB datasets. See Appendix A.3 for more examples. The overall generated text from IMDB encodes a negative sentiment.
5 DISCUSSION AND FUTURE WORK
In this paper we introduced TopicRNN, a RNN-based language model that combines RNNs and latent topics to capture local (syntactic) and global (semantic) dependencies between words. The global dependencies as captured by the latent topics serve as contextual bias to an RNN-based language model. This contextual information is learned jointly with the RNN parameters by maximizing the evidence lower bound of variational inference. TopicRNN yields competitive per-word perplexity on the Penn Treebank dataset compared to previous contextual RNN models. We have reported a competitive classification error rate for sentiment analysis on the IMDB 100K dataset. We have also illustrated the capacity of TopicRNN to generate sensible topics and text. In future work, we will study the performance of TopicRNN when stop words are dynamically discovered during training. We will also extend TopicRNN to other applications where capturing context is important such as in dialog modeling. If successful, this will allow us to have a model that performs well across different natural language processing applications.
8The experiments were solely based on TopicRNN. Experiments using TopicGRU/TopicLSTM are being carried out and will be added as an extended version of this paper.
A APPENDIX
A.1 DIMENSION OF THE PARAMETERS OF THE MODEL:
We use the following notation: C is the vocabulary size (including stop words), H is the number of hidden units of the RNN, K is the number of topics, and E is the dimension of the inference network hidden layer. Table 5 gives the dimension of each of the parameters of the TopicRNN model (ignoring the biases).
A.2 DOCUMENTS USED TO INFER THE DISTRIBUTIONS ON FIGURE 2
Figure on the left: ’the’, ’market’, ’has’, ’grown’, ’relatively’, ’quiet’, ’since’, ’the’, ’china’, ’crisis’, ’but’, ’if’, ’the’, ’japanese’, ’return’, ’in’, ’force’, ’their’, ’financial’, ’might’, ’could’, ’compensate’, ’to’, ’some’, ’extent’, ’for’, ’local’, ’investors’, "’", ’<unk>’, ’commitment’, ’another’, ’and’, ’critical’, ’factor’, ’is’, ’the’, ’u.s.’, ’hong’, ’kong’, "’s", ’biggest’, ’export’, ’market’, ’even’, ’before’, ’the’, ’china’, ’crisis’, ’weak’, ’u.s.’, ’demand’, ’was’, ’slowing’, ’local’, ’economic’, ’growth’, ’<unk>’, ’strong’, ’consumer’, ’spending’, ’in’, ’the’, ’u.s.’, ’two’, ’years’, ’ago’, ’helped’, ’<unk>’, ’the’, ’local’, ’economy’, ’at’, ’more’, ’than’, ’twice’, ’its’, ’current’, ’rate’, ’indeed’, ’a’, ’few’, ’economists’, ’maintain’, ’that’, ’global’, ’forces’, ’will’, ’continue’, ’to’, ’govern’, ’hong’, ’kong’, "’s", ’economic’, ’<unk>’, ’once’, ’external’, ’conditions’, ’such’, ’as’, ’u.s.’, ’demand’, ’swing’, ’in’, ’the’, ’territory’, "’s", ’favor’, ’they’, ’argue’, ’local’, ’businessmen’, ’will’, ’probably’, ’overcome’, ’their’, ’N’, ’worries’, ’and’, ’continue’, ’doing’, ’business’, ’as’, ’usual’, ’but’, ’economic’, ’arguments’, ’however’, ’solid’, ’wo’, "n’t", ’necessarily’, ’<unk>’, ’hong’, ’kong’, "’s", ’N’, ’million’, ’people’, ’many’, ’are’, ’refugees’, ’having’, ’fled’, ’china’, "’s", ’<unk>’, ’cycles’, ’of’, ’political’, ’repression’, ’and’, ’poverty’, ’since’, ’the’, ’communist’, ’party’, ’took’, ’power’, ’in’, ’N’, ’as’, ’a’, ’result’, ’many’, ’of’, ’those’, ’now’, ’planning’, ’to’, ’leave’, ’hong’, ’kong’, ’ca’, "n’t", ’easily’, ’be’, ’<unk>’, ’by’, ’<unk>’, ’improvements’, ’in’, ’the’, ’colony’, "’s", ’political’, ’and’, ’economic’, ’climate’
Figure on the middle: ’it’, ’said’, ’the’, ’man’, ’whom’, ’it’, ’did’, ’not’, ’name’, ’had’, ’been’, ’found’, ’to’, ’have’, ’the’, ’disease’, ’after’, ’hospital’, ’tests’, ’once’, ’the’, ’disease’, ’was’, ’confirmed’, ’all’, ’the’, ’man’, "’s", ’associates’, ’and’, ’family’, ’were’, ’tested’, ’but’, ’none’, ’have’, ’so’, ’far’, ’been’, ’found’, ’to’, ’have’, ’aids’, ’the’, ’newspaper’, ’said’, ’the’, ’man’, ’had’, ’for’, ’a’, ’long’, ’time’, ’had’, ’a’, ’chaotic’, ’sex’, ’life’, ’including’, ’relations’, ’with’, ’foreign’, ’men’, ’the’, ’newspaper’, ’said’, ’the’, ’polish’, ’government’, ’increased’, ’home’, ’electricity’, ’charges’, ’by’, ’N’, ’N’, ’and’, ’doubled’, ’gas’, ’prices’, ’the’, ’official’, ’news’, ’agency’, ’<unk>’, ’said’, ’the’, ’increases’, ’were’, ’intended’, ’to’, ’bring’, ’<unk>’, ’low’, ’energy’, ’charges’, ’into’, ’line’, ’with’, ’production’, ’costs’, ’and’, ’compensate’, ’for’, ’a’, ’rise’, ’in’, ’coal’, ’prices’, ’in’, ’<unk>’, ’news’, ’south’, ’korea’, ’in’, ’establishing’, ’diplomatic’, ’ties’, ’with’, ’poland’, ’yesterday’, ’announced’, ’$’, ’N’, ’million’, ’in’, ’loans’, ’to’, ’the’, ’financially’, ’strapped’, ’warsaw’, ’government’, ’in’, ’a’, ’victory’, ’for’, ’environmentalists’, ’hungary’, "’s", ’parliament’, ’terminated’, ’a’, ’multibillion-dollar’, ’river’, ’<unk>’, ’dam’, ’being’, ’built’, ’by’, ’<unk>’, ’firms’, ’the’, ’<unk>’, ’dam’, ’was’, ’designed’, ’to’, ’be’, ’<unk>’, ’with’, ’another’, ’dam’, ’now’, ’nearly’, ’complete’, ’N’, ’miles’, ’<unk>’, ’in’, ’czechoslovakia’, ’in’, ’ending’, ’hungary’, "’s", ’part’, ’of’, ’the’, ’project’, ’parliament’, ’authorized’, ’prime’, ’minister’, ’<unk>’, ’<unk>’, ’to’, ’modify’, ’a’, ’N’, ’agreement’, ’with’, ’czechoslovakia’, ’which’, ’still’, ’wants’, ’the’, ’dam’, ’to’, ’be’, ’built’, ’mr.’, ’<unk>’, ’said’, ’in’, ’parliament’, ’that’, ’czechoslovakia’, ’and’, ’hungary’, ’would’, ’suffer’, ’environmental’, ’damage’, ’if’, ’the’, ’<unk>’, ’<unk>’, ’were’, ’built’, ’as’, ’planned’
Figure on the right: ’in’, ’hartford’, ’conn.’, ’the’, ’charter’, ’oak’, ’bridge’, ’will’, ’soon’, ’be’, ’replaced’, ’the’, ’<unk>’, ’<unk>’, ’from’, ’its’, ’<unk>’, ’<unk>’, ’to’, ’a’, ’park’, ’<unk>’, ’are’, ’possible’, ’citizens’, ’in’, ’peninsula’, ’ohio’, ’upset’, ’over’, ’changes’, ’to’, ’a’, ’bridge’, ’negotiated’, ’a’, ’deal’, ’the’, ’bottom’, ’half’, ’of’, ’the’, ’<unk>’, ’will’, ’be’, ’type’, ’f’, ’while’, ’the’, ’top’, ’half’, ’will’, ’have’, ’the’, ’old’, ’bridge’, "’s", ’<unk>’, ’pattern’, ’similarly’, ’highway’, ’engineers’, ’agreed’, ’to’, ’keep’, ’the’, ’old’, ’<unk>’, ’on’, ’the’, ’key’, ’bridge’, ’in’, ’washington’, ’d.c.’, ’as’, ’long’, ’as’, ’they’, ’could’, ’install’, ’a’, ’crash’, ’barrier’, ’between’, ’the’, ’sidewalk’, ’and’, ’the’, ’road’, ’<unk>’, ’<unk>’, ’drink’, ’carrier’, ’competes’, ’with’, ’<unk>’, ’<unk>’, ’<unk>’, ’just’, ’got’, ’easier’, ’or’, ’so’, ’claims’, ’<unk>’, ’corp.’, ’the’, ’maker’, ’of’, ’the’, ’<unk>’, ’the’, ’chicago’, ’company’, "’s", ’beverage’, ’carrier’, ’meant’, ’to’, ’replace’, ’<unk>’, ’<unk>’, ’at’, ’<unk>’, ’stands’, ’and’, ’fast-food’, ’outlets’, ’resembles’, ’the’, ’plastic’, ’<unk>’, ’used’, ’on’, ’<unk>’, ’of’, ’beer’, ’only’, ’the’, ’<unk>’, ’hang’, ’from’, ’a’, ’<unk>’, ’of’, ’<unk>’, ’the’, ’new’, ’carrier’, ’can’, ’<unk>’, ’as’, ’many’, ’as’, ’four’, ’<unk>’, ’at’, ’once’, ’inventor’, ’<unk>’, ’marvin’, ’says’, ’his’, ’design’, ’virtually’, ’<unk>’, ’<unk>’
A.3 MORE GENERATED TEXT FROM THE MODEL:
We illustrate below some generated text resulting from training TopicRNN on the PTB dataset. Here we used 50 neurons and 100 topics:
Text1: but the refcorp bond fund might have been unk and unk of the point rate eos house in national unk wall restraint in the property pension fund sold willing to zenith was guaranteed by $ N million at short-term rates maturities around unk products eos deposit posted yields slightly
Text2: it had happened by the treasury ’s clinical fund month were under national disappear institutions but secretary nicholas instruments succeed eos and investors age far compound average new york stock exchange bonds typically sold $ N shares in the N but paying yields further an average rate of long-term funds
We illustrate below some generated text resulting from training TopicRNN on the IMDB dataset. The settings are the same as for the sentiment analysis experiment:
the film ’s greatest unk unk and it will likely very nice movies to go to unk why various david proves eos the story were always well scary friend high can be a very strange unk unk is in love with it lacks even perfect for unk for some of the worst movies come on a unk gave a rock unk eos whatever let ’s possible eos that kyle can ’t different reasons about the unk and was not what you ’re not a fan of unk unk us rock which unk still in unk ’s music unk one as
A.4 TOPICS FROM IMDB:
Below we show some topics resulting from the sentiment analysis on the IMDB dataset. The total number of topics is 200. Note here all the topics turn around movies which is expected since all reviews are about movies.

This work combines a LDA-type topic model with a RNN and models this by having an additive effect on the predictive distribution via the topic parameters. A variational auto-encoder is used to infer the topic distribution for a given piece of text and the RNN is trained as a RNNLM. The last hidden state of the RNNLM and the topic parameters are then concatenated to use as a feature representation.
The paper is well written and easy to understand. Using the topic as an additive effect on the vocabulary allows for easy inference but intuitively I would expect the topic to affect the dynamics too, e.g. the state of the RNN. The results on using this model as a feature extractor for IMDB are quite strong. Is the RNN fine-tuned on the labelled IMDB data? However, the results for PTB are weaker. From the original paper, an ensemble of 2 LSTMs is able to match the topicRNN score. This method of jointly modelling topics and a language model seems effective and relatively easy to implement.
Finally, the IMDB result is no longer state of the art since this result appeared in May (Miyato et al., Adversarial Training Methods for Semi-Supervised Text Classification).
Some questions:
How important is the stop word modelling? What do the results look like if l_t = 0.5 for all t?
It seems surprising that the RNN was more effective than the LSTM. Was gradient clipping tried in the topicLSTM case? Do GRUs also fail to work?
It is also unfortunate that the model requires a stop-word list. Is the link in footnote 4 the one that is used in the experiments?
Does factoring out the topics in this way allow the RNN to scale better with more neurons? How reasonable does the topic distribution look for individual documents? How peaked do they tend to be? Can you show some examples of the inferred distribution? The topics look odd for IMDB with the top word of two of the topics being the same: 'campbell'. It would be interesting to compare these topics with those inferred by LDA on the same datasets.
Minor comments:
Below figure 2: GHz -> GB
\Gamma is not defined.


UNROLLED GENERATIVE ADVERSARIAL NETWORKS
We introduce a method to stabilize Generative Adversarial Networks (GANs) by defining the generator objective with respect to an unrolled optimization of the discriminator. This allows training to be adjusted between using the optimal discriminator in the generator’s objective, which is ideal but infeasible in practice, and using the current value of the discriminator, which is often unstable and leads to poor solutions. We show how this technique solves the common problem of mode collapse, stabilizes training of GANs with complex recurrent generators, and increases diversity and coverage of the data distribution by the generator.
1 INTRODUCTION
The use of deep neural networks as generative models for complex data has made great advances in recent years. This success has been achieved through a surprising diversity of training losses and model architectures, including denoising autoencoders (Vincent et al., 2010), variational autoencoders (Kingma & Welling, 2013; Rezende et al., 2014; Gregor et al., 2015; Kulkarni et al., 2015; Burda et al., 2015; Kingma et al., 2016), generative stochastic networks (Alain et al., 2015), diffusion probabilistic models (Sohl-Dickstein et al., 2015), autoregressive models (Theis & Bethge, 2015; van den Oord et al., 2016a;b), real non-volume preserving transformations (Dinh et al., 2014; 2016), Helmholtz machines (Dayan et al., 1995; Bornschein et al., 2015), and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014).
1.1 GENERATIVE ADVERSARIAL NETWORKS
While most deep generative models are trained by maximizing log likelihood or a lower bound on log likelihood, GANs take a radically different approach that does not require inference or explicit calculation of the data likelihood. Instead, two models are used to solve a minimax game: a generator which samples data, and a discriminator which classifies the data as real or generated. In theory these models are capable of modeling an arbitrarily complex probability distribution. When using the optimal discriminator for a given class of generators, the original GAN proposed by Goodfellow et al. minimizes the Jensen-Shannon divergence between the data distribution and the generator, and extensions generalize this to a wider class of divergences (Nowozin et al., 2016; Sonderby et al., 2016; Poole et al., 2016).
The ability to train extremely flexible generating functions, without explicitly computing likelihoods or performing inference, and while targeting more mode-seeking divergences as made GANs extremely successful in image generation (Odena et al., 2016; Salimans et al., 2016; Radford et al., 2015), and image super resolution (Ledig et al., 2016). The flexibility of the GAN framework has also enabled a number of successful extensions of the technique, for instance for structured prediction (Reed et al., 2016a;b; Odena et al., 2016), training energy based models (Zhao et al., 2016), and combining the GAN loss with a mutual information loss (Chen et al., 2016). ∗Work done as a member of the Google Brain Residency program (g.co/brainresidency) †Work completed as part of a Google Brain internship
In practice, however, GANs suffer from many issues, particularly during training. One common failure mode involves the generator collapsing to produce only a single sample or a small family of very similar samples. Another involves the generator and discriminator oscillating during training, rather than converging to a fixed point. In addition, if one agent becomes much more powerful than the other, the learning signal to the other agent becomes useless, and the system does not learn. To train GANs many tricks must be employed, such as careful selection of architectures (Radford et al., 2015), minibatch discrimination (Salimans et al., 2016), and noise injection (Salimans et al., 2016; Sonderby et al., 2016). Even with these tricks the set of hyperparameters for which training is successful is generally very small in practice.
Once converged, the generative models produced by the GAN training procedure normally do not cover the whole distribution (Dumoulin et al., 2016; Che et al., 2016), even when targeting a modecovering divergence such as KL. Additionally, because it is intractable to compute the GAN training loss, and because approximate measures of performance such as Parzen window estimates suffer from major flaws (Theis et al., 2016), evaluation of GAN performance is challenging. Currently, human judgement of sample quality is one of the leading metrics for evaluating GANs. In practice this metric does not take into account mode dropping if the number of modes is greater than the number of samples one is visualizing. In fact, the mode dropping problem generally helps visual sample quality as the model can choose to focus on only the most common modes. These common modes correspond, by definition, to more typical samples. Additionally, the generative model is able to allocate more expressive power to the modes it does cover than it would if it attempted to cover all modes.
1.2 DIFFERENTIATING THROUGH OPTIMIZATION
Many optimization schemes, including SGD, RMSProp (Tieleman & Hinton, 2012), and Adam (Kingma & Ba, 2014), consist of a sequence of differentiable updates to parameters. Gradients can be backpropagated through unrolled optimization updates in a similar fashion to backpropagation through a recurrent neural network. The parameters output by the optimizer can thus be included, in a differentiable way, in another objective (Maclaurin et al., 2015). This idea was first suggested for minimax problems in (Pearlmutter & Siskind, 2008), while (Zhang & Lesser, 2010) provided a theoretical analysis and experimental results on differentiating through a single step of gradient ascent for simple matrix games. Differentiating through unrolled optimization was first scaled to deep networks in (Maclaurin et al., 2015), where it was used for hyperparameter optimization. More recently, (Belanger & McCallum, 2015; Han et al., 2016; Andrychowicz et al., 2016) backpropagate through optimization procedures in contexts unrelated to GANs or minimax games.
In this work we address the challenges of unstable optimization and mode collapse in GANs by unrolling optimization of the discriminator objective during training.
2 METHOD
2.1 GENERATIVE ADVERSARIAL NETWORKS
The GAN learning problem is to find the optimal parameters θ∗G for a generator function G (z; θG) in a minimax objective,
θ∗G = argmin θG max θD f (θG, θD) (1)
= argmin θG
f (θG, θ ∗ D (θG)) (2)
θ∗D (θG) = argmax θD f (θG, θD) , (3)
where f is commonly chosen to be
f (θG, θD) = Ex∼pdata [log (D (x; θD))] + Ez∼N (0,I) [log (1−D (G (z; θG) ; θD))] . (4) Here x ∈ X is the data variable, z ∈ Z is the latent variable, pdata is the data distribution, the discriminator D (·; θD) : X → [0, 1] outputs the estimated probability that a sample x comes from the data distribution, θD and θG are the discriminator and generator parameters, and the generator function G (·; θG) : Z → X transforms a sample in the latent space into a sample in the data space.
For the minimax loss in Eq. 4, the optimal discriminator D∗ (x) is a known smooth function of the generator probability pG (x) (Goodfellow et al., 2014),
D∗ (x) = pdata (x)
pdata (x) + pG (x) . (5)
When the generator loss in Eq. 2 is rewritten directly in terms of pG (x) and Eq. 5 rather than θG and θ∗D (θG), then it is similarly a smooth function of pG (x). These smoothness guarantees are typically lost when D (x; θD) and G (z; θG) are drawn from parametric families. They nonetheless suggest that the true generator objective in Eq. 2 will often be well behaved, and is a desirable target for direct optimization.
Explicitly solving for the optimal discriminator parameters θ∗D (θG) for every update step of the generator G is computationally infeasible for discriminators based on neural networks. Therefore this minimax optimization problem is typically solved by alternating gradient descent on θG and ascent on θD.
The optimal solution θ∗ = {θ∗G, θ∗D} is a fixed point of these iterative learning dynamics. Additionally, if f (θG, θD) is convex in θG and concave in θD, then alternating gradient descent (ascent) trust region updates are guaranteed to converge to the fixed point, under certain additional weak assumptions (Juditsky et al., 2011). However in practice f (θG, θD) is typically very far from convex in θG and concave in θD, and updates are not constrained in an appropriate way. As a result GAN training suffers from mode collapse, undamped oscillations, and other problems detailed in Section 1.1. In order to address these difficulties, we will introduce a surrogate objective function fK (θG, θD) for training the generator which more closely resembles the true generator objective f (θG, θ∗D (θG)).
2.2 UNROLLING GANS
A local optimum of the discriminator parameters θ∗D can be expressed as the fixed point of an iterative optimization procedure,
θ0D = θD (6)
θk+1D = θ k D + η
k df ( θG, θ k D ) dθkD
(7)
θ∗D (θG) = lim k→∞ θkD, (8)
where ηk is the learning rate schedule. For clarity, we have expressed Eq. 7 as a full batch steepest gradient ascent equation. More sophisticated optimizers can be similarly unrolled. In our experiments we unroll Adam (Kingma & Ba, 2014).
By unrolling for K steps, we create a surrogate objective for the update of the generator, fK (θG, θD) = f ( θG, θ K D (θG, θD) ) . (9)
When K = 0 this objective corresponds exactly to the standard GAN objective, while as K → ∞ it corresponds to the true generator objective function f (θG, θ∗D (G)). By adjusting the number of unrolling steps K, we are thus able to interpolate between standard GAN training dynamics with their associated pathologies, and more costly gradient descent on the true generator loss.
2.3 PARAMETER UPDATES
The generator and discriminator parameter updates using this surrogate loss are
θG ← θG − η dfK (θG, θD)
dθG (10)
θD ← θD + η df (θG, θD)
dθD . (11)
For clarity we use full batch steepest gradient descent (ascent) with stepsize η above, while in experiments we instead use minibatch Adam for both updates. The gradient in Eq. 10 requires backpropagating through the optimization process in Eq. 7. A clear description of differentiation through
gradient descent is given as Algorithm 2 in (Maclaurin et al., 2015), though in practice the use of an automatic differentiation package means this step does not need to be programmed explicitly. A pictorial representation of these updates is provided in Figure 1.
It is important to distinguish this from an approach suggested in (Goodfellow et al., 2014), that several update steps of the discriminator parameters should be run before each single update step for the generator. In that approach, the update steps for both models are still gradient descent (ascent) with respect to fixed values of the other model parameters, rather than the surrogate loss we describe in Eq. 9. Performing K steps of discriminator update between each single step of generator update corresponds to updating the generator parameters θG using only the first term in Eq. 12 below.
2.4 THE MISSING GRADIENT TERM
To better understand the behavior of the surrogate loss fK (θG, θD), we examine its gradient with respect to the generator parameters θG,
dfK (θG, θD) dθG = ∂f
( θG, θ K D (θG, θD) ) ∂θG + ∂f ( θG, θ K D (θG, θD) ) ∂θKD (θG, θD) dθKD (θG, θD) dθG . (12)
Standard GAN training corresponds exactly to updating the generator parameters using only the first term in this gradient, with θKD (θG, θD) being the parameters resulting from the discriminator update step. An optimal generator for any fixed discriminator is a delta function at the x to which the discriminator assigns highest data probability. Therefore, in standard GAN training, each generator update step is a partial collapse towards a delta function.
The second term captures how the discriminator would react to a change in the generator. It reduces the tendency of the generator to engage in mode collapse. For instance, the second term reflects that as the generator collapses towards a delta function, the discriminator reacts and assigns lower probability to that state, increasing the generator loss. It therefore discourages the generator from collapsing, and may improve stability.
As K →∞, θKD goes to a local optimum of f , where ∂f ∂θKD = 0, and therefore the second term in Eq. 12 goes to 0 (Danskin, 1967). The gradient of the unrolled surrogate loss fK (θG, θD) with respect to θG is thus identical to the gradient of the standard GAN loss f (θG, θD) both whenK = 0 and when K → ∞, where we take K → ∞ to imply that in the standard GAN the discriminator is also fully optimized between each generator update. Between these two extremes, fK (θG, θD) captures additional information about the response of the discriminator to changes in the generator.
2.5 CONSEQUENCES OF THE SURROGATE LOSS
GANs can be thought of as a game between the discriminator (D) and the generator (G). The agents take turns taking actions and updating their parameters until a Nash equilibrium is reached. The optimal action for D is to evaluate the probability ratio pdata(x)pG(x)+pdata(x) for the generator’s move x (Eq. 5). The optimal generator action is to move its mass to maximize this ratio.
The initial move forGwill be to move as much mass as its parametric family and update step permits to the single point that maximizes the ratio of probability densities. The action D will then take is quite simple. It will track that point, and to the extent allowed by its own parametric family and update step assign low data probability to it, and uniform probability everywhere else. This cycle of G moving and D following will repeat forever or converge depending on the rate of change of the two agents. This is similar to the situation in simple matrix games like rock-paper-scissors and matching pennies, where alternating gradient descent (ascent) with a fixed learning rate is known not to converge (Singh et al., 2000; Bowling & Veloso, 2002).
In the unrolled case, however, this undesirable behavior no longer occurs. NowG’s actions take into account how D will respond. In particular, G will try to make steps that D will have a hard time responding to. This extra information helps the generator spread its mass to make the next D step less effective instead of collapsing to a point.
In principle, a surrogate loss function could be used for both D and G. In the case of 1-step unrolled optimization this is known to lead to convergence for games in which gradient descent (ascent) fails (Zhang & Lesser, 2010). However, the motivation for using the surrogate generator loss in Section 2.2, of unrolling the inner of two nested min and max functions, does not apply to using a surrogate discriminator loss. Additionally, it is more common for the discriminator to overpower the generator than vice-versa when training a GAN. Giving more information to G by allowing it to ‘see into the future’ may thus help the two models be more balanced.
3 EXPERIMENTS
In this section we demonstrate improved mode coverage and stability by applying this technique to five datasets of increasing complexity. Evaluation of generative models is a notoriously hard problem (Theis et al., 2016). As such the de facto standard in GAN literature has become sample quality as evaluated by a human and/or evaluated by a heuristic (Inception score for example, (Salimans et al., 2016)). While these evaluation metrics do a reasonable job capturing sample quality, they fail to capture sample diversity. In our first 2 experiments diversity is easily evaluated via visual inspection. In our later experiments this is not the case, and we will use a variety of methods to quantify coverage of samples. Our measures are individually strongly suggestive of unrolling reducing mode-collapse and improving stability, but none of them alone are conclusive. We believe that taken together however, they provide extremely compelling evidence for the advantages of unrolling.
When doing stochastic optimization, we must choose which minibatches to use in the unrolling updates in Eq. 7. We experimented with both a fixed minibatch and re-sampled minibatches for each unrolling step, and found it did not significantly impact the result. We use fixed minibatches for all experiments in this section.
We provide a reference implementation of this technique at github.com/poolio/unrolled gan.
3.1 MIXTURE OF GAUSSIANS DATASET
To illustrate the impact of discriminator unrolling, we train a simple GAN architecture on a 2D mixture of 8 Gaussians arranged in a circle. For a detailed list of architecture and hyperparameters see Appendix A. Figure 2 shows the dynamics of this model through time. Without unrolling the generator rotates around the valid modes of the data distribution but is never able to spread out mass. When adding in unrolling steps G quickly learns to spread probability mass and the system converges to the data distribution.
In Appendix B we perform further experiments on this toy dataset. We explore how unrolling compares to historical averaging, and compares to using the unrolled discriminator to update the
generator, but without backpropagating through the generator. In both cases we find that the unrolled objective performs better.
3.2 PATHOLOGICAL MODEL WITH MISMATCHED GENERATOR AND DISCRIMINATOR
To evaluate the ability of this approach to improve trainability, we look to a traditionally challenging family of models to train – recurrent neural networks (RNNs). In this experiment we try to generate MNIST samples using an LSTM (Hochreiter & Schmidhuber, 1997). MNIST digits are 28x28 pixel images. At each timestep of the generator LSTM, it outputs one column of this image, so that after 28 timesteps it has output the entire sample. We use a convolutional neural network as the discriminator. See Appendix C for the full model and training details. Unlike in all previously successful GAN models, there is no symmetry between the generator and the discriminator in this task, resulting in a more complex power balance. Results can be seen in Figure 3. Once again, without unrolling the model quickly collapses, and rotates through a sequence of single modes. Instead of rotating spatially, it cycles through proto-digit like blobs. When running with unrolling steps the generator disperses and appears to cover the whole data distribution, as in the 2D example.
3.3 MODE AND MANIFOLD COLLAPSE USING AUGMENTED MNIST
GANs suffer from two different types of model collapse – collapse to a subset of data modes, and collapse to a sub-manifold within the data distribution. In these experiments we isolate both effects using artificially constructed datasets, and demonstrate that unrolling can largely rescue both types of collapse.
3.3.1 DISCRETE MODE COLLAPSE
To explore the degree to which GANs drop discrete modes in a dataset, we use a technique similar to one from (Che et al., 2016). We construct a dataset by stacking three randomly chosen MNIST digits, so as to construct an RGB image with a different MNIST digit in each color channel. This new dataset has 1,000 distinct modes, corresponding to each combination of the ten MNIST classes in the three channels.
We train a GAN on this dataset, and generate samples from the trained model (25,600 samples for all experiments). We then compute the predicted class label of each color channel using a pre-trained MNIST classifier. To evaluate performance, we use two metrics: the number of modes for which the generator produced at least one sample, and the KL divergence between the model and the expected data distribution. Within this discrete label space, a KL divergence can be estimated tractably between the generated samples and the data distribution over classes, where the data distribution is a uniform distribution over all 1,000 classes.
As presented in Table 1, as the number of unrolling steps is increased, both mode coverage and reverse KL divergence improve. Contrary to (Che et al., 2016), we found that reasonably sized models (such as the one used in Section 3.4) covered all 1,000 modes even without unrolling. As such we use smaller convolutional GAN models. Details on the models used are provided in Appendix E.
We observe an additional interesting effect in this experiment. The benefits of unrolling increase as the discriminator size is reduced. We believe unrolling effectively increases the capacity of the discriminator. The unrolled discriminator can better react to any specific way in which the generator is producing non-data-like samples. When the discriminator is weak, the positive impact of unrolling is thus larger.
3.3.2 MANIFOLD COLLAPSE
In addition to discrete modes, we examine the effect of unrolling when modeling continuous manifolds. To get at this quantity, we constructed a dataset consisting of colored MNIST digits. Unlike in the previous experiment, a single MNIST digit was chosen, and then assigned a single monochromatic color. With a perfect generator, one should be able to recover the distribution of colors used to generate the digits. We use colored MNIST digits so that the generator also has to model the digits, which makes the task sufficiently complex that the generator is unable to perfectly solve it. The color of each digit is sampled from a 3D normal distribution. Details of this dataset are provided in Appendix F. We will examine the distribution of colors in the samples generated by the trained GAN. As will also be true in the CIFAR10 example in Section 3.4, the lack of diversity in generated colors is almost invisible using only visual inspection of the samples. Samples can be found in Appendix F.
In order to recover the color the GAN assigned to the digit, we used k-means with 2 clusters, to pick out the foreground color from the background. We then performed this transformation for both the training data and the generated images. Next we fit a Gaussian kernel density estimator to both distributions over digit colors. Finally, we computed the JS divergence between the model and data distributions over colors. Results can be found in Table 2 for several model sizes. Details of the models are provided in Appendix F.
In general, the best performing models are unrolled for 5-10 steps, and larger models perform better than smaller models. Counter-intuitively, taking 1 unrolling step seems to hurt this measure of diversity. We suspect that this is due to it introducing oscillatory dynamics into training. Taking more unrolling steps however leads to improved performance with unrolling.
3.4 IMAGE MODELING OF CIFAR10
Here we test our technique on a more traditional convolutional GAN architecture and task, similar to those used in (Radford et al., 2015; Salimans et al., 2016). In the previous experiments we tested models where the standard GAN training algorithm would not converge. In this section we improve a standard model by reducing its tendency to engage in mode collapse. We ran 4 configurations of this model, varying the number of unrolling steps to be 0, 1, 5, or 10. Each configuration was run 5 times with different random seeds. For full training details see Appendix D. Samples from each of the 4 configurations can be found in Figure 4. There is no obvious difference in visual quality across these model configurations. Visual inspection however provides only a poor measure of sample diversity.
By training with an unrolled discriminator, we expect to generate more diverse samples which more closely resemble the underlying data distribution. We introduce two techniques to examine sample diversity: inference via optimization, and pairwise distance distributions.
3.4.1 INFERENCE VIA OPTIMIZATION
Since likelihood cannot be tractably computed, over-fitting of GANs is typically tested by taking samples and computing the nearest-neighbor images in pixel space from the training data (Goodfellow et al., 2014). We will do the reverse, and measure the ability of the generative model to generate images that look like specific samples from the training data. If we did this by generating random samples from the model, we would need an exponentially large number of samples. We instead treat finding the nearest neighbor xnearest to a target image xtarget as an optimization task,
znearest = argmin z ||G (z; θG)− xtarget||22 (13)
xnearest = G (znearest; θG) . (14)
This concept of backpropagating to generate images has been widely used in visualizing features from discriminative networks (Simonyan et al., 2013; Yosinski et al., 2015; Nguyen et al., 2016) and has been applied to explore the visual manifold of GANs in (Zhu et al., 2016).
We apply this technique to each of the models trained. We optimize with 3 random starts using LBFGS, which is the optimizer typically used in similar settings such as style transfer (Johnson et al., 2016; Champandard, 2016). Results comparing average mean squared errors between xnearest and xtarget in pixel space can be found in Table 3. In addition we compute the percent of images for which a certain configuration achieves the lowest loss when compared to the other configurations.
In the zero step case, there is poor reconstruction and less than 1% of the time does it obtain the lowest error of the 4 configurations. Taking 1 unrolling step results in a significant improvement in MSE. Taking 10 unrolling steps results in more modest improvement, but continues to reduce the reconstruction MSE.
To visually see this, we compare the result of the optimization process for 0, 1, 5, and 10 step configurations in Figure 5. To select for images where differences in behavior is most apparent, we sort the data by the absolute value of a fractional difference in MSE between the 0 and 10 step models, ∣∣∣ l0step−l10step1 2 (l0step+l10step)
∣∣∣. This highlights examples where either the 0 or 10 step model cannot accurately fit the data example but the other can. In Appendix G we show the same comparison for models initialized using different random seeds. Many of the zero step images are fuzzy and illdefined suggesting that these images cannot be generated by the standard GAN generative model, and come from a dropped mode. As more unrolling steps are added, the outlines become more clear and well defined – the model covers more of the distribution and thus can recreate these samples.
3.4.2 PAIRWISE DISTANCES
A second complementary approach is to compare statistics of data samples to the corresponding statistics for samples generated by the various models. One particularly simple and relevant statistic is the distribution over pairwise distances between random pairs of samples. In the case of mode collapse, greater probability mass will be concentrated in smaller volumes, and the distribution over inter-sample distances should be skewed towards smaller distances. We sample random pairs of images from each model, as well as from the training data, and compute histograms of the `2 distances between those sample pairs. As illustrated in Figure 6, the standard GAN, with zero unrolling steps, has its probability mass skewed towards smaller `2 intersample distances, compared
to real data. As the number of unrolling steps is increased, the histograms over intersample distances increasingly come to resemble that for the data distribution. This is further evidence in support of unrolling decreasing the mode collapse behavior of GANs.
4 DISCUSSION
In this work we developed a method to stabilize GAN training and reduce mode collapse by defining the generator objective with respect to unrolled optimization of the discriminator. We then demonstrated the application of this method to several tasks, where it either rescued unstable training, or reduced the tendency of the model to drop regions of the data distribution.
The main drawback to this method is computational cost of each training step, which increases linearly with the number of unrolling steps. There is a tradeoff between better approximating the true generator loss and the computation required to make this estimate. Depending on the architecture, one unrolling step can be enough. In other more unstable models, such as the RNN case, more are needed to stabilize training. We have some initial positive results suggesting it may be sufficient to further perturb the training gradient in the same direction that a single unrolling step perturbs it. While this is more computationally efficient, further investigation is required.
The method presented here bridges some of the gap between theoretical and practical results for training of GANs. We believe developing better update rules for the generator and discriminator is an important line of work for GAN training. In this work we have only considered a small fraction of the design space. For instance, the approach could be extended to unroll G when updating D as well – letting the discriminator react to how the generator would move. It is also possible to unroll sequences of G and D updates. This would make updates that are recursive: G could react to maximize performance as if G and D had already updated.
ACKNOWLEDGMENTS
We would like to thank Laurent Dinh, David Dohan, Vincent Dumoulin, Liam Fedus, Ishaan Gulrajani, Julian Ibarz, Eric Jang, Matthew Johnson, Marc Lanctot, Augustus Odena, Gabriel Pereyra,
Colin Raffel, Sam Schoenholz, Ayush Sekhari, Jon Shlens, and Dale Schuurmans for insightful conversation, as well as the rest of the Google Brain Team.
B MORE MIXTURE OF GAUSSIAN EXPERIMENTS
B.1 EFFECTS OF TIME DELAY / HISTORICAL AVERAGING
Another comparison we looked at was with regard to historical averaging based approaches. Recently similarly inspired approaches have been used in (Salimans et al., 2016) to stabilize training. For our study, we looked at taking an ensemble of discriminators over time.
First, we looked at taking an ensemble of the last N steps, as shown in Figure App.1.
1
5
20
0
50
5000 10000 15000 20000 25000 30000 35000 40000 45000 50000
Update Steps
N u m
b e r
E n se
m b le
s
Figure App.1: Historical averaging does not visibly increase stability on the mixture of Gaussians task. Each row corresponds to an ensemble of discriminators which consists of the indicated number of immediately preceding discriminators. The columns correspond to different numbers of training steps.
To further explore this idea, we ran experiments with an ensemble of 5 discriminators, but with different periods between replacing discriminators in the ensemble. For example, if I sample at a rate of 100, it would take 500 steps to replace all 5 discriminators. Results can be seen in Figure App.2.
We observe that given longer and longer time delays, the model becomes less and less stable. We hypothesize that this is due to the initial shape of the discriminator loss surface. When training, the discriminator’s estimates of probability densities are only accurate on regions where it was trained. When fixing this discriminator, we are removing the feedback between the generator exploitation
1
10
100
0
1000
5000 10000 15000 20000 25000 30000 35000 40000 45000 50000
Update Steps
S te
p s
B e tw
e e n E
n se
m b le
s
Figure App.2: Introducing longer time delays between the discriminator ensemble results in instability and probability distributions that are not in the window being visualized. The x axis is the number of weight updates and the y axis is how many steps to skip between discriminator updates when selecting the ensemble of 5 discriminators.
and the discriminators ability to move. As a result, the generator is able to exploit these fixed areas of poor performance for older discriminators in the ensemble. New discriminators (over)compensate for this, leading the system to diverge.
B.2 EFFECTS OF THE SECOND GRADIENT
A second factor we analyzed is the effect of backpropagating the learning signal through the unrolling in Equation 12. We can turn on or off this backpropagation through the unrolling by introducing stop gradient calls into our computation graph between each unrolling step. With the stop gradient in place, the update signal corresponds only to the first term in Equation 12. We looked at 3 configurations: without stop gradients; vanilla unrolled GAN, with stop gradients; and with stop gradients but taking the average over the k unrolling steps instead of taking the final value. Results can be see in Figure App.3.
We initially observed no difference between unrolling with and without the second gradient, as both required 3 unrolling steps to become stable. When the discriminator is unrolled to convergence, the second gradient term becomes zero. Due to the simplicity of the problem, we suspect that the discriminator nearly converged for every generator step, and the second gradient term was thus irrelevant.
To test this, we modified the dynamics to perform five generator steps for each discriminator update. Results are shown in Figure App.4. With the discriminator now kept out of equilibrium, successful training can be achieved with half as many unrolling steps when using both terms in the gradient than when only including the first term.
C RNN MNIST TRAINING DETAILS
The network architecture for the experiment in Section 3.2 is as follows:
The MNIST dataset is scaled to [-1, 1).
The generator first scales the 256D noise vector through a 256 unit fully connected layer with relu activation. This is then fed into the initial state of a 256D LSTM(Hochreiter & Schmidhuber, 1997) that runs 28 steps corresponding to the number of columns in MNIST. The resulting sequence of activations is projected through a fully connected layer with 28 outputs with a tanh activation function. All weights are initialized via the ”Xavier” initialization (Glorot & Bengio, 2010). The forget bias on the LSTM is initialized to 1.
The discriminator network feeds the input into a Convolution(16, stride=2) followed by a Convolution(32, stride=2) followed by Convolution(32, stride=2). All convolutions have stride 2. As in (Radford et al., 2015) leaky rectifiers are used with a 0.3 leak. Batch normalization is applied after each layer (Ioffe & Szegedy, 2015). The resulting 4D tensor is then flattened and a linear projection is performed to a single scalar.
Unrolled GAN
0
1
3
5
10
0
20
5000 10000 15000 20000 25000 30000 35000 40000 45000 50000 Update Steps
U n ro
lli n g S
te p s
Unrolled GAN without second gradient
0
1
3
5
10
0
20
5000 10000 15000 20000 25000 30000 35000 40000 45000 50000 Update Steps
U n ro
lli n g S
te p s
Figure App.3: If the discriminator remains nearly at its optimum during learning, then performance is nearly identical with and without the second gradient term in Equation 12. As shown in Figure App.4, when the discriminator lags behind the generator, backpropagating through unrolling aids convergence.
The generator network minimises LG = log(D(G(z))) and the discriminator minimizes LD = log(D(x)) + log(1 −D(G(z))). Both networks are trained with Adam(Kingma & Ba, 2014) with learning rates of 1e-4 and β1=0.5. The network is trained alternating updating the generator and the discriminator for 150k steps. One step consists of just 1 network update.
D CIFAR10/MNIST TRAINING DETAILS
The network architectures for the discriminator, generator, and encoder as as follows. All convolutions have a kernel size of 3x3 with batch normalization and leaky ReLU’s with a 0.3 leak.
The generator network is defined as:
number outputs stride Input: z ∼ N (0, I256) Fully connected 4 * 4 * 512 Reshape to image 4,4,512 Convolution 256 2 Convolution 128 2 Convolution 64 2 Convolution 1 or 3 1
Unrolled GAN with 5 G Steps per D
0
1
3
5
10
0
20
5000 10000 15000 20000 25000 30000 35000 40000 45000 50000 Update Steps
U n ro
lli n g S
te p s
Unrolled GAN with 5 G Steps per D without second gradient
0
1
3
5
10
0
20
5000 10000 15000 20000 25000 30000 35000 40000 45000 50000 Update Steps
U n ro
lli n g S
te p s
Figure App.4: Backpropagating through the unrolling process aids convergence when the discriminator does not fully converge between generator updates. When taking 5 generator steps per discriminator step unrolling greatly increases stability, requiring only 5 unrolling steps to converge. Without the second gradient it requires 10 unrolling steps. Also see Figure App.3.
The discriminator network is defined as:
number outputs stride Input: x ∼ pdata or G Transposed Convolution 64 2 Transposed Convolution 128 2 Transposed Convolution 256 2 Flatten Fully Connected 1
The generator network minimises LG = log(D(G(z))) and the discriminator minimizes LD = log(D(x))+ log(1−D(G(z))). The networks are trained with Adam with a generator learning rate of 1e-4, and a discriminator learning rate of 2e-4. The network is trained alternating updating the generator and the discriminator for 100k steps. One step consists of just 1 network update.
E 1000 CLASS MNIST
number outputs stride Input: z ∼ N (0, I256) Fully connected 4 * 4 * 64 Reshape to image 4,4,64 Convolution 32 2 Convolution 16 2 Convolution 8 2 Convolution 3 1
The discriminator network is parametrized by a size X and is defined as follows. In our tests, we used X of 1/4 and 1/2.
number outputs stride Input: x ∼ pdata or G Transposed Convolution 8*X 2 Transposed Convolution 16*X 2 Transposed Convolution 32*X 2 Flatten Fully Connected 1
F COLORED MNIST DATASET
F.1 DATASET
To generate this dataset we first took the mnist digit, I , scaled between 0 and 1. For each image we sample a color, C, normally distributed with mean=0 and std=0.5. To generate a colored digit between (-1, 1) we do I ∗ C + (I − 1). Finally, we add a small amount of pixel independent noise sampled from a normal distribution with std=0.2, and the resulting values are cliped between (-1, 1). When visualized, this generates images and samples that can be seen in figure App.5. Once again it is very hard to visually see differences in sample diversity when comparing the 128 and the 512 sized models.
Figure App.5: Right: samples from the data distribution. Middle: Samples from 1/4 size model with 0 look ahead steps (worst diversity). Left: Samples from 1/1 size model with 10 look ahead steps (most diversity).
F.2 MODELS
The models used in this section are parametrized by a variable X to control capacity. A value of X=1 is same architecture used in the cifar10 experiments. We used 1/4, 1/2 and 1 as these values.
The generator network is defined as:
number outputs stride Input: z ∼ N (0, I256) Fully connected 4 * 4 * 512*X Reshape to image 4,4,512*X Convolution 256*X 2 Convolution 128*X 2 Convolution 64*X 2 Convolution 3 1
The discriminator network is defined as:
number outputs stride Input: x ∼ pdata or G Transposed Convolution 64*X 2 Transposed Convolution 128*X 2 Transposed Convolution 256*X 2 Flatten Fully Connected 1
G OPTIMIZATION BASED VISUALIZATIONS
More examples of model based optimization. We performed 5 runs with different seeds of each of of the unrolling steps configuration. Bellow are comparisons for each run index. Ideally this would be a many to many comparison, but for space efficiency we grouped the runs by the index in which they were run.
Data
0.026
0 step
0.0154
1 step
0.016
5 step
0.0084
10 step
0.0326 0.0146 0.0158 0.0107
0.0323 0.0103 0.0158 0.0108
0.0327 0.0193 0.0184 0.0113
0.0212 0.0136 0.0133 0.0078
0.0158 0.0108 0.0103 0.006
0.0612 0.027 0.0317 0.0239
0.0254 0.0083 0.0128 0.0102
0.0321 0.0182 0.0177 0.0128
0.0382 0.0139 0.0133 0.0154
0.0142 0.0059 0.0067 0.0059
0.0332 0.0119 0.0186 0.0139
0.0061 0.0049 0.0034 0.0026
Data
0.0351
0 step
0.0341
1 step
0.0216
5 step
0.0149
10 step
0.0183 0.0087 0.0073 0.0078
0.0449 0.0251 0.0238 0.0194
0.0185 0.009 0.0094 0.008
0.0672 0.0366 0.0428 0.0295
0.0146 0.0112 0.0146 0.0064
0.0465 0.0302 0.0272 0.0206
0.05 0.0274 0.0238 0.0223
0.012 0.007 0.0085 0.0054
0.0252 0.0167 0.0172 0.0113
0.0268 0.0157 0.0154 0.0121
0.0405 0.0189 0.0235 0.0185
0.048 0.0325 0.0295 0.0222
Figure App.6: Samples from 1/5 with different random seeds.
Data
0.0183
0 step
0.0151
1 step
0.0068
5 step
0.0051
10 step
0.0216 0.0119 0.0058 0.0066
0.0555 0.0217 0.0182 0.017
0.0317 0.0141 0.0144 0.014
0.0058 0.0035 0.0041 0.0026
0.0488 0.033 0.0316 0.0223
0.0165 0.0085 0.0106 0.0082
0.0221 0.0134 0.0217 0.011
0.0231 0.013 0.0168 0.0117
0.0627 0.0393 0.0536 0.032
0.0151 0.0129 0.0286 0.0079
0.0168 0.0101 0.0104 0.0087
0.0276 0.0217 0.0193 0.0146
Data
0.0338
0 step
0.024
1 step
0.0232
5 step
0.0178
10 step
0.0273 0.0168 0.0259 0.0145
0.0355 0.0232 0.0262 0.0189
0.0151 0.0127 0.0137 0.008
0.0213 0.0159 0.0221 0.0114
0.0368 0.0305 0.0255 0.0199
0.005 0.0046 0.0039 0.0027
0.0292 0.0239 0.0211 0.016
0.0263 0.0201 0.0217 0.0144
0.0334 0.0213 0.0207 0.0183
0.0355 0.0339 0.0215 0.0196
0.0438 0.0211 0.0291 0.0242
0.0226 0.015 0.0156 0.0125
Figure App.7: Samples from 2/5 with different random seeds.
Data
0.03
0 step
0.0075
1 step
0.0066
5 step
0.0053
10 step
0.0213 0.0081 0.0096 0.006
0.0453 0.0157 0.0127 0.0143
0.0161 0.0114 0.0075 0.0063
0.0168 0.0125 0.0137 0.0066
0.0304 0.0186 0.0229 0.013
0.0421 0.0284 0.0272 0.0185
0.0234 0.0147 0.0173 0.0103
0.0242 0.0156 0.0144 0.011
0.0223 0.0152 0.0174 0.0103
0.0341 0.0274 0.02 0.016
0.0079 0.0049 0.0052 0.0037
0.0207 0.0133 0.011 0.0099
Data
0.0403
0 step
0.0246
1 step
0.0352
5 step
0.0197
10 step
0.0227 0.0164 0.0109 0.0111
0.0202 0.0148 0.0146 0.01
0.0269 0.02 0.0221 0.0133
0.0172 0.0144 0.0144 0.0086
0.0223 0.0157 0.0196 0.0111
0.0509 0.0357 0.0351 0.0258
0.021 0.016 0.015 0.0107
0.0219 0.0105 0.013 0.0112
0.0206 0.0177 0.0175 0.0105
0.024 0.0177 0.0256 0.0123
0.0286 0.019 0.0188 0.0146
0.0169 0.015 0.0135 0.0087
Figure App.8: Samples from 3/5 with different random seeds.
Data
0.0259
0 step
0.016
1 step
0.0288
5 step
0.0096
10 step
0.0374 0.0388 0.0301 0.0161
0.0429 0.021 0.0228 0.0191
0.0233 0.0278 0.0222 0.011
0.0528 0.0303 0.0274 0.0249
0.0329 0.0202 0.0274 0.0155
0.0286 0.0204 0.0164 0.0135
0.022 0.0146 0.0155 0.0105
0.0366 0.0248 0.0205 0.0178
0.0336 0.0228 0.0257 0.0164
0.0191 0.0115 0.011 0.0096
0.0557 0.0322 0.0304 0.0282
0.0464 0.0261 0.0269 0.0239
Data
0.0216
0 step
0.0155
1 step
0.0218
5 step
0.0112
10 step
0.0403 0.024 0.0244 0.0212
0.055 0.0363 0.0361 0.0289
0.0341 0.0236 0.0333 0.018
0.0414 0.0215 0.0274 0.0219
0.0167 0.0104 0.0122 0.0089
0.0241 0.0105 0.015 0.0129
0.0063 0.0028 0.0046 0.0034
0.0402 0.0278 0.0314 0.0217
0.0157 0.0133 0.0142 0.0084
0.0301 0.0281 0.0294 0.0163
0.0507 0.0362 0.0494 0.0277
0.0375 0.0323 0.0247 0.0206
Figure App.9: Samples from 4/5 with different random seeds.
Data
0.0452
0 step
0.0064
1 step
0.0064
5 step
0.0081
10 step
0.0333 0.0078 0.0076 0.0065
0.043 0.0124 0.0235 0.0134
0.03 0.0064 0.008 0.0104
0.0128 0.0058 0.0065 0.0058
0.0392 0.0195 0.0218 0.0177
0.0402 0.0308 0.0286 0.0184
0.0168 0.0096 0.0119 0.0077
0.0402 0.0299 0.0233 0.0188
0.026 0.0144 0.0165 0.0122
0.0097 0.0061 0.005 0.0046
0.0105 0.0051 0.0042 0.005
0.0331 0.0236 0.0256 0.0158
Data
0.0557
0 step
0.0373
1 step
0.0344
5 step
0.0271
10 step
0.0565 0.031 0.0364 0.0276
0.0315 0.0115 0.0137 0.0154
0.0285 0.0123 0.0183 0.014
0.0552 0.0314 0.0307 0.0271
0.0327 0.015 0.0172 0.0161
0.0735 0.0577 0.0386 0.0365
0.0204 0.0121 0.0111 0.0102
0.0291 0.0163 0.0195 0.0145
0.0261 0.0135 0.015 0.013
0.0286 0.0189 0.02 0.0143
0.027 0.019 0.019 0.0135
0.0156 0.0091 0.012 0.0078
Figure App.10: Samples from 5/5 with different random seeds.

This work introduces a novel method for training GANs by displacing simultaneous SGD, and unrolling the inner optimization in the minmax game as a computational graph. The paper is very clearly written, and explains the justification very well. The problem being attacked is very significant and important. The approach is novel, however, similar ideas have been tried to solve problems unrelated to GANs.
The first quantitative experiment is section 3.3.1, where the authors attempt to find the best z which can generate training examples. This is done by using L-BFGS on |G(z) - x|. The claim is that if we're able to find such a z, then the generator can generate this particular training example. It's demonstrated that 0-step GANs are not able to generate many training examples, while unrolled GANs do. However, I find this experiment unreasonable. Being able to find a certain z, which generates a certain sample does not guarantee that this particular mode is high probability. In fact, an identity function can potentially beat all the GAN models in the proposed metric. And due to Cantor's proof of equivalence between all powers of real spaces, this applies to smaller dimension of z as well. More realistically, it should be possible to generate *any* image from a generator by finding a very specific z. That a certain z exists which can generate a sample does not prove that the generator is not missing modes. It just proves that the generator is similar enough to an identity function to be able to generate any possible image. This metric is thus measuring something potentially tangential to diversity or mode-dropping. Another problem with this metric is that that showing that the optimization is not able to find a z for a specific training examples does not prove that such a z does not exist, only that it's harder to find. So, this comparison might just be showing that unrolled GANs have a smoother function than 0-step GANs, and thus easier to optimize for z.
The second quantitative experiment considers mean pairwise distance between generated samples, and between data samples. The first number is likely to be small in the case of a mode-dropping GAN. The authors argue that the two numbers being closer to each other is an indication of the generated samples being as diverse as the data. Once again, this metric is not convincing. 1. The distances are being measured in pixel-space. 2. A GAN model could be generating garbage, and yet still perform very well in this metric.
There are no other quantitative results in the paper. Even though the method is optimizing diversity, for a sanity check, scores for quality such as Inception scores or SSL performance would have been useful. Another metric that the authors can consider is training GAN using this approach on the tri-MNIST dataset (concatenation of 3 MNIST digits), which results in 1000 easily-identifiable modes. Then, demonstrate that the GAN is able to generate all the 1000 modes with equal probability. This is not a perfect metric either, but arguably much better than the metrics in this paper. This metric is used in this ICLR submission: