NEWSQA: A MACHINE COMPREHENSION DATASET
We present NewsQA, a challenging machine comprehension dataset of over 100,000 question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting in spans of text from the corresponding articles. We collect this dataset through a fourstage process designed to solicit exploratory questions that require reasoning. A thorough analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (0.198 in F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available at datasets.maluuba.com/NewsQA.
We present NewsQA, a challenging machine comprehension dataset of over 100,000 question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting in spans of text from the corresponding articles. We collect this dataset through a fourstage process designed to solicit exploratory questions that require reasoning. A thorough analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (0.198 in F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available at datasets.maluuba.com/NewsQA.
1 INTRODUCTION
Almost all human knowledge is recorded in the language of text. As such, comprehension of written language by machines, at a near-human level, would enable a broad class of artificial intelligence applications. In human students we evaluate reading comprehension by posing questions based on a text passage and then assessing a student’s answers. Such comprehension tests are appealing because they are objectively gradable and may measure a range of important abilities, from basic understanding to causal reasoning to inference (Richardson et al., 2013). To teach literacy to machines, the research community has taken a similar approach with machine comprehension (MC).
Recent years have seen the release of a host of MC datasets. Generally, these consist of (document, question, answer) triples to be used in a supervised learning framework. Existing datasets vary in size, difficulty, and collection methodology; however, as pointed out by Rajpurkar et al. (2016), most suffer from one of two shortcomings: those that are designed explicitly to test comprehension (Richardson et al., 2013) are too small for training data-intensive deep learning models, while those that are sufficiently large for deep learning (Hermann et al., 2015; Hill et al., 2016; Bajgar et al., 2016) are generated synthetically, yielding questions that are not posed in natural language and that may not test comprehension directly (Chen et al., 2016). More recently, Rajpurkar et al. (2016) sought to overcome these deficiencies with their crowdsourced dataset, SQuAD.
Here we present a challenging new largescale dataset for machine comprehension: NewsQA. NewsQA contains 119,633 natural language questions posed by crowdworkers on 12,744 news articles from CNN. Answers to these questions consist in spans of text within the corresponding article highlighted by a distinct set of crowdworkers. To build NewsQA we utilized a four-stage collection process designed to encourage exploratory, curiosity-based questions that reflect human information seeking. CNN articles were chosen as the source material because they have been used in the past (Hermann et al., 2015) and, in our view, machine comprehension systems are particularly suited to high-volume, rapidly changing information sources like news.
∗These three authors contributed equally.
As Trischler et al. (2016a), Chen et al. (2016), and others have argued, it is important for datasets to be sufficiently challenging to teach models the abilities we wish them to learn. Thus, in line with Richardson et al. (2013), our goal with NewsQA was to construct a corpus of questions that necessitates reasoning mechanisms, such as synthesis of information across different parts of an article. We designed our collection methodology explicitly to capture such questions.
The challenging characteristics of NewsQA that distinguish it from most previous comprehension tasks are as follows:
1. Answers are spans of arbitrary length within an article, rather than single words or entities.
2. Some questions have no answer in the corresponding article (the null span).
3. There are no candidate answers from which to choose.
4. Our collection process encourages lexical and syntactic divergence between questions and answers.
5. A significant proportion of questions requires reasoning beyond simple word- and contextmatching (as shown in our analysis).
In this paper we describe the collection methodology for NewsQA, provide a variety of statistics to characterize it and contrast it with previous datasets, and assess its difficulty. In particular, we measure human performance and compare it to that of two strong neural-network baselines. Unsurprisingly, humans significantly outperform the models we designed and assessed, achieving an F1 score of 0.694 versus 0.496 for the best-performing machine. We hope that this corpus will spur further advances on the challenging task of machine comprehension.
2 RELATED DATASETS
NewsQA follows in the tradition of several recent comprehension datasets. These vary in size, difficulty, and collection methodology, and each has its own distinguishing characteristics. We agree with Bajgar et al. (2016) who have said “models could certainly benefit from as diverse a collection of datasets as possible.” We discuss this collection below.
2.1 MCTEST
MCTest (Richardson et al., 2013) is a crowdsourced collection of 660 elementary-level children’s stories with associated questions and answers. The stories are fictional, to ensure that the answer must be found in the text itself, and carefully limited to what a young child can understand. Each question comes with a set of 4 candidate answers that range from single words to full explanatory sentences. The questions are designed to require rudimentary reasoning and synthesis of information across sentences, making the dataset quite challenging. This is compounded by the dataset’s size, which limits the training of expressive statistical models. Nevertheless, recent comprehension models have performed well on MCTest (Sachan et al., 2015; Wang et al., 2015), including a highly structured neural model (Trischler et al., 2016a). These models all rely on access to the small set of candidate answers, a crutch that NewsQA does not provide.
2.2 CNN/DAILY MAIL
The CNN/Daily Mail corpus (Hermann et al., 2015) consists of news articles scraped from those outlets with corresponding cloze-style questions. Cloze questions are constructed synthetically by deleting a single entity from abstractive summary points that accompany each article (written presumably by human authors). As such, determining the correct answer relies mostly on recognizing textual entailment between the article and the question. The named entities within an article are identified and anonymized in a preprocessing step and constitute the set of candidate answers; contrast this with NewsQA in which answers often include longer phrases and no candidates are given.
Because the cloze process is automatic, it is straightforward to collect a significant amount of data to support deep-learning approaches: CNN/Daily Mail contains about 1.4 million question-answer pairs. However, Chen et al. (2016) demonstrated that the task requires only limited reasoning and, in
fact, performance of the strongest models (Kadlec et al., 2016; Trischler et al., 2016b; Sordoni et al., 2016) nearly matches that of humans.
2.3 CHILDREN’S BOOK TEST
The Children’s Book Test (CBT) (Hill et al., 2016) was collected using a process similar to that of CNN/Daily Mail. Text passages are 20-sentence excerpts from children’s books available through Project Gutenberg; questions are generated by deleting a single word in the next (i.e., 21st) sentence. Consequently, CBT evaluates word prediction based on context. It is a comprehension task insofar as comprehension is likely necessary for this prediction, but comprehension may be insufficient and other mechanisms may be more important.
2.4 BOOKTEST
Bajgar et al. (2016) convincingly argue that, because existing datasets are not large enough, we have yet to reach the full capacity of existing comprehension models. As a remedy they present BookTest. This is an extension to the named-entity and common-noun strata of CBT that increases their size by over 60 times. Bajgar et al. (2016) demonstrate that training on the augmented dataset yields a model (Kadlec et al., 2016) that matches human performance on CBT. This is impressive and suggests that much is to be gained from more data, but we repeat our concerns about the relevance of story prediction as a comprehension task. We also wish to encourage more efficient learning from less data.
2.5 SQUAD
The comprehension dataset most closely related to NewsQA is SQuAD (Rajpurkar et al., 2016). It consists of natural language questions posed by crowdworkers on paragraphs from high-PageRank Wikipedia articles. As in NewsQA, each answer consists of a span of text from the related paragraph and no candidates are provided. Despite the effort of manual labelling, SQuAD’s size is significant and amenable to deep learning approaches: 107,785 question-answer pairs based on 536 articles.
SQuAD is a challenging comprehension task in which humans far outperform machines. The authors measured human accuracy at 0.905 in F1 (we measured human F1 at 0.807 using a different methodology), whereas at the time of the writing, the strongest published model to date achieves only 0.700 in F1 (Wang & Jiang, 2016b).
3 COLLECTION METHODOLOGY
We collected NewsQA through a four-stage process: article curation, question sourcing, answer sourcing, and validation. We also applied a post-processing step with answer agreement consolidation and span merging to enhance the usability of the dataset.
3.1 ARTICLE CURATION
We retrieve articles from CNN using the script created by Hermann et al. (2015) for CNN/Daily Mail. From the returned set of 90,266 articles, we select 12,744 uniformly at random. These cover a wide range of topics that includes politics, economics, and current events. Articles are partitioned at random into a training set (90%), a development set (5%), and a test set (5%).
3.2 QUESTION SOURCING
It was important to us to collect challenging questions that could not be answered using straightforward word- or context-matching. Like Richardson et al. (2013) we want to encourage reasoning in comprehension models. We are also interested in questions that, in some sense, model human curiosity and reflect actual human use-cases of information seeking. Along a similar line, we consider it an important (though as yet overlooked) capacity of a comprehension model to recognize when given information is inadequate, so we are also interested in questions that may not have sufficient evidence in the text. Our question sourcing stage was designed to solicit questions of this nature, and deliberately separated from the answer sourcing stage for the same reason.
Questioners (a distinct set of crowdworkers) see only a news article’s headline and its summary points (also available from CNN); they do not see the full article itself. They are asked to formulate a question from this incomplete information. This encourages curiosity about the contents of the full article and prevents questions that are simple reformulations of sentences in the text. It also increases the likelihood of questions whose answers do not exist in the text. We reject questions that have significant word overlap with the summary points to ensure that crowdworkers do not treat the summaries as mini-articles, and further discouraged this in the instructions. During collection each Questioner is solicited for up to three questions about an article. They are provided with positive and negative examples to prompt and guide them (detailed instructions are shown in Figure 3).
3.3 ANSWER SOURCING
A second set of crowdworkers (Answerers) provide answers. Although this separation of question and answer increases the overall cognitive load, we hypothesized that unburdening Questioners in this way would encourage more complex questions. Answerers receive a full article along with a crowdsourced question and are tasked with determining the answer. They may also reject the question as nonsensical, or select the null answer if the article contains insufficient information. Answers are submitted by clicking on and highlighting words in the article while instructions encourage the set of answer words to consist in a single continuous span (again, we give an example prompt in the Appendix). For each question we solicit answers from multiple crowdworkers (avg. 2.73) with the aim of achieving agreement between at least two Answerers.
3.4 VALIDATION
Crowdsourcing is a powerful tool but it is not without peril (collection glitches; uninterested or malicious workers). To obtain a dataset of the highest possible quality we use a validation process that mitigates some of these issues. In validation, a third set of crowdworkers sees the full article, a question, and the set of unique answers to that question. We task these workers with choosing the best answer from the candidate set or rejecting all answers. Each article-question pair is validated by an average of 2.48 crowdworkers. Validation was used on those questions without answer-agreement after the previous stage, amounting to 43.2% of all questions.
3.5 ANSWER MARKING AND CLEANUP
After validation, 86.0% of all questions in NewsQA have answers agreed upon by at least two separate crowdworkers—either at the initial answer sourcing stage or in the top-answer selection. This improves the dataset’s quality. We choose to include the questions without agreed answers in the corpus also, but they are specially marked. Such questions could be treated as having the null answer and used to train models that are aware of poorly posed questions.
As a final cleanup step we combine answer spans that are less than 3 words apart (punctuation is discounted). We find that 5.68% of answers consist in multiple spans, while 71.3% of multi-spans are within the 3-word threshold. Looking more closely at the data reveals that the multi-span answers often represent lists. These may present an interesting challenge for comprehension models moving forward.
4 DATA ANALYSIS
We provide a thorough analysis of NewsQA to demonstrate its challenge and its usefulness as a machine comprehension benchmark. The analysis focuses on the types of answers that appear in the dataset and the various forms of reasoning required to solve it.1
4.1 ANSWER TYPES
Following Rajpurkar et al. (2016), we categorize answers based on their linguistic type (see Table 1). This categorization relies on Stanford CoreNLP to generate constituency parses, POS tags, and NER
1Additional statistics are available at http://datasets.maluuba.com/NewsQA/stats.
tags for answer spans (see Rajpurkar et al. (2016) for more details). From the table we see that the majority of answers (22.2%) are common noun phrases. Thereafter, answers are fairly evenly spread among the clause phrase (18.3%), person (14.8%), numeric (9.8%), and other (11.2%) types. Clearly, answers in NewsQA are linguistically diverse.
The proportions in Table 1 only account for cases when an answer span exists. The complement of this set comprises questions with an agreed null answer (9.5% of the full corpus) and answers without agreement after validation (4.5% of the full corpus).
4.2 REASONING TYPES
The forms of reasoning required to solve NewsQA directly influence the abilities that models will learn from the dataset. We stratified reasoning types using a variation on the taxonomy presented by Chen et al. (2016) in their analysis of the CNN/Daily Mail dataset. Types are as follows, in ascending order of difficulty:
1. Word Matching: Important words in the question exactly match words in the immediate context of an answer span such that a keyword search algorithm could perform well on this subset.
2. Paraphrasing: A single sentence in the article entails or paraphrases the question. Paraphrase recognition may require synonymy and word knowledge.
3. Inference: The answer must be inferred from incomplete information in the article or by recognizing conceptual overlap. This typically draws on world knowledge.
4. Synthesis: The answer can only be inferred by synthesizing information distributed across multiple sentences.
5. Ambiguous/Insufficient: The question has no answer or no unique answer in the article.
For both NewsQA and SQuAD, we manually labelled 1,000 examples (drawn randomly from the respective development sets) according to these types and compiled the results in Table 2. Some examples fall into more than one category, in which case we defaulted to the more challenging type. We can see from the table that word matching, the easiest type, makes up the largest subset in both datasets (32.7% for NewsQA and 39.8% for SQuAD). Paraphrasing constitutes a much larger proportion in SQuAD than in NewsQA (34.3% vs 27.0%), possibly a result from the explicit encouragement of lexical variety in SQuAD question sourcing. However, NewsQA significantly outnumbers SQuAD on the distribution of the more difficult forms of reasoning: synthesis and inference make up 33.9% of the data in contrast to 20.5% in SQuAD.
5 BASELINE MODELS
We test the performance of three comprehension systems on NewsQA: human data analysts and two neural models. The first neural model is the match-LSTM (mLSTM) system of Wang & Jiang
(2016b). The second is a model of our own design that is computationally cheaper. We describe these models below but omit the personal details of our analysts. Implementation details of the models are described in Appendix A.
5.1 MATCH-LSTM
There are three stages involved in the mLSTM model. First, LSTM networks encode the document and question (represented by GloVe word embeddings (Pennington et al., 2014)) as sequences of hidden states. Second, an mLSTM network (Wang & Jiang, 2016a) compares the document encodings with the question encodings. This network processes the document sequentially and at each token uses an attention mechanism to obtain a weighted vector representation of the question; the weighted combination is concatenated with the encoding of the current token and fed into a standard LSTM. Finally, a Pointer Network uses the hidden states of the mLSTM to select the boundaries of the answer span. We refer the reader to Wang & Jiang (2016a;b) for full details. At the time of writing, mLSTM is state-of-the-art on SQuAD (see Table 3) so it is natural to test it further on NewsQA.
5.2 THE BILINEAR ANNOTATION RE-ENCODING BOUNDARY (BARB) MODEL
The match-LSTM is computationally intensive since it computes an attention over the entire question at each document token in the recurrence. To facilitate faster experimentation with NewsQA we developed a lighter-weight model (BARB) that achieves similar results on SQuAD2. Our model consists in four stages:
Encoding All words in the document and question are mapped to real-valued vectors using the GloVe embedding matrix W ∈ R|V |×d. This yields d1, . . . ,dn ∈ Rd and q1, . . . ,qm ∈ Rd. A bidirectional GRU network (Bahdanau et al., 2015) takes in di and encodes contextual states hi ∈ RD1 for the document. The same encoder is applied to qj to derive contextual states kj ∈ RD1 for the question.3
Bilinear Annotation Next we compare the document and question encodings using a set of C bilinear transformations,
gij = h T i T [1:C]kj , T c ∈ RD1×D1 , gij ∈ RC ,
which we use to produce an (n×m× C)-dimensional tensor of annotation scores, G = [gij ]. We take the maximum over the question-token (second) dimension and call the columns of the resulting
2With the configurations for the results reported in Section 6.2, one epoch of training on NewsQA takes about 3.9k seconds for BARB and 8.1k seconds for mLSTM.
3A bidirectional GRU concatenates the hidden states of two GRU networks running in opposite directions. Each of these has hidden size 1
2 D1.
matrix gi ∈ RC . We use this matrix as an annotation over the document word dimension. Contrasting the multiplicative application of attention vectors, this annotation matrix is to be concatenated to the encoder RNN input in the re-encoding stage.
Re-encoding For each document word, the input of the re-encoding RNN (another biGRU network) consists of three components: the document encodings hi, the annotation vectors gi, and a binary feature qi indicating whether the document word appears in the question. The resulting vectors fi = [hi;gi; qi] are fed into the re-encoding RNN to produce D2-dimensional encodings ei as input in the boundary-pointing stage.
Boundary pointing Finally, we search for the boundaries of the answer span using a convolutional network (in a process similar to edge detection). Encodings ei are arranged in matrix E ∈ RD2×n. E is convolved with a bank of nf filters, F`k ∈ RD2×w, where w is the filter width, k indexes the different filters, and ` indexes the layer of the convolutional network. Each layer has the same number of filters of the same dimensions. We add a bias term and apply a nonlinearity (ReLU) following each convolution, with the result an (nf × n)-dimensional matrix B`. We use two convolutional layers in the boundary-pointing stage. Given B1 and B2, the answer span’s start- and end-location probabilities are computed using p(s) ∝ exp ( vTs B1 + bs ) and p(e) ∝
exp ( vTe B2 + be ) , respectively. We also concatenate p(s) to the input of the second convolutional layer (along the nf -dimension) so as to condition the end-boundary pointing on the start-boundary. Vectors vs, ve ∈ Rnf and scalars bs, be ∈ R are trainable parameters. We also provide an intermediate level of “guidance” to the annotation mechanism by first reducing the feature dimension C in G with mean-pooling, then maximizing the softmax probabilities in the resulting (n-dimensional) vector corresponding to the answer word positions in each document. This auxiliary task is observed empirically to improve performance.
6 EXPERIMENTS4
6.1 HUMAN EVALUATION
We tested four English speakers (three native and one near-native) on a total of 1,000 questions from the NewsQA development set. As given in Table 3, they averaged 0.694 in F1, which likely represents a ceiling for machine performance. Our students’ exact match (EM) scores are relatively low at 0.465. This is because in many cases there are multiple ways to select semantically equivalent answers, e.g., “1996” versus “in 1996”. We also compared human performance on the answers that had agreement with and without validation, finding a difference of only 1.4 percentage points F1. This suggests our validation stage yields good-quality answers.
The original SQuAD evaluation of human performance compares separate answers given by crowdworkers; for a closer comparison with NewsQA, we replicated our human test on the same number of validation data (1,000). We measured their answers against the second group of crowdsourced responses in SQuAD’s development set, as in Rajpurkar et al. (2016). Our students scored 0.807 in F1.
6.2 MODEL PERFORMANCE
Performance of the baseline models and humans is measured by EM and F1 with the official evaluation script from SQuAD and listed in Table 3. Unless otherwise stated, hyperparameters are determined by hyperopt (Appendix A). The gap between human and machine performance on NewsQA is a striking 0.198 points F1 — much larger than the gap on SQuAD (0.098) under the same human evaluation scheme. The gaps suggest a large margin for improvement with automated methods.
Figure 1 stratifies model (BARB) performance according to answer type (left) and reasoning type (right) as defined in Sections 4.1 and 4.2, respectively. The answer-type stratification suggests that
4All experiments in this section use the subset of NewsQA dataset with answer agreements (92,549 samples for training, 5,166 for validation, and 5,126 for testing). We leave the challenge of identifying the unanswerable questions for future work.
Table 3: Performance of several methods and humans on the SQuAD and NewsQA datasets. Superscript 1 indicates the results are taken from Rajpurkar et al. (2016), and 2 from Wang & Jiang (2016b).
SQuAD Exact Match F1
Model Dev Test Dev Test
Random1 0.11 0.13 0.41 0.43 mLSTM2 0.591 0.595 0.700 0.703 BARB 0.591 - 0.709 -
Human1 0.803 0.770 0.905 0.868 Human (ours) 0.650 - 0.807 -
NewsQA Exact Match F1
Model Dev Test Dev Test
Random 0.00 0.00 0.30 0.30 mLSTM 0.344 0.349 0.496 0.500 BARB 0.361 0.341 0.496 0.482
Human 0.465 - 0.694 -
Date/time
Numeric
Person Adjective Phrase
Location
Prepositional Phrase Common Noun Phrase
Other Other entity Clause Phrase
Verb Phrase
0 0.2 0.4 0.6 0.8
F1 EM
Word Matching
Paraphrasing
Inference
Synthesis
Ambiguous/ Insufficient
0.000 0.150 0.300 0.450 0.600 0.750 0.900
NewsQA SQuAD
Figure 1: Left: BARB performance (F1 and EM) stratified by answer type on the full development set of NewsQA. Right: BARB performance (F1) stratified by reasoning type on the human-assessed subset on both NewsQA and SQuAD. Error bars indicate performance differences between BARB and human annotators.
the model is better at pointing to named entities compared to other types of answers. The reasoningtype stratification, on the other hand, shows that questions requiring inference and synthesis are, not surprisingly, more difficult for the model. Consistent with observations in Table 3, stratified performance on NewsQA is significantly lower than on SQuAD. The difference is smallest on word matching and largest on synthesis. We postulate that the longer stories in NewsQA make synthesizing information from separate sentences more difficult, since the relevant sentences may be farther apart. This requires the model to track longer-term dependencies.
6.3 SENTENCE-LEVEL SCORING
We propose a simple sentence-level subtask as an additional quantitative demonstration of the relative difficulty of NewsQA. Given a document and a question, the goal is to find the sentence containing the answer span. We hypothesize that simple techniques like word-matching are inadequate to this task owing to the more involved reasoning required by NewsQA.
We employ a technique that resembles inverse document frequency (idf ), which we call inverse sentence frequency (isf ). Given a sentence Si from an article and its corresponding question Q, the isf score is given by the sum of the idf scores of the words common to Si and Q (each sentence is treated as a document for the idf computation). The sentence with the highest isf is taken as the answer sentence S∗, that is,
S∗ = argmax i ∑ w∈Si∩Q isf (w).
The isf method achieves an impressive 79.4% sentence-level accuracy on SQuAD’s development set but only 35.4% accuracy on NewsQA’s development set, highlighting the comparative difficulty of the latter. To eliminate the difference in article length as a possible cause of the performance difference, we also artificially increased the article lengths in SQuAD by concatenating adjacent SQuAD articles from the same Wikipedia document. Accuracy decreases as expected with the increased SQuAD article length, yet remains significantly higher than that on NewsQA with comparable or even larger article length (Table 4).
7 CONCLUSION
We have introduced a challenging new comprehension dataset: NewsQA. We collected the 100,000+ examples of NewsQA using teams of crowdworkers, who variously read CNN articles or highlights, posed questions about them, and determined answers. Our methodology yields diverse answer types and a significant proportion of questions that require some reasoning ability to solve. This makes the corpus challenging, as confirmed by the large performance gap between humans and deep neural models (0.198 in F1). By its size and complexity, NewsQA makes a significant extension to the existing body of comprehension datasets. We hope that our corpus will spur further advances in machine comprehension and guide the development of literate artificial intelligence.
ACKNOWLEDGMENTS
The authors would like to thank Çağlar Gülçehre, Sandeep Subramanian and Saizheng Zhang for helpful discussions, and Pranav Subramani for the graphs.
B DATA COLLECTION USER INTERFACE
Here we present the user interfaces used in question sourcing, answer sourcing, and question/answer validation.
5https://github.com/hyperopt/hyperopt

It would seem that the shelf life of a dataset has decreased rapidly in recent literature. SQuAD dataset has been heavily pursued as soon as it hit online couple months ago, the best performance on their leaderboard now reaching to 82%. This is rather surprising when taking into account the fact that the formal conference presentation of the dataset took place only a month ago at EMNLP’16, and that the reported machine performance (at the time of paper submission) was only at 51%. One reasonable speculation is that the dataset may have not been hard enough.
NewsQA, the paper in submission, aims to address this concern by presenting a dataset of a comparable scale created through different QA collection strategies. Most notably, the authors solicit questions without requiring answers from the same turkers, in order to promote more diverse and hard-to-answer questions. Another notable difference is that the questions are gathered without showing the content of the news articles, and the dataset makes use of a bigger subset of CNN/Daily corpus (12K / 90K), as opposed to a much smaller subset (500 / 90K) used by SQuAD.
In sum, I think NewsQA dataset presents an effort to construct a harder, large-scale reading comprehension challenge, a recently hot research topic for which we don’t yet have satisfying datasets. While not without its own weaknesses, I think this dataset presents potential values compared to what are available out there today.
That said, the paper does read like it was prepared in a hurry, as there are numerous small things that the authors could have done better. As a result, I do wonder about the quality of the dataset. For one, human performance of SQuAD measured by the authors (70.5 - 82%) is lower than that reported by SQuAD (80.3 - 90.5%). I think this sort of difference can easily happen depending on the level of carefulness the annotators can maintain. After all, not all humans have the same level of carefulness or even the same level of reading comprehension. I think it’d be the best if the authors can try to explain the reason behind these differences, and if possible, perform a more careful measurement of human performance. If anything, I don’t think it looks favorable for NewsQA if the human performance is only at the level of 74.9%, as it looks as if the difficulty of the dataset comes mainly from the potential noise from the QA collection process, which implies that the low model performance could result from not necessarily because of the difficulty of the comprehension and reasoning, but because of incorrect answers given by human annotators.
I’m also not sure whether the design choice of not presenting the news article when soliciting the questions was a good one. I can imagine that people might end up asking similar generic questions when not enough context has been presented. Perhaps taking a hybrid, what I would like to suggest is to present news articles where some sentences or phrases are randomly redacted, so that the question generators can have a bit more context while not having the full material in front of them.
Yet another way of encouraging the turkers from asking too trivial questions is to engage an automatic QA system on the fly — turkers must construct a QA pair for which an existing state-of-the-art system cannot answer correctly.


ADVERSARIAL FEATURE LEARNING
The ability of the Generative Adversarial Networks (GANs) framework to learn generative models mapping from simple latent distributions to arbitrarily complex data distributions has been demonstrated empirically, with compelling results showing that the latent space of such generators captures semantic variation in the data distribution. Intuitively, models trained to predict these semantic latent representations given data may serve as useful feature representations for auxiliary problems where semantics are relevant. However, in their existing form, GANs have no means of learning the inverse mapping – projecting data back into the latent space. We propose Bidirectional Generative Adversarial Networks (BiGANs) as a means of learning this inverse mapping, and demonstrate that the resulting learned feature representation is useful for auxiliary supervised discrimination tasks, competitive with contemporary approaches to unsupervised and self-supervised feature learning.
1 INTRODUCTION
Deep convolutional networks (convnets) have become a staple of the modern computer vision pipeline. After training these models on a massive database of image-label pairs like ImageNet (Russakovsky et al., 2015), the network easily adapts to a variety of similar visual tasks, achieving impressive results on image classification (Donahue et al., 2014; Zeiler & Fergus, 2014; Razavian et al., 2014) or localization (Girshick et al., 2014; Long et al., 2015) tasks. In other perceptual domains such as natural language processing or speech recognition, deep networks have proven highly effective as well (Bahdanau et al., 2015; Sutskever et al., 2014; Vinyals et al., 2015; Graves et al., 2013). However, all of these recent results rely on a supervisory signal from large-scale databases of hand-labeled data, ignoring much of the useful information present in the structure of the data itself.
Meanwhile, Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have emerged as a powerful framework for learning generative models of arbitrarily complex data distributions. The GAN framework learns a generator mapping samples from an arbitrary latent distribution to data, as well as an adversarial discriminator which tries to distinguish between real and generated samples as accurately as possible. The generator’s goal is to “fool” the discriminator by producing samples which are as close to real data as possible. When trained on databases of natural images, GANs produce impressive results (Radford et al., 2016; Denton et al., 2015).
Interpolations in the latent space of the generator produce smooth and plausible semantic variations, and certain directions in this space correspond to particular semantic attributes along which the data distribution varies. For example, Radford et al. (2016) showed that a GAN trained on a database of human faces learns to associate particular latent directions with gender and the presence of eyeglasses.
A natural question arises from this ostensible “semantic juice” flowing through the weights of generators learned using the GAN framework: can GANs be used for unsupervised learning of rich feature representations for arbitrary data distributions? An obvious issue with doing so is that the
generator maps latent samples to generated data, but the framework does not include an inverse mapping from data to latent representation.
Hence, we propose a novel unsupervised feature learning framework, Bidirectional Generative Adversarial Networks (BiGAN). The overall model is depicted in Figure 1. In short, in addition to the generator G from the standard GAN framework (Goodfellow et al., 2014), BiGAN includes an encoder E which maps data x to latent representations z. The BiGAN discriminator D discriminates not only in data space (x versus G(z)), but jointly in data and latent space (tuples (x, E(x)) versus (G(z), z)), where the latent component is either an encoder output E(x) or a generator input z.
It may not be obvious from this description that the BiGAN encoder E should learn to invert the generator G. The two modules cannot directly “communicate” with one another: the encoder never “sees” generator outputs (E(G(z)) is not computed), and vice versa. Yet, in Section 3, we will both argue intuitively and formally prove that the encoder and generator must learn to invert one another in order to fool the BiGAN discriminator.
Because the BiGAN encoder learns to predict features z given data x, and prior work on GANs has demonstrated that these features capture semantic attributes of the data, we hypothesize that a trained BiGAN encoder may serve as a useful feature representation for related semantic tasks, in the same way that fully supervised visual models trained to predict semantic “labels” given images serve as powerful feature representations for related visual tasks. In this context, a latent representation z may be thought of as a “label” for x, but one which came for “free,” without the need for supervision.
An alternative approach to learning the inverse mapping from data to latent representation is to directly model p(z|G(z)), predicting generator input z given generated data G(z). We’ll refer to this alternative as a latent regressor, later arguing (Section 4.1) that the BiGAN encoder may be preferable in a feature learning context, as well as comparing the approaches empirically.
BiGANs are a robust and highly generic approach to unsupervised feature learning, making no assumptions about the structure or type of data to which they are applied, as our theoretical results will demonstrate. Our empirical studies will show that despite their generality, BiGANs are competitive with contemporary approaches to self-supervised and weakly supervised feature learning designed specifically for a notoriously complex data distribution – natural images.
Dumoulin et al. (2016) independently proposed an identical model in their concurrent work, exploring the case of a stochastic encoder E and the ability of such models to learn in a semi-supervised setting.
2 PRELIMINARIES
Let pX(x) be the distribution of our data for x ∈ ΩX (e.g. natural images). The goal of generative modeling is capture this data distribution using a probabilistic model. Unfortunately, exact modeling of this probability density function is computationally intractable (Hinton et al., 2006; Salakhutdinov & Hinton, 2009) for all but the most trivial models. Generative Adversarial Networks (GANs) (Good-
fellow et al., 2014) instead model the data distribution as a transformation of a fixed latent distribution pZ(z) for z ∈ ΩZ. This transformation, called a generator, is expressed as a deterministic feed forward network G : ΩZ → ΩX with pG(x|z) = δ (x−G(z)) and pG(x) = Ez∼pZ [pG(x|z)]. The goal is to train a generator such that pG(x) ≈ pX(x). The GAN framework trains a generator, such that no discriminative model D : ΩX 7→ [0, 1] can distinguish samples of the data distribution from samples of the generative distribution. Both generator and discriminator are learned using the adversarial (minimax) objective min
G max D V (D,G), where
V (D,G) := Ex∼pX [logD(x)] + Ex∼pG [log (1−D(x))]︸ ︷︷ ︸ Ez∼pZ [log(1−D(G(z)))]
(1)
Goodfellow et al. (2014) showed that for an ideal discriminator the objective C(G) := maxD V (D,G) is equivalent to the Jensen-Shannon divergence between the two distributions pG and pX.
The adversarial objective 1 does not directly lend itself to an efficient optimization, as each step in the generator G requires a full discriminator D to be learned. Furthermore, a perfect discriminator no longer provides any gradient information to the generator, as the gradient of any global or local maximum of V (D,G) is 0. To provide a strong gradient signal nonetheless, Goodfellow et al. (2014) slightly alter the objective between generator and discriminator updates, while keeping the same fixed point characteristics. They also propose to optimize (1) using an alternating optimization switching between updates to the generator and discriminator. While this optimization is not guaranteed to converge, empirically it works well if the discriminator and generator are well balanced.
Despite the empirical strength of GANs as generative models of arbitrary data distributions, it is not clear how they can be applied as an unsupervised feature representation. One possibility for learning such representations is to learn an inverse mapping regressing from generated data G(z) back to the latent input z. However, unless the generator perfectly models the data distribution pX, a nearly impossible objective for a complex data distribution such as that of high-resolution natural images, this idea may prove insufficient.
3 BIDIRECTIONAL GENERATIVE ADVERSARIAL NETWORKS
In Bidirectional Generative Adversarial Networks (BiGANs) we not only train a generator, but additionally train an encoder E : ΩX → ΩZ. The encoder induces a distribution pE(z|x) = δ(z − E(x)) mapping data points x into the latent feature space of the generative model. The discriminator is also modified to take input from the latent space, predicting PD(Y |x, z), where Y = 1 if x is real (sampled from the real data distribution pX), and Y = 0 if x is generated (the output of G(z), z ∼ pZ). The BiGAN training objective is defined as a minimax objective
min G,E max D V (D,E,G) (2)
where V (D,E,G) := Ex∼pX [ Ez∼pE(·|x) [logD(x, z)]︸ ︷︷ ︸
logD(x,E(x))
] + Ez∼pZ [ Ex∼pG(·|z) [log (1−D(x, z))]︸ ︷︷ ︸
log(1−D(G(z),z))
] .
(3)
We optimize this minimax objective using the same alternating gradient based optimization as Goodfellow et al. (2014). See Section 3.4 for details.
BiGANs share many of the theoretical properties of GANs (Goodfellow et al., 2014), while additionally guaranteeing that at the global optimum, G and E are each other’s inverse. BiGANs are also closely related to autoencoders with an `0 loss function. In the following sections we highlight some of the appealing theoretical properties of BiGANs.
Definitions Let pGZ(x, z) := pG(x|z)pZ(z) and pEX(x, z) := pE(z|x)pX(x) be the joint distributions modeled by the generator and encoder respectively. Ω := ΩX × ΩZ is the joint latent and
data space. For a region R ⊆ Ω, PEX(R) := ∫ Ω pEX(x, z)1[(x,z)∈R] d(x, z) = ∫ ΩX pX(x) ∫ ΩZ pE(z|x)1[(x,z)∈R] dzdx
PGZ(R) := ∫
Ω pGZ(x, z)1[(x,z)∈R] d(x, z) = ∫ ΩZ pZ(z) ∫ ΩX pG(x|z)1[(x,z)∈R] dx dz
are probability measures over that region. We also define PX(RX) := ∫
ΩX pX(x)1[x∈RX] dx PZ(RZ) := ∫ ΩZ pZ(z)1[z∈RZ] dz
as measures over regions RX ⊆ ΩX and RZ ⊆ ΩZ. We refer to the set of features and data samples in the support of PX and PZ as Ω̂X := supp(PX) and Ω̂Z := supp(PZ) respectively. DKL (P ||Q ) and DJS (P ||Q ) respectively denote the Kullback-Leibler (KL) and Jensen-Shannon divergences between probability measures P and Q. By definition,
DKL (P ||Q ) := Ex∼P [log fPQ(x)] DJS (P ||Q ) := 12 ( DKL ( P ∣∣∣∣∣∣ P+Q2 )+ DKL (Q ∣∣∣∣∣∣ P+Q2 )) ,
where fPQ := dPdQ is the Radon-Nikodym (RN) derivative of measure P with respect to measure Q, with the defining property that P (R) = ∫ R fPQ dQ. The RN derivative fPQ : Ω 7→ R≥0 is defined for any measures P and Q on space Ω such that P is absolutely continuous with respect to Q: i.e., for any R ⊆ Ω, P (R) > 0 =⇒ Q(R) > 0.
3.1 OPTIMAL DISCRIMINATOR, GENERATOR, & ENCODER
We start by characterizing the optimal discriminator for any generator and encoder, following Goodfellow et al. (2014). This optimal discriminator then allows us to reformulate objective (3), and show that it reduces to the Jensen-Shannon divergence between the joint distributions PEX and PGZ.
Proposition 1 For any E and G, the optimal discriminator D∗EG := arg maxD V (D,E,G) is the Radon-Nikodym derivative fEG := dPEXd(PEX+PGZ) : Ω 7→ [0, 1] of measure PEX with respect to measure PEX + PGZ.
Proof. Given in Appendix A.1.
This optimal discriminator now allows us to characterize the optimal generator and encoder.
Proposition 2 The encoder and generator’s objective for an optimal discriminator C(E,G) := maxD V (D,E,G) = V (D ∗ EG, E,G) can be rewritten in terms of the Jensen-Shannon divergence between measures PEX and PGZ as C(E,G) = 2 DJS (PEX ||PGZ )− log 4.
Proof. Given in Appendix A.2.
Theorem 1 The global minimum of C(E,G) is achieved if and only if PEX = PGZ. At that point, C(E,G) = − log 4 and D∗EG = 12 .
Proof. From Proposition 2, we have that C(E,G) = 2 DJS (PEX ||PGZ ) − log 4. The JensenShannon divergence DJS (P ||Q ) ≥ 0 for any P and Q, and DJS (P ||Q ) = 0 if and only if P = Q. Therefore, the global minimum of C(E,G) occurs if and only if PEX = PGZ, and at this point the value is C(E,G) = − log 4. Finally, PEX = PGZ implies that the optimal discriminator is chance: D∗EG = dPEX d(PEX+PGZ) = dPEX2 dPEX = 1 2 .
The optimal discriminator, encoder, and generator of BiGAN are similar to the optimal discriminator and generator of the GAN framework (Goodfellow et al., 2014). However, an important difference is that BiGAN optimizes a Jensen-Shannon divergence between a joint distribution over both data X and latent features Z. This joint divergence allows us to further characterize properties of G and E, as shown below.
3.2 OPTIMAL GENERATOR & ENCODER ARE INVERSES
We first present an intuitive argument that, in order to “fool” a perfect discriminator, a deterministic BiGAN encoder and generator must invert each other. (Later we will formally state and prove this
property.) Consider a BiGAN discriminator input pair (x, z). Due to the sampling procedure, (x, z) must satisfy at least one of the following two properties:
(a) x ∈ Ω̂X ∧ E(x) = z (b) z ∈ Ω̂Z ∧ G(z) = x If only one of these properties is satisfied, a perfect discriminator can infer the source of (x, z) with certainty: if only (a) is satisfied, (x, z) must be an encoder pair (x, E(x)) and D∗EG(x, z) = 1; if only (b) is satisfied, (x, z) must be a generator pair (G(z), z) and D∗EG(x, z) = 0.
Therefore, in order to fool a perfect discriminator at (x, z) (so that 0 < D∗EG(x, z) < 1), E and G must satisfy both (a) and (b). In this case, we can substitute the equality E(x) = z required by (a) into the equality G(z) = x required by (b), and vice versa, giving the inversion properties x = G(E(x)) and z = E(G(z)).
Formally, we show in Theorem 2 that the optimal generator and encoder invert one another almost everywhere on the support Ω̂X and Ω̂Z of PX and PZ.
Theorem 2 If E and G are an optimal encoder and generator, then E = G−1 almost everywhere; that is, G(E(x)) = x for PX-almost every x ∈ ΩX, and E(G(z)) = z for PZ-almost every z ∈ ΩZ.
Proof. Given in Appendix A.4.
While Theorem 2 characterizes the encoder and decoder at their optimum, due to the non-convex nature of the optimization, this optimum might never be reached. Experimentally, Section 4 shows that on standard datasets, the two are approximate inverses; however, they are rarely exact inverses. It is thus also interesting to show what objective BiGAN optimizes in terms of E and G. Next we show that BiGANs are closely related to autoencoders with an `0 loss function.
3.3 RELATIONSHIP TO AUTOENCODERS
As argued in Section 1, a model trained to predict features z given data x should learn useful semantic representations. Here we show that the BiGAN objective forces the encoder E to do exactly this: in order to fool the discriminator at a particular z, the encoder must invert the generator at that z, such that E(G(z)) = z.
Theorem 3 The encoder and generator objective given an optimal discriminator C(E,G) := maxD V (D,E,G) can be rewritten as an `0 autoencoder loss function
C(E,G) = Ex∼pX [ 1[E(x)∈Ω̂Z∧G(E(x))=x] log fEG(x, E(x)) ] +
Ez∼pZ [ 1[G(z)∈Ω̂X∧E(G(z))=z] log (1− fEG(G(z), z)) ] with log fEG ∈ (−∞, 0) and log (1− fEG) ∈ (−∞, 0) PEX-almost and PGZ-almost everywhere.
Proof. Given in Appendix A.5.
Here the indicator function 1[G(E(x))=x] in the first term is equivalent to an autoencoder with `0 loss, while the indicator 1[E(G(z))=z] in the second term shows that the BiGAN encoder must invert the generator, the desired property for feature learning. The objective further encourages the functions E(x) and G(z) to produce valid outputs in the support of PZ and PX respectively. Unlike regular autoencoders, the `0 loss function does not make any assumptions about the structure or distribution of the data itself; in fact, all the structural properties of BiGAN are learned as part of the discriminator.
3.4 LEARNING
In practice, as in the GAN framework (Goodfellow et al., 2014), each BiGAN module D, G, and E is a parametric function (with parameters θD, θG, and θE , respectively). As a whole, BiGAN can be optimized using alternating stochastic gradient steps. In one iteration, the discriminator parameters θD are updated by taking one or more steps in the positive gradient direction ∇θDV (D,E,G), then the encoder parameters θE and generator parameters θG are together updated by taking a step in the negative gradient direction −∇θE ,θGV (D,E,G). In both cases, the expectation terms of
V (D,E,G) are estimated using mini-batches of n samples {x(i) ∼ pX}ni=1 and {z(i) ∼ pZ}ni=1 drawn independently for each update step.
Goodfellow et al. (2014) found that an objective in which the real and generated labels Y are swapped provides stronger gradient signal to G. We similarly observed in BiGAN training that an “inverse” objective provides stronger gradient signal to G and E. For efficiency, we also update all modules D, G, and E simultaneously at each iteration, rather than alternating between D updates and G, E updates. See Appendix B for details.
3.5 GENERALIZED BIGAN
It is often useful to parametrize the output of the generator G and encoder E in a different, usually smaller, space Ω′X and Ω ′ Z rather than the original ΩX and ΩZ. For example, for visual feature learning, the images input to the encoder should be of similar resolution to images used in the evaluation. On the other hand, generating high resolution images remains difficult for current generative models. In this situation, the encoder may take higher resolution input while the generator output and discriminator input remain low resolution.
We generalize the BiGAN objective V (D,G,E) (3) with functions gX : ΩX 7→ Ω′X and gZ : ΩZ 7→ Ω′Z, and encoderE : ΩX 7→ Ω′Z, generatorG : ΩZ 7→ Ω′X, and discriminatorD : Ω′X×Ω′Z 7→ [0, 1]:
Ex∼pX [ Ez′∼pE(·|x) [logD(gX(x), z
′)]︸ ︷︷ ︸ logD(gX(x),E(x))
] + Ez∼pZ [ Ex′∼pG(·|z) [log (1−D(x
′, gZ(z)))]︸ ︷︷ ︸ log(1−D(G(z),gZ(z))) ] An identity gX(x) = x and gZ(z) = z (and Ω′X = ΩX, Ω ′ Z = ΩZ) yields the original objective. For visual feature learning with higher resolution encoder inputs, gX is an image resizing function that downsamples a high resolution image x ∈ ΩX to a lower resolution image x′ ∈ Ω′X, as output by the generator. (gZ is identity.)
In this case, the encoder and generator respectively induce probability measures PEX′ and PGZ′ over regions R ⊆ Ω′ of the joint space Ω′ := Ω′X × Ω′Z, with PEX′(R) :=∫
ΩX ∫ Ω′X ∫ Ω′Z pEX(x, z ′)1[(x′,z′)∈R]δ(gX(x) − x′) dz′ dx′ dx = ∫ ΩX pX(x)1[(gX(x),E(x))∈R] dx, and PGZ′ defined analogously. For optimal E and G, we can show PEX′ = PGZ′ : a generalization of Theorem 1. When E and G are deterministic and optimal, Theorem 2 – that E and G invert one another – can also be generalized: ∃z∈Ω̂Z{E(x) = gZ(z) ∧ G(z) = gX(x)} for PX-almost every x ∈ ΩX, and ∃x∈Ω̂X{E(x) = gZ(z) ∧ G(z) = gX(x)} for PZ-almost every z ∈ ΩZ.
4 EVALUATION
We evaluate the feature learning capabilities of BiGANs by first training them unsupervised as described in Section 3.4, then transferring the encoder’s learned feature representations for use in auxiliary supervised learning tasks. To demonstrate that BiGANs are able to learn meaningful feature representations both on arbitrary data vectors, where the model is agnostic to any underlying structure, as well as very high-dimensional and complex distributions, we evaluate on both permutation-invariant MNIST (LeCun et al., 1998) and on the high-resolution natural images of ImageNet (Russakovsky et al., 2015).
In all experiments, each module D, G, and E is a parametric deep (multi-layer) network. The BiGAN discriminator D(x, z) takes data x as its initial input, and at each linear layer thereafter, the latent representation z is transformed using a learned linear transformation to the hidden layer dimension and added to the non-linearity input.
4.1 BASELINE METHODS
Besides the BiGAN framework presented above, we considered alternative approaches to learning feature representations using different GAN variants.
Discriminator The discriminatorD in a standard GAN takes data samples x ∼ pX as input, making its learned intermediate representations natural candidates as feature representations for related tasks.
97.39 97.30 97.44 97.13 97.58 97.63
This alternative is appealing as it requires no additional machinery, and is the approach used for unsupervised feature learning in Radford et al. (2016). On the other hand, it is not clear that the task of distinguishing between real and generated data requires or benefits from intermediate representations that are useful as semantic feature representations. In fact, if G successfully generates the true data distribution pX(x), D may ignore the input data entirely and predict P (Y = 1) = P (Y = 1|x) = 12 unconditionally, not learning any meaningful intermediate representations.
Latent regressor We consider an alternative encoder training by minimizing a reconstruction loss L(z, E(G(z))), after or jointly during a regular GAN training, called latent regressor or joint latent regressor respectively. We use a sigmoid cross entropy loss L as it naturally maps to a uniformly distributed output space. Intuitively, a drawback of this approach is that, unlike the encoder in a BiGAN, the latent regressor encoder E is trained only on generated samples G(z), and never “sees” real data x ∼ pX. While this may not be an issue in the theoretical optimum where pG(x) = pX(x) exactly – i.e., G perfectly generates the data distribution pX – in practice, for highly complex data distributions pX, such as the distribution of natural images, the generator will almost never achieve this perfect result. The fact that the real data x are never input to this type of encoder limits its utility as a feature representation for related tasks, as shown later in this section.
4.2 PERMUTATION-INVARIANT MNIST
We first present results on permutation-invariant MNIST (LeCun et al., 1998). In the permutationinvariant setting, each 28×28 digit image must be treated as an unstructured 784D vector (Goodfellow et al., 2013). In our case, this condition is met by designing each module as a multi-layer perceptron (MLP), agnostic to the underlying spatial structure in the data (as opposed to a convnet, for example). See Appendix C.1 for more architectural and training details. We set the latent distribution pZ = [U(−1, 1)]50 – a 50D continuous uniform distribution. Table 1 compares the encoding learned by a BiGAN-trained encoder E with the baselines described in Section 4.1, as well as autoencoders (Hinton & Salakhutdinov, 2006) trained directly to minimize either `2 or `1 reconstruction error. The same architecture and optimization algorithm is used across all methods. All methods, including BiGAN, perform at roughly the same level. This result is not overly surprising given the relative simplicity of MNIST digits. For example, digits generated by G in a GAN nearly perfectly match the data distribution (qualitatively), making the latent regressor (LR) baseline method a reasonable choice, as argued in Section 4.1. Qualitative results are presented in Figure 2.
4.3 IMAGENET
Next, we present results from training BiGANs on ImageNet LSVRC (Russakovsky et al., 2015), a large-scale database of natural images. GANs trained on ImageNet cannot perfectly reconstruct
the data, but often capture some interesting aspects. Here, each of D, G, and E is a convnet. In all experiments, the encoder E architecture follows AlexNet (Krizhevsky et al., 2012) through the fifth and last convolution layer (conv5). We also experiment with an AlexNet-based discriminator D as a baseline feature learning approach. We set the latent distribution pZ = [U(−1, 1)]200 – a 200D continuous uniform distribution. Additionally, we experiment with higher resolution encoder input images – 112× 112 rather than the 64× 64 used elsewhere – using the generalization described in Section 3.5. See Appendix C.2 for more architectural and training details.
Qualitative results The convolutional filters learned by each of the three modules are shown in Figure 3. We see that the filters learned by the encoder E have clear Gabor-like structure, similar to those originally reported for the fully supervised AlexNet model (Krizhevsky et al., 2012). The filters also have similar “grouping” structure where one half (the bottom half, in this case) is more color sensitive, and the other half is more edge sensitive. (This separation of the filters occurs due to the AlexNet architecture maintaining two separate filter paths for computational efficiency.)
In Figure 4 we present sample generations G(z), as well as real data samples x and their BiGAN reconstructions G(E(x)). The reconstructions, while certainly imperfect, demonstrate empirically that
the BiGAN encoder E and generator G learn approximate inverse mappings, as shown theoretically in Theorem 2. In Appendix C.2, we present nearest neighbors in the BiGAN learned feature space.
ImageNet classification Following Noroozi & Favaro (2016), we evaluate by freezing the first N layers of our pretrained network and randomly reinitializing and training the remainder fully supervised for ImageNet classification. Results are reported in Table 2.
VOC classification, detection, and segmentation We evaluate the transferability of BiGAN representations to the PASCAL VOC (Everingham et al., 2014) computer vision benchmark tasks, including classification, object detection, and semantic segmentation. The classification task involves simple binary prediction of presence or absence in a given image for each of 20 object categories. The object detection and semantic segmentation tasks go a step further by requiring the objects to be localized, with semantic segmentation requiring this at the finest scale: pixelwise prediction of object identity. For detection, the pretrained model is used as the initialization for Fast R-CNN (Girshick, 2015) (FRCN) training; and for semantic segmentation, the model is used as the initialization for Fully Convolutional Network (Long et al., 2015) (FCN) training, in each case replacing the AlexNet (Krizhevsky et al., 2012) model trained fully supervised for ImageNet classification. We report results on each of these tasks in Table 3, comparing BiGANs with contemporary approaches to unsupervised (Krähenbühl et al., 2016) and self-supervised (Doersch et al., 2015; Agrawal et al., 2015; Wang & Gupta, 2015; Pathak et al., 2016) feature learning in the visual domain, as well as the baselines discussed in Section 4.1.
4.4 DISCUSSION
Despite making no assumptions about the underlying structure of the data, the BiGAN unsupervised feature learning framework offers a representation competitive with existing self-supervised and even weakly supervised feature learning approaches for visual feature learning, while still being a purely generative model with the ability to sample data x and predict latent representation z. Furthermore, BiGANs outperform the discriminator (D) and latent regressor (LR) baselines discussed in Section 4.1, confirming our intuition that these approaches may not perform well in the regime of highly complex data distributions such as that of natural images. The version in which the encoder takes a higher resolution image than output by the generator (BiGAN 112× 112 E) performs better still, and this strategy is not possible under the LR and D baselines as each of those modules take generator outputs as their input.
Although existing self-supervised approaches have shown impressive performance and thus far tended to outshine purely unsupervised approaches in the complex domain of high-resolution images, purely unsupervised approaches to feature learning or pre-training have several potential benefits.
BiGAN and other unsupervised learning approaches are agnostic to the domain of the data. The self-supervised approaches are specific to the visual domain, in some cases requiring weak supervision from video unavailable in images alone. For example, the methods are not applicable in the permutation-invariant MNIST setting explored in Section 4.2, as the data are treated as flat vectors rather than 2D images.
Furthermore, BiGAN and other unsupervised approaches needn’t suffer from domain shift between the pre-training task and the transfer task, unlike self-supervised methods in which some aspect of the data is normally removed or corrupted in order to create a non-trivial prediction task. In the context prediction task (Doersch et al., 2015), the network sees only small image patches – the global image structure is unobserved. In the context encoder or inpainting task (Pathak et al., 2016), each image is corrupted by removing large areas to be filled in by the prediction network, creating inputs with dramatically different appearance from the uncorrupted natural images seen in the transfer tasks.
Other approaches (Agrawal et al., 2015; Wang & Gupta, 2015) rely on auxiliary information unavailable in the static image domain, such as video, egomotion, or tracking. Unlike BiGAN, such approaches cannot learn feature representations from unlabeled static images.
We finally note that the results presented here constitute only a preliminary exploration of the space of model architectures possible under the BiGAN framework, and we expect results to improve significantly with advancements in generative image models and discriminative convolutional networks alike.
ACKNOWLEDGMENTS
The authors thank Evan Shelhamer, Jonathan Long, and other Berkeley Vision labmates for helpful discussions throughout this work. This work was supported by DARPA, AFRL, DoD MURI award N000141110688, NSF awards IIS-1427425 and IIS-1212798, and the Berkeley Artificial Intelligence Research laboratory. The GPUs used for this work were donated by NVIDIA.
APPENDIX A ADDITIONAL PROOFS
A.1 PROOF OF PROPOSITION 1 (OPTIMAL DISCRIMINATOR)
Proposition 1 For any E and G, the optimal discriminator D∗EG := arg maxD V (D,E,G) is the Radon-Nikodym derivative fEG := dPEXd(PEX+PGZ) : Ω 7→ [0, 1] of measure PEX with respect to measure PEX + PGZ.
Proof. For measures P and Q on space Ω, with P absolutely continuous with respect to Q, the RN derivative fPQ := dPdQ exists, and we have
Ex∼P [g(x)] = ∫
Ω g dP = ∫ Ω g dPdQ dQ = ∫ Ω gfPQ dQ = Ex∼Q [fPQ(x)g(x)] . (4)
Let the probability measure PEG := PEX+PGZ2 denote the average of measures PEX and PGZ. Both PEX and PGZ are each absolutely continuous with respect to PEG. Hence the RN derivatives fEG :=
dPEX d(PEX+PGZ) = 12 dPEX dPEG and fGE := dPGZd(PEX+PGZ) = 1 2 dPGZ dPEG exist and sum to 1:
fEG + fGE = dPEX d(PEX+PGZ) + dPGZd(PEX+PGZ) = d(PEX+PGZ) d(PEX+PGZ) = 1. (5)
We use (4) and (5) to rewrite the objective V (3) as a single expectation under measure PEG:
V (D,E,G) = E(x,z)∼PEX [logD(x, z)] + E(x,z)∼PGZ [log (1−D(x, z))] = E(x,z)∼PEG [2fEG︸ ︷︷ ︸
dPEX dPEG (x, z) logD(x, z)] + E(x,z)∼PEG [2fGE︸ ︷︷ ︸ dPGZ dPEG (x, z) log (1−D(x, z))]
= 2E(x,z)∼PEG [fEG(x, z) logD(x, z) + fGE(x, z) log (1−D(x, z))] = 2E(x,z)∼PEG [fEG(x, z) logD(x, z) + (1− fEG(x, z)) log (1−D(x, z))] .
Note that arg maxy {a log y + (1− a) log(1− y)} = a for any a ∈ [0, 1]. Thus, D∗EG = fEG.
A.2 PROOF OF PROPOSITION 2 (ENCODER AND GENERATOR OBJECTIVE)
Proposition 2 The encoder and generator’s objective for an optimal discriminator C(E,G) := maxD V (D,E,G) = V (D ∗ EG, E,G) can be rewritten in terms of the Jensen-Shannon divergence between measures PEX and PGZ as C(E,G) = 2 DJS (PEX ||PGZ )− log 4.
Proof. Using Proposition 1 along with (5) (1−D∗EG = 1− fEG = fGE) we rewrite the objective
C(E,G) = maxDV (D,E,G) = V (D ∗ EG, E,G)
= E(x,z)∼PEX [logD ∗ EG(x, z)] + E(x,z)∼PGZ [log (1−D ∗ EG(x, z))]
= E(x,z)∼PEX [log fEG(x, z)] + E(x,z)∼PGZ [log fGE(x, z)] = E(x,z)∼PEX [log (2fEG(x, z))] + E(x,z)∼PGZ [log (2fGE(x, z))]− log 4 = DKL (PEX ||PEG ) + DKL (PGZ ||PEG )− log 4 = DKL ( PEX ∣∣∣∣ PEX+PGZ 2 ) + DKL ( PGZ ∣∣∣∣ PEX+PGZ 2 ) − log 4 = 2 DJS (PEX ||PGZ )− log 4.
A.3 MEASURE DEFINITIONS FOR DETERMINISTIC E AND G
While Theorem 1 and Propositions 1 and 2 hold for any encoder pE(z|x) and generator pG(x|z), stochastic or deterministic, Theorems 2 and 3 assume the encoderE and generatorG are deterministic functions; i.e., with conditionals pE(z|x) = δ(z− E(x)) and pG(x|z) = δ(x−G(z)) defined as δ functions.
For use in the proofs of those theorems, we simplify the definitions of measures PEX and PGZ given in Section 3 for the case of deterministic functions E and G below:
PEX(R) = ∫
ΩX pX(x) ∫ ΩZ pE(z|x)1[(x,z)∈R] dzdx
= ∫
ΩX pX(x) (∫ ΩZ δ(z− E(x))1[(x,z)∈R] dz ) dx
= ∫
ΩX pX(x)1[(x,E(x))∈R] dx PGZ(R) = ∫
ΩZ pZ(z) ∫ ΩX pG(x|z)1[(x,z)∈R] dx dz
= ∫
ΩZ pZ(z) (∫ ΩX δ(x−G(z))1[(x,z)∈R] dx ) dz
= ∫
ΩZ pZ(z)1[(G(z),z)∈R] dz
A.4 PROOF OF THEOREM 2 (OPTIMAL GENERATOR AND ENCODER ARE INVERSES)
Theorem 2 If E and G are an optimal encoder and generator, then E = G−1 almost everywhere; that is, G(E(x)) = x for PX-almost every x ∈ ΩX, and E(G(z)) = z for PZ-almost every z ∈ ΩZ.
Proof. Let R0X := {x ∈ ΩX : x 6= G(E(x))} be the region of ΩX in which the inversion property x = G(E(x)) does not hold. We will show that, for optimal E and G, R0X has measure zero under PX (i.e., PX(R0X) = 0) and therefore x = G(E(x)) holds PX-almost everywhere.
Let R0 := {(x, z) ∈ Ω : z = E(x) ∧ x ∈ R0X} be the region of Ω such that (x, E(x)) ∈ R0 if and only if x ∈ R0X. We’ll use the definitions of PEX and PGZ for deterministicE andG (Appendix A.3), and the fact that PEX = PGZ for optimal E and G (Theorem 1).
PX(R 0 X) = ∫ ΩX pX(x)1[x∈R0X] dx
= ∫
ΩX pX(x)1[(x,E(x))∈R0] dx
= PEX(R 0)
= PGZ(R 0) = ∫
ΩZ pZ(z)1[(G(z),z)∈R0] dz = ∫
ΩZ pZ(z)1[z=E(G(z))∧G(z)∈R0X] dz = ∫
ΩZ pZ(z) 1[z=E(G(z))∧G(z)6=G(E(G(z)))]︸ ︷︷ ︸
=0 for any z, as z=E(G(z)) =⇒ G(z)=G(E(G(z)))
dz
= 0.
Hence region R0X has measure zero (PX(R 0 X) = 0), and the inversion property x = G(E(x)) holds PX-almost everywhere.
An analogous argument shows that R0Z := {z ∈ ΩZ : z 6= E(G(z))} has measure zero on PZ (i.e., PZ(R 0 Z) = 0) and therefore z = E(G(z)) holds PZ-almost everywhere.
A.5 PROOF OF THEOREM 3 (RELATIONSHIP TO AUTOENCODERS)
As shown in Proposition 2 (Section 3), the BiGAN objective is equivalent to the Jensen-Shannon divergence between PEX and PGZ. We now go a step further and show that this Jensen-Shannon divergence is closely related to a standard autoencoder loss. Omitting the 12 scale factor, a KL divergence term of the Jensen-Shannon divergence is given as
DKL ( PEX ∣∣∣∣ PEX+PGZ 2 ) = log 2 + ∫ Ω log dPEX d(PEX + PGZ) dPEX
= log 2 + ∫ Ω log f dPEX, (6)
where we abbreviate as f the Radon-Nikodym derivative fEG := dPEXd(PEX+PGZ) ∈ [0, 1] defined in Proposition 1 for most of this proof.
We’ll make use of the definitions of PEX and PGZ for deterministic E and G found in Appendix A.3. The integral term of the KL divergence expression given in (6) over a particular region R ⊆ Ω will be denoted by
F (R) := ∫ R log dPEX d (PEX + PGZ) dPEX = ∫ R log f dPEX.
Next we will show that f > 0 holds PEX-almost everywhere, and hence F is always well defined and finite. We then show that F is equivalent to an autoencoder-like reconstruction loss function.
Proposition 3 f > 0 PEX-almost everywhere.
Proof. Let Rf=0 := {(x, z) ∈ Ω : f(x, z) = 0} be the region of Ω in which f = 0. Using the definition of the Radon-Nikodym derivative f , the measure PEX(Rf=0) = ∫ Rf=0 f d(PEX +
PGZ) = ∫ Rf=0 0 d(PEX + PGZ) = 0 is zero. Hence f > 0 PEX-almost everywhere.
Proposition 3 ensures that log f is defined PEX-almost everywhere, and F (R) is well-defined. Next we will show that F (R) mimics an autoencoder with `0 loss, meaning F is zero for any region in which G(E(x)) 6= x, and non-zero otherwise.
Proposition 4 The KL divergence F outside the support of PGZ is zero: F (Ω \ supp(PGZ)) = 0.
We’ll first show that in region RS := Ω \ supp(PGZ), we have f = 1 PEX-almost everywhere. Let Rf<1 := {(x, z) ∈ RS : f(x, z) < 1} be the region of RS in which f < 1. Let’s assume that PEX(R f<1) > 0 has non-zero measure. Then, using the definition of the Radon-Nikodym derivative,
PEX(R f<1) = ∫ Rf<1 f d(PEX + PGZ) = ∫ Rf<1
f︸︷︷︸ ≤ε<1
dPEX + ∫ Rf<1
f dPGZ︸ ︷︷ ︸ 0 ≤ εPEX(Rf<1)
< PEX(R f<1),
where ε is a constant smaller than 1. But PEX(Rf<1) < PEX(Rf<1) is a contradiction; hence PEX(R
f<1) = 0 and f = 1 PEX-almost everywhere in RS , implying log f = 0 PEX-almost everywhere in RS . Hence F (RS) = 0.
By definition, F (Ω \ supp(PEX)) = 0 is also zero. The only region where F might be non-zero is R1 := supp(PEX) ∩ supp(PGZ).
Proposition 5 f < 1 PEX-almost everywhere in R1.
Let Rf=1 := { (x, z) ∈ R1 : f(x, z) = 1 }
be the region in which f = 1. Let’s assume the set Rf=1 6= ∅ is not empty. By definition of the support1, PEX(Rf=1) > 0 and PGZ(Rf=1) > 0. The Radon-Nikodym derivative on Rf=1 is then given by
PEX(R f=1) = ∫ Rf=1 f d(PEX + PGZ) = ∫ Rf=1 1 d(PEX + PGZ) = PEX(R f=1) + PGZ(R f=1),
which implies PGZ(Rf=1) = 0 and contradicts the definition of support. Hence Rf=1 = ∅ and f < 1 PEX-almost everywhere on R1, implying log f < 0 PEX-almost everywhere.
Theorem 3 The encoder and generator objective given an optimal discriminator C(E,G) := maxD V (D,E,G) can be rewritten as an `0 autoencoder loss function
C(E,G) = Ex∼pX [ 1[E(x)∈Ω̂Z∧G(E(x))=x] log fEG(x, E(x)) ] +
Ez∼pZ [ 1[G(z)∈Ω̂X∧E(G(z))=z] log (1− fEG(G(z), z)) ] with log fEG ∈ (−∞, 0) and log (1− fEG) ∈ (−∞, 0) PEX-almost and PGZ-almost everywhere.
Proof. Proposition 4 (F (Ω \ supp(PGZ)) = 0) and F (Ω \ supp(PEX)) = 0 imply that R1 := supp(PEX) ∩ supp(PGZ) is the only region of Ω where F may be non-zero; hence F (Ω) = F (R1).
1We use the definition U ∩ C 6= ∅ =⇒ µ(U ∩ C) > 0 here.
Note that
supp(PEX) = {(x, E(x)) : x ∈ Ω̂X} supp(PGZ) = {(G(z), z) : z ∈ Ω̂Z}
=⇒ R1 := supp(PEX) ∩ supp(PGZ) = {(x, z) : E(x) = z ∧ x ∈ Ω̂X ∧G(z) = x ∧ z ∈ Ω̂Z}
So a point (x, E(x)) is in R1 if x ∈ Ω̂X, E(x) ∈ Ω̂Z, and G(E(x)) = x. (We can omit the x ∈ Ω̂X condition from inside an expectation over PX, as PX-almost all x /∈ Ω̂X have 0 probability.) Therefore,
DKL ( PEX ∣∣∣∣ PEX+PGZ 2 ) − log 2 = F (Ω) = F (R1)
= ∫ R1 log f(x, z) dPEX
= ∫
Ω 1[(x,z)∈R1] log f(x, z) dPEX = E(x,z)∼PEX [ 1[(x,z)∈R1] log f(x, z) ] = Ex∼pX [ 1[(x,E(x))∈R1] log f(x, E(x))
] = Ex∼pX [ 1[E(x)∈Ω̂Z∧G(E(x))=x] log f(x, E(x)) ] .
Finally, with Propositions 3 and 5, we have f ∈ (0, 1) PEX-almost everywhere in R1, and therefore log f ∈ (−∞, 0), taking a finite and strictly negative value PEX-almost everywhere. An analogous argument (along with the fact that fEG + fGE = 1) lets us rewrite the other KL divergence term
DKL ( PGZ ∣∣∣∣ PEX+PGZ 2 ) − log 2 = Ez∼pZ [ 1[G(z)∈Ω̂X∧E(G(z))=z] log fGE(G(z), z) ] = Ez∼pZ [ 1[G(z)∈Ω̂X∧E(G(z))=z] log (1− fEG(G(z), z))
] The Jensen-Shannon divergence is the mean of these two KL divergences, giving C(E,G):
C(E,G) = 2 DJS (PEX ||PGZ )− log 4 = DKL ( PEX ∣∣∣∣ PEX+PGZ 2 ) + DKL ( PGZ ∣∣∣∣ PEX+PGZ 2 ) − log 4
= Ex∼pX [ 1[E(x)∈Ω̂Z∧G(E(x))=x] log fEG(x, E(x)) ] +
Ez∼pZ [ 1[G(z)∈Ω̂X∧E(G(z))=z] log (1− fEG(G(z), z)) ]
APPENDIX B LEARNING DETAILS
In this section we provide additional details on the BiGAN learning protocol summarized in Section 3.4. Goodfellow et al. (2014) found for GAN training that an objective in which the real and generated labels Y are swapped provides stronger gradient signal to G. We similarly observed in BiGAN training that an “inverse” objective Λ (with the same fixed point characteristics as V ) provides stronger gradient signal to G and E, where
Λ(D,G,E) = Ex∼pX [ Ez∼pE(·|x) [log (1−D(x, z))]︸ ︷︷ ︸
log(1−D(x,E(x)))
] + Ez∼pZ [ Ex∼pG(·|z) [logD(x, z)]︸ ︷︷ ︸
logD(G(z),z)
] .
In practice, θG and θE are updated by moving in the positive gradient direction of this inverse objective ∇θE ,θGΛ, rather than the negative gradient direction of the original objective. We also observed that learning behaved similarly when all parameters θD, θG, θE were updated simultaneously at each iteration rather than alternating between θD updates and θG, θE updates, so we took the simultaneous updating (non-alternating) approach for computational efficiency. (For standard GAN training, simultaneous updates of θD, θG performed similarly well, so our standard GAN experiments also follow this protocol.)
APPENDIX C MODEL AND TRAINING DETAILS
In the following sections we present additional details on the models and training protocols used in the permutation-invariant MNIST and ImageNet evaluations presented in Section 4.
Optimization For unsupervised training of BiGANs and baseline methods, we use the Adam optimizer (Kingma & Ba, 2015) to compute parameter updates, following the hyperparameters (initial step size α = 2 × 10−4, momentum β1 = 0.5 and β2 = 0.999) used by Radford et al. (2016). The step size α is decayed exponentially to α = 2 × 10−6 starting halfway through training. The mini-batch size is 128. `2 weight decay of 2.5 × 10−5 is applied to all multiplicative weights in linear layers (but not to the learned bias β or scale γ parameters applied after batch normalization). Weights are initialized from a zero-mean normal distribution with a standard deviation of 0.02, with one notable exception: BiGAN discriminator weights that directly multiply z inputs to be added to spatial convolution outputs have initializations scaled by the convolution kernel size – e.g., for a 5× 5 kernel, weights are initialized with a standard deviation of 0.5, 25 times the standard initialization.
Software & hardware We implement BiGANs and baseline feature learning methods using the Theano (Theano Development Team, 2016) framework, based on the convolutional GAN implementation provided by Radford et al. (2016). ImageNet transfer learning experiments (Section 4.3) use the Caffe (Jia et al., 2014) framework, per the Fast R-CNN (Girshick, 2015) and FCN (Long et al., 2015) reference implementations. Most computation is performed on an NVIDIA Titan X or Tesla K40 GPU.
C.1 PERMUTATION-INVARIANT MNIST
In all permutation-invariant MNIST experiments (Section 4.2), D, G, and E each consist of two hidden layers with 1024 units. The first hidden layer is followed by a non-linearity; the second is followed by (parameter-free) batch normalization (Ioffe & Szegedy, 2015) and a non-linearity. The second hidden layer in each case is the input to a linear prediction layer of the appropriate size. In D and E, a leaky ReLU (Maas et al., 2013) non-linearity with a “leak” of 0.2 is used; in G, a standard ReLU non-linearity is used. All models are trained for 400 epochs.
C.2 IMAGENET
In all ImageNet experiments (Section 4.3), the encoder E architecture follows AlexNet (Krizhevsky et al., 2012) through the fifth and last convolution layer (conv5), with local response normalization (LRN) layers removed and batch normalization (Ioffe & Szegedy, 2015) (including the learned scaling and bias) with leaky ReLU non-linearity applied to the output of each convolution at unsupervised training time. (For supervised evaluation, batch normalization is not used, and the pre-trained scale and bias is merged into the preceding convolution’s weights and bias.)
In most experiments, both the discriminatorD and generatorG architecture are those used by Radford et al. (2016), consisting of a series of four 5× 5 convolutions (or “deconvolutions” – fractionallystrided convolutions – for the generator G) applied with 2 pixel stride, each followed by batch normalization and rectified non-linearity.
The sole exception is our discriminator baseline feature learning experiment, in which we let the discriminatorD be the AlexNet variant described above. Generally, using AlexNet (or similar convnet architecture) as the discriminator D is detrimental to the visual fidelity of the resulting generated images, likely due to the relatively large convolutional filter kernel size applied to the input image, as well as the max-pooling layers, which explicitly discard information in the input. However, for fair comparison of the discriminator’s feature learning abilities with those of BiGANs, we use the same architecture as used in the BiGAN encoder.
Preprocessing To produce a data sample x, we first sample an image from the database, and resize it proportionally such that its shorter edge has a length of 72 pixels. Then, a 64× 64 crop is randomly selected from the resized image. The crop is flipped horizontally with probability 12 . Finally, the crop is scaled to [−1, 1], giving the sample x.
Timing A single epoch (one training pass over the 1.2 million images) of BiGAN training takes roughly 40 minutes on a Titan X GPU. Models are trained for 100 epochs, for a total training time of under 3 days.
Nearest neighbors In Figure 5 we present nearest neighbors in the feature space of the BiGAN encoder E learned in unsupervised ImageNet training.

The authors extend GANs by an inference path from the data space to the latent space and a discriminator that operates on the joint latend/data space. They show that the theoretical properties of GANs still hold for BiGAN and evaluate the features learned unsupervised in the inference path with respect to performance on supervised tasks after retraining deeper layers.
I see one structural issue with this paper: Given that, as stated in the abstract, the main purpose of the paper is to learn unsupervised features (and not to improve GANs), the paper might spent too much space on detailing the relationship to GANs and all the theoretical properties. It is not clear whether they actually would help with the goal of learning good features. While reading the paper, I actually totally forgot about the unsupervised features until they reappeared on page 6. I think it would be helpful if the text of the paper would be more aligned with this main story.
Still, the BiGAN framework is an elegant and compelling extension to GANs. However, it is not obvious how much the theoretical properties help us as the model is clearly not fully converged. To me, especially Figure 4 seems to suggest that G(E(x)) might be doing not much more than some kind of nearest neighbour retrival (and indeed one criticism for GANs has always been that they might just memorize some samples). By the way, it would be very interesting to know how well the discriminator actually performs after training.
Coming back to the goal of learning powerful features: The method does not reach state-of-the-art performance on most evaluated tasks (Table 2 and 3) but performs competitive and it would be interesting to see how much this improves if the BiGAN training (and the convolutional architecture used) would be improved.
The paper is very well written and provides most necessary details, although some more details on the training (learning rates, initialization) would be helpful for reproducing the results.
Overall I think the paper provides a very interesting framework for further research, even though the results presented here are not too impressive both with respect to the feature evaluation (and the GAN learning).
Minor: It might be helpful to highlight the best performance numbers in Tables 2 and 3.