Generative adversarial nets (GANs) have been successfully applied to the artificial generation of image data. In terms of text data, much has been done on the artificial generation of natural language from a single corpus. We consider multiple text corpora as the input data, for which there can be two applications of GANs: (1) the creation of consistent cross-corpus word embeddings given different word embeddings per corpus; (2) the generation of robust bag-of-words document embeddings for each corpora. We demonstrate our GAN models on real-world text data sets from different corpora, and show that embeddings from both models lead to improvements in supervised learning problems.
The paper proposes to use Generative Adversarial Networks (GANs) in the context of natural language processing and introduces two models for generating document embeddings. The first model, weGAN, aggregates multiple sets of single-corpus word2vec embeddings into one set of cross-corpus word representations; document embeddings are a function of these updated word embeddings. The second model, deGAN, side-steps word-level embeddings and directly generates document-level representations. For both models, the real examples come from word2vec and tf-idf, while the artificial examples are the ones generated by the network. The authors show that their document embeddings are better than the word2vec/tf-idf baseline at clustering documents according to the corpus they originate from.
While the context in which GANs are used is novel and creative, the reasons for rejection outweigh the positives. The main reason is the fact that the paper is outdated by two/three years. There is no mention of the shift towards contextual embeddings that has been happening in the last few years (InferSent, ELMo, BERT) and that became ubiquitous in NLP.  Instead, all comparisons are against word2vec (published in 2013) that is no longer a relevant baseline, even for non-contextual word embeddings. In fact, the newest cited papers are from 2017. The Github link in the paper was last updated on 23rd December 2017. My objection is not that this work was started a long time ago, but that it has not been updated in years. In the lack of proper comparison against more novel techniques, it is hard to estimate whether the reported gains over word2vec are meaningful.
Another argument for rejection is the fact that the experimental section does not make a convincing case that the positive results matter beyond the very specific datasets and tasks selected.
Before commenting in more detail on the selection of datasets, tasks and metrics, I want to document my understanding of the experimental setup; I found it non-trivial to understand what constitutes a "single corpus" in the experiments. My mental model is the following: the paper operates on four datasets (CNN, TIME, 20 Newsgroups, and Reuters-21578). These are four different experiments, and the four datasets never interact. The cross-corpus property of the model is exercised within each of these datasets, which have a natural grouping of topics; for instance, CNN is divided into "US", "world", and "politics". In other words, a cross-corpus CNN model is trained on three corpora: US, world, and politics. With this mental model of the experimental setup, the experiments raise a series of questions:
1) Do the selected datasets / classification tasks matter in the grad scheme of NLP?
The end-task is to classify the corpus of origin for each document (for instance, for the CNN experiment, the model has to decide between US, world, and politics). If I understand correctly, the weGAN model jointly trained its word embeddings with this objective. As a final step, the classifier was additionally fine-tuned on this task. So it is not very surprising that weGAN is better at clustering documents than word2vec, since it was explicitly trained to do so. It would have been more informative to see weGAN performing better on a task that was solely fine-tuned on (e.g. some sentiment classification task that was not part of the training objective when embeddings were trained). In that case, one could argue that the enhanced embeddings can be used as an out-of-the box tool for a lot of different tasks. Another option would be evaluating on the same document membership task, but on datasets that were not seen during training.
Additionally, the fact that the authors evaluated on *new* datasets with a *new* task distances them even further from any potential comparisons against prior work. Even for the two already-existing datasets with results included in the appendix, there are no references to previous SoTA.
2) Are the reported positive results meaningful?
Another problem with not showing results from previous work is that it is hard to understand whether the gains over the word2vec baseline are meaningful. For instance, the classification accuracy for the CNN corpus is increased from 92.05% (word2vec) to 92.29% (deGAN) and 92.36% (weGAN). The authors show that this is *statistically significant*, but is it meaningful? Is a 0.24% or 0.31% increase in accuracy worth the increased complexity of the model? For reference, what ratio of gains/computational cost would BERT incur?
3) What is the message of qualitative evaluation?
The authors offer qualitative analysis of their model. For instance, Table 2 shows the top 10 synonyms for words "Obama", "Trump" and "US", as produced by word2vec and weGAN. The lists of synonyms are extremely similar and don't seem to convey much more information besides the fact that weGAN makes minimal changes. I find the particular examples that the authors choose to underline the superiority of their algorithm unconvincing. For instance, the fact that, in the list of synonyms for US, the word British (selected by word2vec as the 9th closest word) is turned into American (on the same 9th position) loses its significance when you also consider the fact that weGAN did not derank any of the arguably less similar terms from higher positions (e.g. Turkish remains the 6th most similar term to US).
Other nit-picks:
- Formatting issues: In Sections 3.1 and 3.2, the numerical references to equations are broken. Figures 3 and 4 cover the text above. Tables 2, 4, 6, 7 are hard to read. It almost feels like the paper was updated from one conference format to another, without the necessary Latex adjustments.
- The paper has no "Conclusion" section
- Improper use of singular / plural of "corpus" / "corpora"
To end on a more positive note, here are some reasons why I did appreciate the paper:
- The usage of GANs in this context is very creative; the models do not generate *raw* text, but rather intermediate representations; this might be a promising direction for using GANs in NLP.
- The description of the models in sections 3.1 and 3.2 is rigorous, with useful explanations that break down the reasoning behind the loss function choices.