Learning discrete representations of data and then generating data from the discovered representations have been increasingly studied because the obtained discrete representations can benefit unsupervised learning. However, the performance of learning discrete representations of textual data with deep generative models has not been widely explored. In addition, although generative adversarial networks(GAN) have shown impressing results in many areas such as image generation, for text generation, it is notorious for extremely difficult to train. In this work, we propose TopicGAN, a two-step text generative model, which is able to solve those two important problems simultaneously. In the first step, it discovers the latent topics and produced bag-of-words according to the latent topics. In the second step, it generates text from the produced bag-of-words. In our experiments, we show our model can discover meaningful discrete latent topics of texts in an unsupervised fashion and generate high quality natural language from the discovered latent topics.
This paper proposes TopicGAN, a generative adversarial approach to topic modeling and text generation. The model basically combines two steps: first to generate words (bag-of-words) for a topic, then second to generate the sequence of the words.
While the idea is interesting, there are several important limitations. First, the paper is difficult to understand, and some of the explanations are not convincing. For example, in section 4.1.1, it says "... our method assumes that the documents are produced from a single topic ... Our assumption aligns well with human intuition that most documents are generated from a single main topic." This goes very much against the common assumption of a generative topic model, such as LDA, which the model compares against. I don't mean to argue either way, but if the paper presents a viewpoint which is quite different from the commonly accepted viewpoint (within the specific research field), then there needs to be a much deeper explanation, ideally with concrete evidence to support it. Another sentence from the same paragraph states that their "model outperforms LDA because LDA is a statistical model, while our generator is a deep generative model." This argument also seems flawed and without concrete evidence. There are other parts in the paper where the logic seems strange and without evidence, and they make it difficult to understand and accept the major claims of the paper.
Second, the model does not offer much novelty. It seems that the two-stage model simply puts the two pieces, a GAN-style generator and an LSTM sequence model together. Perhaps I am not understanding the model, but the model description was also not clear nor easy to understand with respect to its novelty.
Third, the evaluation is somewhat weak. There are two main evaluations tasks: text classification and text generation. For the first task, classification is not the main purpose of topic models, and while text classification _is_ used in many topic modeling papers, it is almost always accompanied by other evaluation metrics such as held-out perplexity and topic coherence. This is because the main purpose of topic modeling is to actually infer the topics (per-topic word distribution and per-document topic distribution) and model the corpus. Thus I feel it is not a fair evaluation to just compare the models using text classification tasks. The second evaluation task of text generation is not explained enough. For the human evaluation, who were the annotators, and how were they trained? How many people annotated each output, and what was the inter-rater agreement? How many sentences were evaluated, and how were they chosen? Without these details, it is difficult to judge whether this evaluation was valid.
Lastly, the results are mediocre. Besides the classification task, the others do not show significant improvements over the baseline models. Perplexity (table 3) shows similar results for DBPedia and worse results (than WGAN-gp) for Gigaword. Table 4 shows slightly better results for "Preference" for TopicGAN with joint training, but "Accuracy" is measured only for the proposed model and not the baseline model.