In this paper, we propose a novel adversarial learning framework, namely Delib-GAN, for generating high-quality sentences without supervision. Our framework consists of a coarse-to-fine generator, which contains a first-pass decoder and a second-pass decoder, and a multiple instance discriminator. And we propose two training mechanisms DelibGAN-I and DelibGAN-II. The discriminator is used to fine-tune the second-pass decoder in DelibGAN-I and further evaluate the importance of each word and tune the first-pass decoder in DelibGAN-II. We compare our models with several typical and state-of-the-art unsupervised generic text generation models on three datasets (a synthetic dataset, a descriptive text dataset and a sentimental text dataset). Both qualitative and quantitative experimental results show that our models produce more realistic samples, and DelibGAN-II performs best.
Summary:
The authors propose a novel adversarial learning framework consisting of a coarse-to-fine generator and a multiple instance discriminator to generate high-quality sentences without supervision, along with two training mechanisms to make the generator produce higher-quality sentences.
The coarse-to-fine generator is implemented as a two-stage decoder where the first stage decoder is initialized with a noise vector z and produces an initial sequence by sampling from its distribution at each step. The second stage decoder is able to attend to first stage decoder hidden states when generating its sequence and is initialized with the last hidden state of the first state decoder
Two training frameworks are defined to help the coarse-to-fine generator learn to produce coherent text. In both frameworks, a multi-instance learner is used as a discriminator and the overall score for the sequence is an average of word-level scores. 
In DelibGAN-I, the first stage decoder is just trained to minimize the negative log likelihood of producing the sampled sequence. I’m not sure I understand how G1 learns to generate coherent text by just maximizing the probability of its samples.
In DelibGAN-2, the word-level scores from passing the first stage decoder’s output through the discriminator is used to score the sequence. This makes more sense to me w.r.t to why the first stage generator could learn to produce coherent text.
Review:
The writing could be clearer. Spacing is consistently an issue in w.r.t. to citations, referencing equations an figures, and using punctuations. Equations are difficult to understand, as well. See Equation 5. I’m confused by how the discriminator score of x_t is computed in equation 5. It seems to be the score of the multi-instance learner when applied to the concatenation of the previous generated tokens and the most recently generated token. This isn’t really a roll-out policy, however, since only a single step is taken in the future. It’s just scoring the next token according to the discriminator. In this case, I’m not sure what the summation over j is supposed to represent. It seems to index different histories when scoring word x_t. The authors should clarify exactly what the scoring procedure is for sequences generated by G2 and make sure that clarification matches what is written in the equation.
Because deliberation has previous been explored in Xia et al., 2017, the novelty of this work rests on the application of deliberation to GAN training of generative text models, as well as the development of the multi-instance discriminator for assigning word-level scores to the first-stage decoder's actions. However, it’s difficult to know how much of the improvement in performance is due to the modeling improvement because the evaluations are missing key details.
First, no information about the model’s hyperparameter setting were provided. Naturally, having two generators would double the number of parameters of the total model. Was an ablation run that looked at doubling the parameters of a single-generator? How powerful are the models being used in this new setup. With powerful models such as the OpenAI GPT being available for generative modeling, how powerful are the base units in this framework?
Second, I don’t have much intuition for how difficult the tasks being performed are. Because the authors aren’t evaluating on particularly difficult language datasets, they should provide statistics about the datasets w.r.t to vocabulary size, average length, etc.
Consequently, a lot of information that is necessary for evaluating the strengths and weaknesses of this paper are missing from the write-up.
Questions:
Equation 2 is slightly confusing. Is the representation of the word at that time step not conditioned on previous and future hidden states?
Is the multi-instance discriminator used in DelibGAN-II or is G2 only scored using the sentence level discriminator score?
Small:
-Wang & Wan, 2018a are definitely not the first work to discover that using word-level losses to train models for sentence-level evaluations is suboptimal
-Please fix issues with spacing. There are issues when citing papers, or referencing equations, or using punctuation.
-Equation 3 is missing a closing parenthesis on the softmax