Generating coherent and cohesive long-form texts is a challenging problem in natural language generation. Previous works relied on a large amount of humangenerated texts to train neural language models, however, few attempted to explicitly model the desired linguistic properties of natural language text, such as coherence and cohesion using neural networks. In this work, we train two expert discriminators for coherence and cohesion to provide hierarchical feedback for text generation. We also propose a simple variant of policy gradient, called negative-critical sequence training in which the reward baseline is constructed from randomly generated negative samples. We demonstrate the effectiveness of our approach through empirical studies, showing improvements over the strong baseline -attention-based bidirectional MLE-trained neural language model -in a number of automated metrics. The proposed model can serve as baseline architectures to promote further research in modeling additional linguistic properties for downstream NLP tasks.
The paper proposes a method for improving the quality of text generation by optimizing for coherence and cohesion. The authors develop two discriminators--a "coherence discriminator" which takes as input all of the sentence embeddings (i.e. averaged word embeddings) of the document and assigns a score, and a "cohesion discriminator" which takes as input the word embeddings of two consecutive sentences and assigns a score. In the former, the score is the cosine similarity between the encodings of the first and second half of the document. In the latter, the score is the cosine similarity between the encodings of the two sentences. Both discriminators use CNNs to encode the inputs. The discriminators are trained to rank true text over randomly drawn negative samples, which consist of randomly permuted sentence orderings and/or random combinations of first/second half of documents. This discriminators are then used to train a text generation model. The output of the text generation model is scored by various automatic metrics, including NLL, PPL, BLEU, and number of unique ngrams in the outputs. The improvements over a generically-trained generation model are very small.
Overall, I did not find this paper to be convincing. The initial motivation is good--we need to find a way to capture richer linguistic properties of text and to encourage NLG to produce such properties. However, the discriminators presented do not actually capture the nuances that they purport to capture. As I understand it, these models are just being trained to incentivize high cosine similarity between the words in the first/second half of a document (or sentence/following sentence). That is not reflective of the definitions of coherence and cohesion, which should reflect deeper discourse and even syntactic structure. Rather, these are just models which capture topical similarity, and naively at that. Moreover, training this model to discriminate real text from randomly perturbed text seems problematic since 1) randomly shuffled text should be trivially easy to distinguish from real text in terms of topical similarity and 2) these negative samples are not (I don't think) at all reflective of the types of texts that the discriminators actually need to discriminate, i.e. automatically generated texts. Thus, even ignoring the fact that I disagree with the authors on exactly what the discriminators are/should be doing, it is still not clear to me that the discriminators are well trained to do the thing the authors want them to do. I have various other concerns about the claims, the approach, and the evaluation. A list of more specific questions/comments for the authors is below.
- There are a *lot* of unsubstantiated claims and speculation about the linguistic properties that these discriminators capture, and no motivation of analysis as to how they are capturing it. Claims like the following definitely need to be removed: "learn to inspect the higher-level role of T, such as but not limited to, whether it supports the intent of S, transitions smoothly against S, or avoids redundancy", "such as grammar of each of the sentences and the logical flow between arbitrary two consecutive sentences"
- You only use automated metrics, despite acknowledging that there is no good way to evaluate generation. Why not use human eval? This is not difficult to carry out, and when you are arguing about such subtle properties of language, human eval is essential. There is no reason that BLEU, for example, would be sensitive to coherence or cohesion, so why would this be a good way to evaluate a model aimed to capture exactly those things?
- Also related to human eval, there should be an intrinsic evaluation of the discriminators. Do they correlate with human judgments of coherence and cohesion? You cannot take it for granted that they capture these things (I very much believe they do not), so present some evidence that the models do what you claim they do.
- The reported improvements are minuscule, to the extent that I would read them as "no difference". The only metric where there is a real difference is on number of unique ngrams generated cross inputs, which is presumably because its just learning (being encouraged to) spit out words that were in the input. I'd like to see the baseline of just copying the input as the output.
- You mention several times that these models will pick up on redundancy. It is not clear to me how they could do that. Aren't they simply using a cosine similarity between feature vectors? Perhaps I am missing something, but I don't see how this could learn to disincentivize redundancy but simultaneously encourage topical similarity. Could you explain this claim?