  Recently, contrastive learning attracts increasing interests in neural text
generation as a new solution to alleviate the exposure bias problem. It
introduces a sequence-level training signal which is crucial to generation
tasks that always rely on auto-regressive decoding. However, previous methods
using contrastive learning in neural text generation usually lead to inferior
performance. In this paper, we analyse the underlying reasons and propose a new
Contrastive Neural Text generation framework, CoNT. CoNT addresses bottlenecks
that prevent contrastive learning from being widely adopted in generation tasks
from three aspects -- the construction of contrastive examples, the choice of
the contrastive loss, and the strategy in decoding. We validate CoNT on five
generation tasks with ten benchmarks, including machine translation,
summarization, code comment generation, data-to-text generation and commonsense
generation. Experimental results show that CoNT clearly outperforms the
conventional training framework on all the ten benchmarks with a convincing
margin. Especially, CoNT surpasses previous the most competitive contrastive
learning method for text generation, by 1.50 BLEU on machine translation and
1.77 ROUGE-1 on summarization, respectively. It achieves new state-of-the-art
on summarization, code comment generation (without external data) and
data-to-text generation.

Computation and Language