Large pretrained language models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that adapt a pretrained model for arbitrary downstream tasks. However, it is an open question how to use similar techniques for language generation. Early results in the encoder-agnostic setting have been mostly negative. In this work, we explore methods for adapting a pretrained language model to arbitrary conditional input. We observe that pretrained transformer models are sensitive to large parameter changes during tuning. Therefore, we propose an adaptation that directly injects arbitrary conditioning into self attention, an approach we call pseudo self attention. Through experiments on four diverse conditional text generation tasks, we show that this encoder-agnostic technique outperforms strong baselines, produces coherent generations, and is data-efficient.
This paper proposes a simple yet effective method to adapt large-scale pre-trained language models, which have been shown to substantially improve performance on broadly classification-based NLU tasks, to NLG. The approach is explored in the encoder-agnostic {X}-to-text setup, where the source encoding {X} could represent arbitrary modalities, such as text or images.
More concretely, the paper leverages a pre-trained, large-scale language model (in this case a GPT-2), and examines how to best cast such unconditional language model into a decoder that generates text conditional on the source information {X}. As self-attention inherently works with sequences of any length, the proposed pseudo self-attention approach simply injects the encoder representation as additional conditioning context (using some additional projection matrices that are learned from scratch) into the pre-trained self-attention layers of the decoder. Extensive experiments and analysis on four diverse tasks demonstrate that pseudo self-attention generally outperforms two other ways of pre-training the decoder, improves NLG data efficiency, and produces texts that are judged more favourably by human evaluators.
Overall, this paper presents a simple, general, and effective method for adapting large-scale pre-trained language models to conditional text generation. Based on the pros and cons that I have listed below, I am giving a rating of "Accept". I hope that some of my concerns will be addressed in the authors' response.
Pros:
1. The paper is well-written and the methodology is explained very clearly. Figure 1 is particularly helpful in illustrating the differences between pseudo self-attention and the baselines.
2. The paper addresses a very important problem, and helps make sure that the advances that have been made in language modelling (which can leverage large amounts of unlabelled data), would transfer well to conditional text generation tasks, which hold immediate practical value yet often require expensive annotations.
3. The approach is simple and easy-to-implement, but has been shown to be effective across a broad range of problems, multiple modalities, and various evaluation metric. 
4. The paper features extensive reference to relevant prior work, and clearly highlights the key similarities and differences with prior approaches. 
Cons:
1. It is still unclear how using language model pre-training affects adequacy (as opposed to fluency). The paper shows that using pseudo self-attention results in a decoder that diverges less from its language model initialisation. One potential risk is that the decoder may prefer fluent, "safe" outputs (which is arguably what a language model would prefer since it is an unconditional model) that are nevertheless less faithful to the source information. Since none of the evaluation metric specifically assesses for adequacy on its own, it would be good to isolate the effect of pseudo self-attention on adequacy, and compare it with the baselines, in addition to a Transformer trained from scratch on each downstream task. How to measure adequacy is naturally still an open question, but there are a few things that can be done (e.g. recall of salient information, reverse perplexity to see how much of the source information can be "reconstructed" given the predicted target text, etc.).
2. It would be interesting to further examine the interaction between encoder pre-training and the decoder pre-training that is explored in this work. Another interesting experiment to run is whether end-to-end training (including fine-tuning the encoder) would help, since prior work has shown the benefits of end-to-end learning (at least when large amounts of data are available).
Questions:
1. Why is the context-attn model performance not included in Table 6? Is it because of the optimisation issue associated with that model in Table 3?
2. In page 7, it is mentioned that "Both models have similar attention distributions ... at the first layer, which precedes the introduction of new parameters in both models". Does the first layer here refer to the token + position embedding layer?