Large transformer-based generative models trained on huge corpora have shown unparalleled language generation ability. While these models are powerful, finegrained control of attributes of the generated language (e.g. gradually switching topic or sentiment) is difficult without modifying the model architecture to allow extra attribute inputs or fine-tuning with attribute-specific data. Both entirely change the original generative function -which, if done poorly, cannot be undone -and also entail the significant cost of retraining. We instead propose the simple Plug and Play Language Model (PPLM) approach for controlled language generation. PPLM consists of plugging in simple attribute classifiers (which may be single layer models or even a bag-of-words), and making updates in the activation space, without changing any model parameters. Such a control scheme provides vast flexibility and allows full recovery of the original generative function. The results demonstrate fine-grained control over a range of topics and sentiment styles, as well as the ability to detoxify generated texts. Our experiments, including human evaluation studies, show that text generated via this control scheme is aligned with desired attributes, while retaining fluency.
The authors describe a method for training plug and play language models, a way to incorporate control elements into pre-trained LMs. In contrast to existing work, which often trains conditioned upon the control element, the authors emphasize that their method does not require re-training the initial LM. This is exciting and a great research direction. It is evaluated in a number of different settings.
1. The authors claim that this method is a baseline for controlled text generation (see e.g. the title). However, there does not appear to be any evaluation with any existing work that performs controlled text generation. I don't see how this can be proposed as a baseline for controlled text generation is there is no comparison to other methods. I imagine the authors will emphasize that that's not fair - because their method doesn't require retraining the language model - but it is relevant to demonstrate if there is a gap in performance or not. As is, there is only one baseline- unconditional language model - and to me this is mostly a way to calibrate the evaluators and not a way to compare their model against other models. 
2. Can the authors make a point or discuss the relationship of this work to neural style transfer? Compared to unsupervised style transfer approaches, which also use lists of words or attributes to learn to dis-entangle content and style, what are the benefits of the proposed approach and how would it compare?
3. Can the authors discuss the effectiveness of their control mechanism for less logical control settings? For example, what if there was "religion" for "the potato" prompt? Does the model still respect these settings, or no? 
4. Can the authors add analysis on how much the model respects the control variables? This is quite common in existing controlled generation papers. If the model is updated to have the control variables and then is not provided with one at test time, what happens? Can you also control very easy to measure attributes, such as length?
This question ties in with a general point I am ambivalent to in this paper- that it is very long, but there is very little analysis done on what makes the method work, why it is better than other control methods or control baselines, where the proposed control mechanism is not effective, how the model scales if there are large quantities of topics rather than just a few of them, if the BoW and discriminator attribute models work well together or if certain attributes are easier to learn than others, so the model focuses more on those when there are conflicts, etc
5. Missing citations: 
Previous work has investigated controlling various attributes of text generation. Several of these works have also controlled multiple attributes simultaneously. For example, here's a list of a few of the works that were missed:
Kikuchi et al 2016
Ficler and Goldberg, 2017
Wang et al, 2017
Fan et al, 2018
Baheti et al, 2018
See et al, 2019
Martin et al, 2019
The related work section only focuses on very recent work, e.g. only one paper is discussed amongst a large body of existing work. I feel this is not an accurate reflection of how much previous work has investigated these techniques and analyzed how models deal with control variables. 
Please also cite:
- which dataset was used for story generation, appears to be missing
- top-k sampling 
I have read the author response. Thanks for the details and additional analysis in the paper.