summaries: This paper proposes a simple yet effective method, cocktail fine-tuning, for the natural language generation tasks. Their results show that cocktail fine-tuning can handle both In-domain data and Out-of-domain data effectively by combing adapter-finetuning and full-finetuning through knowledge distillation and overall has comparable performance compared to their ensembles. It also provides theoretical analysis on multi-class logistic regression to explain why it works. This paper proposes an ensemble model between a full fine-tuning model and a parameter-efficient fine-tuning model to improve the out-of-distribution (OOD) performance of a full fine-tuning model. The proposed method is inspired by the observation that full fine-tuning model achieves good in-distribution (ID) performance while parameter-efficient finetuning model achieves better OOD performance. There are two ensembling methods presented in the paper: linear interpolation between the predictions of the two models; and distill from the predictions of a parameter-efficient model with ID training data. Improved OOD performance is observed with this ensemble method. This paper presents interesting an idea of combining lightweight fine-tuning and full fine-tuning to achieve the best of both approaches, i.e. perform best on out-of-domain and in-domain data. The authors proposed two approaches: a simple ensemble method and a so-called cocktail fine-tuning that combines two fine-tuning methods in one single model. They evaluated their tasks in three datasets: WebNLG, XSUM and OpenQA and obtained mixed results. The authors also provided good analyses for more insights.  The present paper first discusses the trade-off between performance for out-of-domain data and in-domain data with respect to whether the model is fully fine-tuned or lightweight fine-tuned on NLG tasks. Second, it argues that such a trade-off is not necessary if one can make use of both of these two fine-tuning schema in a clever way. To this end, it proposes cocktail fine-tuning, which augments full fine-tuning via distillation from a lightweight model and which achieves equal performance as an ensemble of the two fine-tuning schema. At length, this paper also explains the behavior of the cocktail fine-tuning through a toy model. 
abstractText: When finetuning a pretrained language model for natural language generation tasks, one is currently faced with a tradeoff. Lightweight finetuning (e.g., prefixtuning, adapters), which freezes all or most of the parameters of the pretrained model, has been shown to achieve stronger out-of-distribution (OOD) performance than full finetuning, which tunes all of the parameters. However, lightweight finetuning can underperform full finetuning in-distribution (ID). In this work, we present methods to combine the benefits of full and lightweight finetuning, achieving strong performance both ID and OOD. First, we show that an ensemble of the lightweight and full finetuning models achieves the best of both worlds: performance matching the better of full and lightweight finetuning, both ID and OOD. Second, we show that we can achieve similar improvements using a single model instead of two with our proposed cocktail finetuning, which augments full finetuning via distillation from a lightweight model. Finally, we provide some explanatory theory in a multiclass logistic regression setting with a large number of classes, describing how distillation on ID data can transfer the OOD behavior of one model to another.
authors.name: ROBUST FINETUNING
references.authors: Samira Abnar Mostafa Dehghani Willem Zuidema
references.title: Transferring inductive biases through knowledge distillation
references.venue: arXiv preprint arXiv:2006.00555,
references.authors: Jonathan Berant Andrew Chou Roy Frostig Percy Liang
references.title: Semantic parsing on Freebase from question-answer pairs
references.venue: In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,
references.authors: David Berthelot Rebecca Roelofs Kihyuk Sohn Nicholas Carlini Alex Kurakin
references.title: Adamatch: A unified approach to semi-supervised learning and domain adaptation
references.venue: CoRR, abs/2106.04732,
references.authors: Cristian Bucilă Rich Caruana Alexandru Niculescu-Mizil
references.title: Model compression
references.venue: In International Conference on Knowledge Discovery and Data Mining (KDD),
references.authors: Yair Carmon Aditi Raghunathan Ludwig Schmidt Percy Liang John C. Duchi
references.title: Unlabeled data improves adversarial robustness
references.venue: In Advances in Neural Information Processing Systems (NeurIPS),
references.authors: Tommaso Furlanello Zachary Lipton Michael Tschannen Laurent Itti Anima Anandkumar
references.title: Born again neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Claire Gardent Anastasia Shimorina Shashi Narayan Laura Perez-Beltrachini
references.title: The WebNLG challenge: Generating text from RDF data
references.venue: In Proceedings of the 10th International Conference on Natural Language Generation,
references.authors: Geoffrey Hinton Oriol Vinyals Jeffrey Dean
references.title: Distilling the knowledge in a neural network
references.venue: In NIPS Deep Learning and Representation Learning Workshop,
references.authors: Neil Houlsby Andrei Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin De Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly
references.title: Parameter-efficient transfer learning for NLP
references.venue: Proceedings of the 36th International Conference on Machine Learning,
references.authors: Neil Houlsby Andrei Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin de Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly
references.title: Parameter-efficient transfer learning for NLP. arXiv, 2019b
references.authors: Jeremy Howard Sebastian Ruder
references.title: Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
references.authors: Mihir Kale Abhinav Rastogi
references.title: Text-to-text pre-training for data-to-text tasks
references.venue: In Proceedings of the 13th International Conference on Natural Language Generation,
references.authors: Jungo Kasai Hao Peng Yizhe Zhang Dani Yogatama Gabriel Ilharco Nikolaos Pappas Yi Mao Weizhu Chen Noah A Smith
references.title: Finetuning pretrained transformers into rnns
references.venue: arXiv preprint arXiv:2103.13076,
references.authors: Fereshte Khani Percy Liang
references.title: Removing spurious features can hurt accuracy and affect groups disproportionately
references.venue: In ACM Conference on Fairness, Accountability, and Transparency (FAccT),
references.authors: Tom Kwiatkowski Jennimaria Palomaki Olivia Redfield Michael Collins Ankur Parikh Chris Alberti Danielle Epstein Illia Polosukhin Matthew Kelcey Jacob Devlin Kenton Lee Kristina N. Toutanova Llion Jones Ming-Wei Chang Andrew Dai Jakob Uszkoreit Quoc Le Slav Petrov
references.title: Natural questions: a benchmark for question answering research
references.venue: Transactions of the Association of Computational Linguistics,
references.authors: Kenton Lee Ming-Wei Chang Kristina Toutanova
references.title: Latent retrieval for weakly supervised open domain question answering
references.venue: In Association for Computational Linguistics (ACL),
references.authors: Lisa Lee Benjamin Eysenbach Emilio Parisotto Eric Xing Sergey Levine Ruslan Salakhutdinov
references.title: Efficient exploration via state marginal matching
references.authors: Brian Lester Rami Al-Rfou Noah Constant
references.title: The power of scale for parameter-efficient prompt tuning
references.authors: Mike Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdelrahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer
references.title: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Xiang Lisa Li Percy Liang
references.title: Prefix-tuning: Optimizing continuous prompts for generation
references.venue: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582–4597,
references.authors: Chin-Yew Lin
references.title: ROUGE: A package for automatic evaluation of summaries
references.venue: In Text Summarization Branches Out,
references.authors: Pengfei Liu Weizhe Yuan Jinlan Fu Zhengbao Jiang Hiroaki Hayashi Graham Neubig
references.title: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing
references.venue: ArXiv, abs/2107.13586,
references.authors: Robert L Logan IV Ivana Balažević Eric Wallace Fabio Petroni Sameer Singh Sebastian Riedel
references.title: Cutting down on prompts and parameters: Simple few-shot learning with language models
references.venue: arXiv preprint arXiv:2106.13353,
references.authors: Amir Najafi Shin ichi Maeda Masanori Koyama Takeru Miyato
references.title: Robustness to adversarial perturbations in learning from incomplete data
references.venue: In Advances in Neural Information Processing Systems (NeurIPS),
references.authors: Shashi Narayan Shay B. Cohen Mirella Lapata
references.title: Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels,
references.authors: Deepak Narayanan Keshav Santhanam Amar Phanishayee Matei Zaharia
references.title: Accelerating Deep Learning Workloads through Efficient Multi-Model Execution
references.venue: In NeurIPS Workshop on Systems for Machine Learning,
references.authors: Kishore Papineni Salim Roukos Todd Ward Wei-Jing Zhu
references.title: Bleu: A method for automatic evaluation of machine translation
references.venue: In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics,
references.authors: Kishore Papineni Salim Roukos Todd Ward Wei-Jing Zhu
references.title: BLEU: A method for automatic evaluation of machine translation
references.venue: In Association for Computational Linguistics (ACL),
references.authors: Jonas Pfeiffer Aishwarya Kamath Andreas Rücklé Kyunghyun Cho Iryna Gurevych
references.title: Adapterfusion: Non-destructive task composition for transfer learning, 2020
references.authors: Guanghui Qin Jason Eisner
references.title: Learning how to ask: Querying LMs with mixtures of soft prompts
references.venue: pp. 5203–5212,
references.authors: Alec Radford Jong Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever
references.title: Learning transferable visual models from natural language supervision, 2021
references.authors: Aditi Raghunathan Sang Michael Xie Fanny Yang John C. Duchi Percy Liang
references.title: Understanding and mitigating the tradeoff between robustness and accuracy
references.venue: In International Conference on Machine Learning (ICML),
references.authors: Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang
references.title: SQuAD: 100,000+ questions for machine comprehension of text
references.venue: In Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Sylvestre-Alvise Rebuffi Hakan Bilen Andrea Vedaldi
references.title: Learning multiple visual domains with residual adapters
references.venue: Advances in Neural Information Processing Systems,
references.authors: Adam Roberts Colin Raffel Noam Shazeer
references.title: How much knowledge can you pack into the parameters of a language model
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Taylor Shin Yasaman Razeghi Robert L. Logan IV au Eric Wallace Sameer Singh
references.title: Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020
references.authors: Jonathan Uesato Jean-Baptiste Alayrac Po-Sen Huang Robert Stanforth Alhussein Fawzi Pushmeet Kohli
references.title: Are labels required for improving adversarial robustness
references.venue: In Advances in Neural Information Processing Systems (NeurIPS),
references.authors: Prasetya Ajie Utama Nafise Sadat Moosavi Victor Sanh Iryna Gurevych
references.title: Avoiding inference heuristics in few-shot prompt-based finetuning, 2021
references.authors: Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Pierric Cistac Tim Rault Rémi Louf Morgan Funtowicz Joe Davison Sam Shleifer Patrick von Platen Clara Ma Yacine Jernite Julien Plu Canwen Xu Teven Le Scao Sylvain Gugger Mariama Drame Quentin Lhoest Alexander M. Rush
references.title: Transformers: State-of-the-art natural language processing
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro-
references.authors: Mitchell Wortsman Gabriel Ilharco Mike Li Jong Wook Kim Hannaneh Hajishirzi Ali Farhadi Hongseok Namkoong Ludwig Schmidt
references.title: Robust fine-tuning of zero-shot models
references.venue: arXiv preprint arXiv:2109.01903,
references.authors: Sang Michael Xie Ananya Kumar Robert Jones Fereshte Khani Tengyu Ma Percy Liang
references.title: InN-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness
references.venue: In International Conference on Learning Representations (ICLR),
references.authors: Elad Ben Zaken Shauli Ravfogel Yoav Goldberg
references.title: Bitfit: Simple parameter-efficient finetuning for transformer-based masked language-models
references.venue: CoRR, abs/2106.10199,
references.authors: Kaiyang Zhou Jingkang Yang Chen Change Loy Ziwei Liu
references.title: Learning to prompt for visionlanguage models, 2021
references.authors: George Kingsley Zipf
references.title: Human behavior and the principle of least effort
sections.text: 1 INTRODUCTION
  When finetuning a pretrained language model for natural language generation tasks like summarization (Narayanan et al., 2018) and table-to-text (Gardent et al., 2017), one is currently faced with a tradeoff. One can achieve strong in-distribution (ID) performance or strong out-of-distribution (OOD) performance—but not both—by choosing one of two families of finetuning approaches. Finetuning all parameters of the pretrained language model achieves strong ID performance (Howard & Ruder, 2018). We call this full finetuning. Alternatively, freezing most of the pretrained parameters during finetuning, in e.g., adapters (Rebuffi et al., 2017; Houlsby et al., 2019a), prefix-tuning (Li & Liang, 2021; Lester et al., 2021) and bitfit (Zaken et al., 2021), achieves stronger OOD performance than full finetuning (Li & Liang, 2021; Lester et al., 2021), but worse ID performance. We call these methods lightweight finetuning.
  In this work, we are the first to demonstrate that this tradeoff is not necessary in natural language generation: we can achieve the best of full and lightweight finetuning both ID and OOD. We test this across three
  natural language generation tasks in English: summarization (Narayanan et al., 2018), table-to-text (Gardent et al., 2017), and open-domain question answering (Roberts et al., 2020; Lee et al., 2019a), and two lightweight finetuning methods: adapters and prefix-tuning. Achieving the best of both worlds is important since across our 6 settings (2 lightweight finetuning methods and 3 datasets),
  full finetuning significantly outperforms lightweight finetuning ID in 5 settings, and lightweight finetuning significantly outperforms full finetuning OOD in 6 settings.
  We present two methods for achieving the best of full and lightweight finetuning. We first show that a weighted ensemble of one full finetuning model and one lightweight finetuning model achieves performance comparable to the best of the two, both ID and OOD. To avoid running two models at once, we propose cocktail finetuning: first train a lightweight finetuning model, and then train a full finetuning model on a mixture of student-teacher distillation loss (Bucilă et al., 2006; Hinton et al., 2015) (with the lightweight model as teacher) and cross-entropy loss, on the original ID data.1 The ensemble weight (resp. the cocktail mixture loss weight) is chosen to optimize for ID performance, and we observe that the resultant model achieves comparable OOD performance to lightweight finetuning.
  Our work contributes to an ongoing realization in machine learning: robust models that deviate minimally from a pretrained model can be gainfully combined with a model adapted more precisely to a specific dataset. Most saliently, in concurrent work, Wortsman et al. (2021) find that zeroshot image recognition through CLIP (Radford et al., 2021) can be combined with a finetuned network to perform well ID and OOD. Our work demonstrates that similar results hold true in natural language generation as well. We draw from simple, well-known methods: ensembles, student-teacher distillation (Abnar et al., 2020; Hinton et al., 2015)—but our application to achieving the best of ID and OOD performance in natural language generation is novel.
  To better understand how cocktail finetuning can transfer the out-of-distribution behavior of the lightweight finetuning model when distilling only on in-distribution data, we theoretically study multiclass logistic regression with a large number of output classes, viewing language generation as a classification problem over a large output space. We study a distribution shift setting where not all classes are seen in the ID training data (as is true in language generation due to Zipf’s law (Zipf, 1949)), and getting good OOD accuracy requires generalizing to unseen outputs. First, we prove that a model trained with gradient descent cannot distinguish the unseen output classes, leading to poor OOD accuracy. Second, we prove that distilling a teacher model with good OOD accuracy on only ID training data can transfer the OOD behavior of the teacher to the student, such that the student and teacher make the same prediction on any new input. As a result, the student also has good OOD accuracy.
  In summary, we present two simple methods—ensembling and cocktail finetuning—for achieving strong ID and OOD natural language generation with only the original ID data, achieving the “best of both worlds” of full and lightweight finetuning methods.
sections.heading: 2 RELATED WORK
sections.text: Lightweight finetuning. Recently, as pretrained models scale up in the number of parameters, lightweight finetuning techniques have been developed to avoid updating all of the parameters. These techniques update a subset of the parameters (Zaken et al., 2021), introduce a small number of new parameters between layers of the model (called “adapters”) (Rebuffi et al., 2017; Houlsby et al., 2019b; Pfeiffer et al., 2020), or learn auxiliary inputs that are prepended to the models inputs (called “prompts”) (Shin et al., 2020; Li & Liang, 2021; Qin & Eisner, 2021; Liu et al., 2021b; Logan IV et al., 2021; and see Liu et al., 2021a for a survey). These techniques empirically have the benefit of being able to achieve strong out-of-distribution extrapolation performance at the cost of worse in-distribution performance compared to full finetuning models both in classification and generation settings (Li & Liang, 2021; Lester et al., 2021; Zhou et al., 2021; Utama et al., 2021). In our work, we aim to overcome this trade-off by proposing techniques for achieving the best of both worlds, with a focus on language generation.
  Student-teacher distillation. Buciluǎ et al. (2006) introduce distillation for compressing large ensembles into smaller, faster, more efficient models, and Hinton et al. (2015) generalize their method and call it knowledge distillation. Kasai et al. (2021) distill Transformers into RNNs for linear-time inference. Abnar et al. (2020) show that knowledge distillation can transfer the inductive bias of one architecture to another: for example, distilling an LSTM to a Transformer leads to a Transformer with
  1We use the term cocktail as it is a refreshing mixture involving distillation.
  representations that are more similar to independently trained LSTMs than independently trained Transformers. The efficiacy of our proposed cocktail finetuning is surprising even in context of these results, as it empirically transfers just the strong OOD performance of the teacher lightweight model to the student, while not transferring the weak ID performance.
  Robust finetuning in vision. In concurrent work, Wortsman et al. (2021) propose a similar approach to ours, but focus on image classification: they combine a full finetuning model with a lightweight finetuning (linear probe) model by ensembling the two models in weight space, and show that their approach maintains both good ID and OOD performance. It is yet surprising that in the structured prediction setting of natural language generation, we further show that we can achieve high OOD performance even when weight-space ensembling is impossible (for instance, since there are weights not held in common between the full finetuning model and an optimized prompt). Moreover, we further this line of work by providing a theoretical study into how distillation can transfer OOD behavior while only using ID data.
  Self-training for robustness. In adversarial robustness, standard models achieve good accuracy on clean examples but are not robust to adversarial examples, while adversarially trained models are more robust but are less accurate on clean examples. Raghunathan et al. (2020) show that the benefits of these two models can be combined using robust self-training (RST) (Carmon et al., 2019; Uesato et al., 2019; Najafi et al., 2019) on unlabeled data to improve clean and robust accuracy. Similarly, Khani & Liang (2021) show that RST can be used to combine models with spurious features removed (which are more robust) and without removing spurious features (which are more accurate). In general OOD robustness, In-N-Out (Xie et al., 2021) uses self-training to transfer the robustness benefits of a model pre-trained with auxiliary self-supervision to a model trained with extra auxiliary features. Berthelot et al. (2021) use unlabeled data and a consistency regularization loss for domain adaptation. While many of these methods combine the benefits of two models or use a self-training/distillation loss to improve robustness, cocktail finetuning differs in that it performs distillation only on ID training data without any additional ID or OOD unlabeled examples.
sections.heading: 3 PROBLEM STATEMENT
sections.heading: 3.1 SETUP
sections.text: We consider a conditional generation task from an input space X (e.g., a news article) to an output space Y (e.g., a summary of the article), where both spaces consist of sequences of tokens from a shared vocabulary V .
  Data. Let Pid and Pood denote the distribution of (x, y) pairs in-distribution and out-of-distribution, respectively, where x ∈ X and y ∈ Y . In summarization, Pid may consist of articles about world and business news while Pood consists of health and technology news. The training data Dtrain and validation data Dval consist of n and nval in-distribution (x, y) pairs respectively, drawn from Pid. Model and metrics. Let qγ(y | x) = ∏len(y) j=1 qγ(yj | x, y<j) be a pretrained neural autoregressive sequence model with parameters γ. Finetuning (full or lightweight) produces a finetuned model pθ(y | x) from qγ , where θ denotes the set of trainable parameters. In full finetuning, θ is the same set of parameters as γ; in lightweight finetuning, θ often refers to the parameters of inserted modules (e.g., adapters modules or prefixes). The final predictor is fθ(x) which approximates argmaxy pθ(y | x) through, e.g., beam search. The goal is to learn parameters θ such that the predictor fθ obtains good ID and OOD performance:
  Sid(fθ) = Ex,y∼Pid [score(y, fθ(x))] (1) Sood(fθ) = Ex,y∼Pood [score(y, fθ(x))] (2)
  where score may be defined differently for each task (e.g., exact match score or BLEU (Papineni et al., 2002b)).
sections.heading: 3.2 BASELINE FINE-TUNING METHODS
sections.text: Full finetuning has been the defacto approach to adapt pretrained language models for downstream generation tasks (Lewis et al., 2020; Kale & Rastogi, 2020). Under this framework, the model pθ is initialized with its pretrained parameters γ, and we optimize the following log-likelihood objective.
  L(θ) = E(x,y)∼Dtrain [− log pθ(y | x)] (3)
  where the set of trainable parameters θ refer to the set of pretrained parameters γ.
  Lightweight finetuning is another family of adaptation approaches, which freezes most of the pretrained parameters and augments the model with small trainable modules. Lightweight finetuning shares the same training objective as full finetuning, but the trainable parameters θ are different. Within the lightweight finetuning family, we focus on adapter-tuning (Houlsby et al., 2019a) and prefix-tuning (Li & Liang, 2021). Both approaches keep the pretrained parameters intact. Adaptertuning inserts task-specific MLP layers between each layer of the pretrained language models, and the trainable parameter θ consists of these inserted MLP parameters. Prefix-tuning prepends a sequence of trainable, task-specific prefix vectors to the input, and θ consists of these prefix parameters.2
sections.heading: 4 EXPERIMENT SETUP
sections.heading: 4.1 DATASETS AND METRICS
sections.text: Table-to-text. We evaluate on the WebNLG dataset (Gardent et al., 2017), where the input x is a sequence of (entity, predicate, entity) triples and the output y is a natural language description that covers the input information. Each example in this dataset is labeled with one of 14 topic (e.g., airports, comic characters). We select 9 topics as in-distribution and the remaining 5 topics as out-of-distribution. We evaluate the performance using the official evaluation scripts and report the BLEU scores (Papineni et al., 2002a).
  Summarization. We use the XSUM (Narayan et al., 2018) dataset, which is an abstractive summarization dataset on BBC news articles. Each example is labeled with a news category. We select world, UK and business categories to be in-distribution and let the remaining news categories such as health and technology to be out-of-distribution. We report the ROUGE-2 scores (Lin, 2004).
  Open-domain, closed-book QA (Open-QA). We use the open-domain variant (Lee et al., 2019b) of the Natural Questions dataset (Kwiatkowski et al., 2019) as the in-distribution data and the Web Questions dataset (Berant et al., 2013) as the out-of-distribution data. The distribution shift is induced by different data collection process and their year of collection (roughly 2018 versus 2013). We
  2For specifics of these methods, see their respective papers.
  report the exact match accuracy. Note that this task, unlike most QA (e.g., SQuAD (Rajpurkar et al., 2016)), does not provide systems with a text from which to extract answers. Roberts et al. (2020) demonstrated that one could finetune an LM to generate answers from knowledge stored during pretraining.
sections.heading: 4.2 MODEL ARCHITECTURES AND HYPERPARAMETERS
sections.text: We use GPT-2-medium (Radford et al., 2019) as the pretrained model for table-to-text and opendomain QA tasks. GPT-2 is an autoregressive language model; therefore, we parametrize pθ(y | x) by concatenating x, a separator token, and y (e.g., [x; SEP; y]). We define the loss function to sum over the token losses that correspond to y.
  We use BART-large (Lewis et al., 2020) as the pretrained model for the summarization task. BART uses an encoder-decoder architecture and parametrizes pθ(y | x) by first feeding x to the encoder, and decoding y conditioned on the encoder representation.
  For open-domain closed-book QA, this is to our knowledge the first time GPT-2 has been finetuned for the task. We used separate development OOD data to test a handful (<10) of hyperparameter settings to determine whether we’d observe a similar ID/OOD tradeoff to the other tasks. We note this because, in general, one cannot do hyperparameter optimization on OOD data; we did this to provide another testbed for our methods.
  Our implementation is based on the Hugging Face Transformers package (Wolf et al., 2020). We use most of the hyperparameters suggested by the Hugging Face default setup. The hyperparameters we tune include the number of epochs, batch size, learning rate, and prefix length (for prefix-tuning), all reported in Table 3 in the appendix.
sections.heading: 5 RESULTS OF FULL AND LIGHTWEIGHT FINETUNING
sections.text: There is a trade-off in ID/OOD performance when choosing between full finetuning and lightweight finetuning: full finetuning consistently outperforms in-distribution, whereas lightweight finetuning consistently outperforms out-of-distribution (Li & Liang, 2021; Lester et al., 2021). To quantify this trade-off, we examine the generation performance of both methods on ID and OOD data.
  As shown in Table 2, when evaluating in-distribution, full finetuning achieves significantly stronger results than lightweight finetuning in five out of six settings (across 3 datasets and two lightweight finetuning methods. Only prefix-tuning on WebNLG achieves comparable accuracy to full finetuning. When evaluating out-of-distribution, lightweight finetuning achieves significantly stronger results than full finetuning in five of six settings. Only adapter-tuning on XSUM fails to outperform full finetuning.3
sections.heading: 6 COMBINING MODELS WITH A WEIGHTED ENSEMBLE
sections.text: As we observe in Section 5, full finetuning models attain better ID performance but worse OOD performance, whereas lightweight finetuning models attain better OOD performance at the cost of good ID performance. Is it possible to achieve both good ID and OOD performance simultaneously? In this section, we show that a simple ensemble of the two methods accomplishes this.
  First, we train a lightweight finetuning model p(`) and a full finetuning model p(f) on Dtrain. At decoding time, we combine the prediction of the p(`) and p(f) by interpolating them at each token.
  p(Ens)(yt | x, y<t) = λ · p(`)(yt | x, y<t) + (1− λ) · p(f)(yt | x, y<t) (4)
  At each time step t, both models output a next token distribution conditioned on the common history y<t, and the two distributions are interpolated with a mixture weight λ. The output is generated by beam searching the ensemble distribution p(Ens)(yt | x, y<t).
  3Though it is worth noting again that for open-domain QA, we performed a small amount of manual, exploratory hyperparameter optimization to optimize the ID of full finetuning and the OOD of lightweight finetuning.
  6.1 CHOOSING THE MIXTURE WEIGHT λ
  We start with a set of {λi}Ki=1 where λi ∈ [0, 1]. For example we use λ ∈ {0., 0.1, 0.25, 0.5, 0.75, 0.9, 1.0} in this experiment. We choose the mixture weight λi that attains the best score on Dval.
  This method of choosing λ is useful because (1) it crucially does not use OOD data, which is typically unaccessible at model selection time, and (2) it guarantees that ID performance is not sacrificed, because we can match ID performance by setting λ = 0.
sections.heading: 6.2 RESULTS
sections.text: As shown in Table 2, the ensemble matches and slightly outperforms both the ID performance skyline (achieved by full finetuning) and the OOD performance skyline (achieved by lightweight finetuning). Across all datasets, ensembling achieves significantly improved performance over full finetuning OOD, and lightweight finetuning ID. Further, in eleven out of twelve settings (across 3 datasets, 2 methods, and both ID and OOD), ensembling does not perform significantly worse than the better of full and lightweight finetuning.
sections.heading: 6.3 RUNTIME AND MEMORY
sections.text: Despite its ability to combine the best of both worlds, ensembling is expensive at inference time because two models are run simultaneously, doubling the amount of GPU memory required. Furthermore, two forward passes are required, doubling the runtime for decoding.4 For deployment efficiency, it’s better to have a single model at inference time.
sections.heading: 7 COCKTAIL FINETUNING: COMBINING WITH DISTILLATION
sections.text: The goal of cocktail finetuning is to approximate the ensemble in Section 6 with a single model.5
sections.heading: 7.1 METHOD
sections.text: Train a lightweight finetuning model p(`). Fix mixture weight λ ∈ [0, 1]. Train all parameters of the pretrained model qγ(y | x) on the following objective. First, define a token-level student-teacher
  4The two forward passes are parallelizable, but at the cost of another 2x GPU memory. 5One might wish to approximate the two with a single lightweight model, a direction we explored but eventually discarded due to the difficulty of approximating a full finetuning model with a lightweight finetuning model. Anecdotally, this is evidence towards the speculation of Lester et al. (2021), that lightweight finetuning is limited in its ability to alter a pretrained LM.
  distillation loss at token j: distillp(`)(x, y, j) = KL ( p(`)(yj | x, y<j)‖pθ(yj | x, y<j)) ) (5)
  The objective is the sum over the length of y of the λ-weighted mixture of distillation and loglikelihood loss:
  Lλ(θ) = E(x,y)∼Dtrain [ len(y)∑ j=1 λdistillp(`)(x, y, j)− (1− λ) log pθ(yj | x, y<j) ]
  (6)
  Let fθ,λ be the predictor resulting from optimizing on Lλ(θ) and estimate the ID score Sid(fθ,λ) on Dval. Perform this across a set of mixture weights {λi}i. The cocktail finetuning model is the predictor fθ,λi that maximizes the in-distribution score Sid(fθ, λi) across the mixture weights λi. 6
sections.heading: 7.2 RESULTS
sections.text: As shown in Table 2, cocktail finetuning matches and slightly outperforms the ID performance skyline (achieved by full finetuning) and achieves comparable to or slightly worse than the OOD performance skyline (achieved by lightweight finetuning). Across all datasets, cocktail finetuning achieves significantly improved performance over full finetuning OOD, and over lightweight finetuning ID. Further, in seven out of twelve settings (across 3 datasets, 2 methods, and both ID and OOD), cocktail finetuning does not perform significantly worse than the better of full and lightweight finetuning.
sections.heading: 8 ANALYSIS OF THE WEIGHTING OF FULL AND LIGHTWEIGHT FINETUNING
sections.text: Trade-off curves. Our ensembling and cocktail finetuning methods are useful in practice because one can choose the mixture coefficient λ without OOD data. Now, we explore how ID and OOD performance vary as a function of λ. Figure 3 shows results across five random seeds of ensembling and cocktail finetuning on our three datasets, each dot colored according to its λ value. In each plot, we see a smooth tradeoff curve, in which a range of λ improve on the OOD, ID, or both of each constituent model. Interestingly, in two of three tasks, there is effectively only one pareto-optimal λ. This means the λ is best for both for ID and OOD
  performance, which is why optimizing for ID performance also leads to good OOD performance.
  6For cocktail finetuning, we use the hyperparameters reported for full finetuning; no hyperparameter optimization was performed.
  Ablations. One hypothesis to explain the strong performance of ensembles is the known benefits of ensembling two models trained with different random seeds. Likewise for cocktail finetuning, one could point to the effectiveness of born-again networks (Furlanello et al., 2018): distilling a teacher model to an identically parameterized student with the same model architecture. As an ablation to test this hypothesis, we distill from a full finetuning model into another full finetuning model, again using λ to mix the distillation loss with the label loss. The results, in Figure 4, show that this distillation does not improve over a single full finetuning model, for any of the λ.
sections.heading: 9 WHY DOES DISTILLATION ON ID DATA IMPROVE OOD?
sections.text: Prior works have used unlabeled OOD data and self-training (similar to distillation) to transfer good OOD performance from another model (Raghunathan et al., 2020; Xie et al., 2021). In contrast, cocktail finetuning performs distillation only on ID data. In this section, we provide an explanation for why this can occur in multiclass logistic regression with a large number of classes.
  Prediction problem and distribution shift. Motivated by the large output space in language generation, we consider a prediction problem from inputs X ⊆ Rd to labels Y = {1, . . . , k} where k is large. The labeled training data is {(xi, yi)}ni=1. For convenience, we also let yi be the one-hot vector of length k denoting the correct class. The labels in the training data do not cover all the possible labels, and we let the subset of seen labels be S. In this problem, the class marginals of the ID and OOD distributions Pid and Pood are different. To have good OOD accuracy, the model must generalize to unseen classes.
  Model and losses. We consider training a multi-class logistic regression model using full batch gradient descent where the learned parameters are θ ∈ Rk×d. In full fine-tuning, we run gradient descent on the negative log-likelihood.
  In cocktail finetuning, we also add a distillation loss using soft labels (a probability vector) from the prefix-tuned model, which we denote px
  θ̃ = pθ̃(· | x) ∈ Rk with parameters θ̃, given an input x.
  Similarly, the predicted class probability vector of the cocktail model given an input x is denoted
  pxθ = pθ(· | x) ∈ Rk. In this setting, the cocktail loss (Equation 6) with parameter λ ∈ [0, 1] is
  `ct(θ) = n∑ i=1 ((1− λ)yi + λpxiθ̃ ) > log pxiθ . (7)
  The cocktail loss encourages the model to learn to output a mixture of the hard label and the soft label on the training data. The final score for the classification task is computed with respect to the accuracy metric, where score(y, ŷ) = 1− `0−1(y, ŷ) and `0−1 is the 0-1 loss.
  Poor OOD performance with full finetuning. Our first result shows that when training a logistic regression model using gradient descent (initialized at 0) on the only labeled training set (as in full finetuning), the learned parameters θj for an unseen label j /∈ S are all the same.
  Proposition 1. Let θ(t) be the parameters at iteration t of gradient descent and let θ(0) = 0. At any iteration t and for any unseen label index j /∈ S, θj = gt where gt is the same for all j /∈ S.
  As a result, the model cannot distinguish between any of the unseen classes, resulting in poor OOD performance. Suppose that the model outputs a fixed, arbitrary unseen class j ∈ S when the probabilities of the unseen classes are the largest. Then, assuming that the OOD distribution Pood is such that the probability of drawing an example from an unseen class (excluding j) Pood(y /∈ S∪{j}) is at least c ∈ [0, 1], then the OOD accuracy is bounded as
  Sood ≤ 1− Pood(y /∈ S ∪ {j}) ≤ 1− c (8)
  This OOD accuracy can be low when c is large, which can occur with very large output spaces.
  Disillation transfers OOD behavior through soft logits. We investigate how distillation on ID training data can affect OOD behavior. In the following, let X ∈ Rn×d be the training data matrix. Proposition 2. Assume the data matrixX has full column rank. Let θ be the minimizer of the cocktail loss when λ = 1, subject to the constraint that 1> exp(θX) = α1> exp(θ̃X) for some constant α > 0, such that the model learns the same normalizing constants (up to scaling) as the prefix-tuned model on the training data only. Then
  pθ(· | x) = pθ̃(· | x) (9)
  for any new input x, such that the cocktail model has the same OOD accuracy as the prefix model.
  Therefore, if the prefix-tuning model has high OOD accuracy, then Sood is high. The proof proceeds by showing that the extra information in the soft logits can help the model recover the parameters of the prefix-tuned model (up to scaling), which in turn inherits its OOD behavior. Thus, in generation problems where the output space is large, distillation on soft logits can transfer information about OOD behavior even through ID data.
sections.heading: 10 CONCLUSION
sections.text: Considered together, our work and the work of Wortsman et al. (2021) provide strong cross-modal evidence that simple ensembles of models that have minimally deviated from pretraining (for them, zero-shot; for us, lightweight finetuning models) with full finetuning models provides strong ID and OOD accuracy. The reasons for this “best of both worlds” behavior are still poorly understood, motivating future work. Since ensemble models effectively weight by model confidence (though modulated by λ), one explanation may be that lightweight finetuning models are less confident than full finetuning models in-distribution, yet more confident out-of-distribution.
  In summary, we provide simple, effective methods for achieving the best of lightweight and full finetuning—strong OOD and ID performance—in language generation. Our cocktail finetuning provides system builders with the opportunity to optimize for in-distribution performance (in choosing lambda), often beating full finetuning, while resting assured that the out-of-distribution performance (i.e., what they observe with real users) will likely be higher than had they performed full finetuning.
  ETHICS STATEMENT
  The methods presented in this work have the potential to improve a wide range of natural language generation tasks. Natural language generation has known dual-use issues, like scaling misinformation generation and generating naturalistic text in scams. The WebNLG XSUM, Natural Questions, and WebQuestions datasets we evaluate on contain personal information (though public and published) information about real people and organizations.
  REPRODUCIBILITY STATEMENT
  All code (including experiment scripts and plot creation scripts) will be released upon publication. All datasets used in this paper are publicly available and will be linked in our released code. All reported numbers represent the average of results from 5 random seeds.
sections.heading: A HYPERPARAMETERS AND DATASET DETAILS
sections.text: A.1 HYPERPARAMETERS
  Relevant hyperparameters beyond the defaults of Huggingface Transformers Wolf et al. (2020) are reported in Table 3.
  A.2 DATASET DETAILS
  Table 5 shows the number of examples in each dataset split.
  For the WebNLG dataset, the ID topics include {Astronaut, University, Monument, Building, ComicsCharacter, Food, Airport, SportsTeam, City, and WrittenWork} and the OOD topics include {Athlete, Artist, MeanOfTransportation, CelestialBody, Politician}. For the XSUM dataset, the ID topics include {uk, business, world } news and the OOD topics includes { entertainment, election, technology, science, health, education, explainers, live, ukpolitics, world-us-canada, world-europe, uk-england, uk-scotland, world-asia, science-environment, uk-scotland-scotland-politics, disability}.
sections.heading: B ADDITIONAL RESULTS
sections.text: Due to space constraints, we did not include in Figure 3 the tradeoff plots using adapters lightweight finetuning method (instead reporting only those for prefix-tuning.) The plots for adapters are provided in Figure 5.
sections.heading: C ANALYSIS
sections.text: We restate and expand upon some of the setup for the analysis here.
  Prediction problem. We consider a prediction problem from inputs X ⊆ Rd to labels Y = {1, . . . , k} where k is large. The labeled training data is {(xi, yi)}ni=1. For convenience, we also let yi be the one-hot vector of length k denoting the correct class, and will make this clear. The labels in the training data do not cover all the possible labels, and we let the subset of seen labels be S.
  Model. We consider training a multi-class logistic regression model using full batch gradient descent. The learned parameters are θ ∈ Rk×d and the model prediction given input x is
  fθ(x) = argmax j pθ(x)j = argmax j θ>j x∑ j′ θ > j′x
  (10)
  where θj is the j-th row of θ. In standard fine-tuning, we run gradient descent on the negative log-likelihood
  `(A) = − n∑ i=1 y>i log p xi θ (11)
  where yi ∈ Rk is a one-hot vector of the class label.
  Distillation loss. In the Cocktail model, we also add a distillation loss using soft labels from the prefix-tuned model, which we denote pθ̃ with parameters θ̃. The Cocktail loss with parameter λ ∈ [0, 1] is
  `ct(A) = n∑ i=1 ( (1− λ)y>i log p xi θ + λ(p xi θ̃ )> log pxiθ ) (12)
  = n∑ i=1 ((1− λ)yi + λpxiθ̃ ) > log pxiθ (13)
  The Cocktail loss encourages the model to learn to output a mixture of the hard label and the soft label on the training data.
  C.1 PROOF OF PROPOSITION 1
  Proof. By induction on t. By zero initialization, the statement holds with g0 = 0.
  Suppose for iterations up to t, we have θ(t)j = gt for all j /∈ S . We focus on some j /∈ S WLOG. The gradient for θ(t)j is
  ∇ θ (t) j `(θ(t)) = ∇ θ (t) j
  ( −
  n∑ i=1 y>i log p xi θ
  ) (14)
  = n∑ i=1 −yijxi + exp((θ (t) j ) >xi)∑ j′ exp((θ (t) j′ ) >xi) xi (15)
  = n∑ i=1 exp((θ (t) j ) >xi)∑ j′ exp((θ (t) j′ ) >xi) xi (16)
  = n∑ i=1 exp(g>t xi)∑ j′ exp((θ (t) j′ ) >xi) xi (17)
  since yij = 0 for all i as j is unseen. This gradient does not vary with j, which shows the result.
  C.2 PROOF OF PROPOSITION 2
  Proof. Let Yθ ∈ Rn×k denote the predicted class probabilities of the model on the training data, and Yθ̃ be the class probabilities of the prefix-tuned model on training data. With λ = 1, the gradient of the loss is
  ∇θ`ct(θ) = n∑ i=1 −pxi θ̃ x>i + p xi θ x > i (18)
  = −Yθ̃X + YθX (19) Since θ is the minimizer, this gradient is 0 and thus
  Yθ̃X = YθX =⇒ Yθ̃ = Yθ (20) for X with full column rank (right-invertible). We let the right-inverse of X be X†. This implies that for all i,
  exp(θ̃xi)∑ j exp(θ̃ > j xi) = exp(θxi)∑ j exp(θ > j xi) . (21)
  In matrix form, this is equivalent to
  exp(θ̃X) · diag(1> exp(θ̃X))−1 = exp(θX) · diag(1> exp(θX))−1 (22) =⇒ θ̃X − 1(log 1> exp(θ̃X))> = θX − 1(log 1> exp(θX))> (23)
  =⇒ θ = θ̃ + 1 [ log 1> exp(θX) log 1> exp(θ̃X) ] X†. (24)
  On a new input x,
  θx = θ̃x+ 1 [ log 1> exp(θX) log 1> exp(θ̃X) ] X†x (25)
  = θ̃x+ (log(α) + 1>X†x)1 (26)
  by using the assumption on normalizing constants. Therefore the probability of the j-th class under the model is
  (pxθ )j = exp(θ>j x)∑ j′ exp(θ > j′x)
  (27)
  = exp(θ̃>j x) exp(log(α) + 1 >X†x)∑ j′ exp(θ̃ > j′x) exp(log(α) + 1 >X†x) (28) = exp(θ̃>j x)∑ j′ exp(θ̃ > j′x) = (px θ̃ )j (29)