title: Impossible Distillation: from Low-Quality Model to High-Quality Dataset
    & Model for Summarization and Paraphrasing
summary: It is commonly perceived that the strongest language models (LMs) rely on a
  combination of massive scale, instruction data, and human feedback to perform
  specialized tasks -- e.g. summarization and paraphrasing, without supervision.
  In this paper, we propose that language models can learn to summarize and
  paraphrase sentences, with none of these 3 factors. We present Impossible
  Distillation, a framework that distills a task-specific dataset directly from
  an off-the-shelf LM, even when it is impossible for the LM itself to reliably
  solve the task. By training a student model on the generated dataset and
  amplifying its capability through self-distillation, our method yields a
  high-quality model and dataset from a low-quality teacher model, without the
  need for scale or supervision. Using Impossible Distillation, we are able to
  distill an order of magnitude smaller model (with only 770M parameters) that
  outperforms 175B parameter GPT-3, in both quality and controllability, as
  confirmed by automatic and human evaluations. Furthermore, as a useful
  byproduct of our approach, we obtain DIMSUM+, a high-quality dataset with 3.4M
  sentence summaries and paraphrases. Our analyses show that this dataset, as a
  purely LM-generated corpus, is more diverse and more effective for
  generalization to unseen domains than all human-authored datasets -- including
  Gigaword with 4M samples.
paper_authors._id: 64741aac8b7a55cfa92187ae
paper_authors.name: Jaehun Jung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T18:00:08.902Z
paper_authors._id: 61703fa3dff0ef663e421ab5
paper_authors.avatarUrl: /avatars/9c525aab827384c3657a37eee43935f7.svg
paper_authors.fullname: Jung
paper_authors.type: user
paper_authors.user: Jaehun
paper_authors._id: 64741aac8b7a55cfa92187af
paper_authors.name: Peter West
paper_authors._id: 64741aac8b7a55cfa92187b0
paper_authors.name: Liwei Jiang
paper_authors._id: 64741aac8b7a55cfa92187b1
paper_authors.name: Faeze Brahman
paper_authors._id: 64741aac8b7a55cfa92187b2
paper_authors.name: Ximing Lu
paper_authors._id: 64741aac8b7a55cfa92187b3
paper_authors.name: Jillian Fisher
paper_authors._id: 64741aac8b7a55cfa92187b4
paper_authors.name: Taylor Sorensen
paper_authors._id: 64741aac8b7a55cfa92187b5
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-10T08:39:09.396Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_title: Impossible Distillation: from Low-Quality Model to High-Quality Dataset
    & Model for Summarization and Paraphrasing
paper_summary: It is commonly perceived that the strongest language models (LMs) rely on a
  combination of massive scale, instruction data, and human feedback to perform
  specialized tasks -- e.g. summarization and paraphrasing, without supervision.
  In this paper, we propose that language models can learn to summarize and
  paraphrase sentences, with none of these 3 factors. We present Impossible
  Distillation, a framework that distills a task-specific dataset directly from
  an off-the-shelf LM, even when it is impossible for the LM itself to reliably
  solve the task. By training a student model on the generated dataset and
  amplifying its capability through self-distillation, our method yields a
  high-quality model and dataset from a low-quality teacher model, without the
  need for scale or supervision. Using Impossible Distillation, we are able to
  distill an order of magnitude smaller model (with only 770M parameters) that
  outperforms 175B parameter GPT-3, in both quality and controllability, as
  confirmed by automatic and human evaluations. Furthermore, as a useful
  byproduct of our approach, we obtain DIMSUM+, a high-quality dataset with 3.4M
  sentence summaries and paraphrases. Our analyses show that this dataset, as a
  purely LM-generated corpus, is more diverse and more effective for
  generalization to unseen domains than all human-authored datasets -- including
  Gigaword with 4M samples.
paper_ai_keywords: distillation task-specific dataset off-the-shelf LM student model self-distillation high-quality model low-quality teacher model sentence summaries paraphrases LM-generated corpus human-authored datasets Gigaword