Impossible Distillation: from Low-Quality Model to High-Quality Dataset
  & Model for Summarization and Paraphrasing
It is commonly perceived that the strongest language models (LMs) rely on a
combination of massive scale, instruction data, and human feedback to perform
specialized tasks -- e.g. summarization and paraphrasing, without supervision.
In this paper, we propose that language models can learn to summarize and
paraphrase sentences, with none of these 3 factors. We present Impossible
Distillation, a framework that distills a task-specific dataset directly from
an off-the-shelf LM, even when it is impossible for the LM itself to reliably
solve the task. By training a student model on the generated dataset and
amplifying its capability through self-distillation, our method yields a
high-quality model and dataset from a low-quality teacher model, without the
need for scale or supervision. Using Impossible Distillation, we are able to
distill an order of magnitude smaller model (with only 770M parameters) that
outperforms 175B parameter GPT-3, in both quality and controllability, as
confirmed by automatic and human evaluations. Furthermore, as a useful
byproduct of our approach, we obtain DIMSUM+, a high-quality dataset with 3.4M
sentence summaries and paraphrases. Our analyses show that this dataset, as a
purely LM-generated corpus, is more diverse and more effective for
generalization to unseen domains than all human-authored datasets -- including
Gigaword with 4M samples.
64741aac8b7a55cfa92187ae
Jaehun Jung
claimed_verified
2023-05-30T18:00:08.902Z
61703fa3dff0ef663e421ab5
/avatars/9c525aab827384c3657a37eee43935f7.svg
Jung
user
Jaehun
64741aac8b7a55cfa92187af
Peter West
64741aac8b7a55cfa92187b0
Liwei Jiang
64741aac8b7a55cfa92187b1
Faeze Brahman
64741aac8b7a55cfa92187b2
Ximing Lu
64741aac8b7a55cfa92187b3
Jillian Fisher
64741aac8b7a55cfa92187b4
Taylor Sorensen
64741aac8b7a55cfa92187b5
Yejin Choi
admin_assigned
2023-08-10T08:39:09.396Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
Impossible Distillation: from Low-Quality Model to High-Quality Dataset
  & Model for Summarization and Paraphrasing
It is commonly perceived that the strongest language models (LMs) rely on a
combination of massive scale, instruction data, and human feedback to perform
specialized tasks -- e.g. summarization and paraphrasing, without supervision.
In this paper, we propose that language models can learn to summarize and
paraphrase sentences, with none of these 3 factors. We present Impossible
Distillation, a framework that distills a task-specific dataset directly from
an off-the-shelf LM, even when it is impossible for the LM itself to reliably
solve the task. By training a student model on the generated dataset and
amplifying its capability through self-distillation, our method yields a
high-quality model and dataset from a low-quality teacher model, without the
need for scale or supervision. Using Impossible Distillation, we are able to
distill an order of magnitude smaller model (with only 770M parameters) that
outperforms 175B parameter GPT-3, in both quality and controllability, as
confirmed by automatic and human evaluations. Furthermore, as a useful
byproduct of our approach, we obtain DIMSUM+, a high-quality dataset with 3.4M
sentence summaries and paraphrases. Our analyses show that this dataset, as a
purely LM-generated corpus, is more diverse and more effective for
generalization to unseen domains than all human-authored datasets -- including
Gigaword with 4M samples.
Impossible Distillation enables a small model to outperform a large language model in summarization and paraphrasing tasks through self-distillation and generation of a high-quality dataset.
distillation student model self-distillation teacher model language models summaries paraphrases DIMSUM+ Gigaword