PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
<|reference_start|>PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning: Recent advancements in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression. While knowledge distillation (KD) is a prominent method for this, research on KD for generative language models like LLMs is relatively sparse, and the approach of distilling student-friendly knowledge, which has shown promising performance in KD for classification models, remains unexplored in generative language models. To explore this approach, we propose PromptKD, a simple yet effective method that utilizes prompt tuning - for the first time in KD - to enable generative language models to transfer student-friendly knowledge. Unlike previous works in classification that require fine-tuning the entire teacher model for extracting student-friendly knowledge, PromptKD achieves similar effects by adding a small number of prompt tokens and tuning only the prompt with student guidance. Extensive experiments on instruction-following datasets show that PromptKD achieves state-of-the-art performance while adding only 0.0007% of the teacher's parameters as prompts. Further analysis suggests that distilling student-friendly knowledge alleviates exposure bias effectively throughout the entire training process, leading to performance enhancements.<|reference_end|>
@article{kim2024promptkd:,
    title={PromptKD: Distilling Student-Friendly Knowledge for Generative Language
  Models via Prompt Tuning},
    author={Gyeongman Kim, Doohyuk Jang, Eunho Yang},
    journal={arXiv preprint arXiv:2402.12842},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.12842},
    primaryClass={cs.CL cs.AI cs.LG}
}