DistiLLM: Towards Streamlined Distillation for Large Language Models
<|reference_start|>DistiLLM: Towards Streamlined Distillation for Large Language Models: Knowledge distillation (KD) is widely used for compressing a teacher model to a smaller student model, reducing its inference cost and memory footprint while preserving model capabilities. However, current KD methods for auto-regressive sequence models (e.g., large language models) suffer from missing a standardized objective function. Moreover, the recent use of student-generated outputs to address training-inference mismatches has significantly escalated computational costs. To tackle these issues, we introduce DistiLLM, a more effective and efficient KD framework for auto-regressive language models. DistiLLM comprises two components: (1) a novel skew Kullback-Leibler divergence loss, where we unveil and leverage its theoretical properties, and (2) an adaptive off-policy approach designed to enhance the efficiency in utilizing student-generated outputs. Extensive experiments, including instruction-following tasks, demonstrate the effectiveness of DistiLLM in building high-performing student models while achieving up to 4.3$\times$ speedup compared to recent KD methods.<|reference_end|>
@article{ko2024distillm:,
    title={DistiLLM: Towards Streamlined Distillation for Large Language Models},
    author={Jongwoo Ko, Sungnyun Kim, Tianyi Chen, Se-Young Yun},
    journal={arXiv preprint arXiv:2402.03898},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.03898},
    primaryClass={cs.CL cs.AI cs.LG}
}