MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
<|reference_start|>MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models: Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical model BERT, they could be further improved in two aspects: the relation-level knowledge could be further explored to improve model performance; and the setting of student attention head number could be more flexible to decrease inference time. Therefore, we are motivated to propose a novel knowledge distillation method MLKD-BERT to distill multi-level knowledge in teacher-student framework. Extensive experiments on GLUE benchmark and extractive question answering tasks demonstrate that our method outperforms state-of-the-art knowledge distillation methods on BERT. In addition, MLKD-BERT can flexibly set student attention head number, allowing for substantial inference time decrease with little performance drop.<|reference_end|>
@article{zhang2024mlkd-bert:,
    title={MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language
  Models},
    author={Ying Zhang and Ziheng Yang and Shufan Ji},
    journal={arXiv preprint arXiv:2407.02775},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.02775},
    primaryClass={cs.CL cs.LG}
}