Co-training and Co-distillation for Quality Improvement and Compression
  of Language Models
Knowledge Distillation (KD) compresses computationally expensive pre-trained
language models (PLMs) by transferring their knowledge to smaller models,
allowing their use in resource-constrained or real-time settings. However, most
smaller models fail to surpass the performance of the original larger model,
resulting in sacrificing performance to improve inference speed. To address
this issue, we propose Co-Training and Co-Distillation (CTCD), a novel
framework that improves performance and inference speed together by co-training
two models while mutually distilling knowledge. The CTCD framework successfully
achieves this based on two significant findings: 1) Distilling knowledge from
the smaller model to the larger model during co-training improves the
performance of the larger model. 2) The enhanced performance of the larger
model further boosts the performance of the smaller model. The CTCD framework
shows promise as it can be combined with existing techniques like architecture
design or data augmentation, replacing one-way KD methods, to achieve further
performance improvement. Extensive ablation studies demonstrate the
effectiveness of CTCD, and the small model distilled by CTCD outperforms the
original larger model by a significant margin of 1.66 on the GLUE benchmark.
6549c7d81e5610f28bd56caa
Hayeon Lee
admin_assigned
2023-11-07T17:16:28.248Z
61a5a9009656217b5e171ac6
/avatars/6cc949bfa7402fc4b2efd498b1ecdbde.svg
HayeonLee
user
HayeonLee
6549c7d81e5610f28bd56cab
Rui Hou
admin_assigned
2023-11-07T17:16:36.921Z
6407d236a7bc7c3865adcc3a
/avatars/8b75cc07fd049acde5363d268c5b88d4.svg
Rui Hou
user
rayhou
6549c7d81e5610f28bd56cac
Jongpil Kim
admin_assigned
2023-11-07T17:16:46.267Z
64218cd42143035270d919ce
/avatars/8dfb17d124ee957782b96d5daacbc1f1.svg
Kim
user
jongpill
6549c7d81e5610f28bd56cad
Davis Liang
extracted_confirmed
2023-11-07T05:15:35.567Z
64b6e1a9f1603ffd84dd09ba
https://cdn-avatars.huggingface.co/v1/production/uploads/64b6e1a9f1603ffd84dd09ba/uL39d0Eojf0ADSCP0lfr4.png
Davis Liang
user
davisliang
6549c7d81e5610f28bd56cae
Hongbo Zhang
6549c7d81e5610f28bd56caf
Sung Ju Hwang
6549c7d81e5610f28bd56cb0
Alexander Min
Co-training and Co-distillation for Quality Improvement and Compression
  of Language Models
Knowledge Distillation (KD) compresses computationally expensive pre-trained
language models (PLMs) by transferring their knowledge to smaller models,
allowing their use in resource-constrained or real-time settings. However, most
smaller models fail to surpass the performance of the original larger model,
resulting in sacrificing performance to improve inference speed. To address
this issue, we propose Co-Training and Co-Distillation (CTCD), a novel
framework that improves performance and inference speed together by co-training
two models while mutually distilling knowledge. The CTCD framework successfully
achieves this based on two significant findings: 1) Distilling knowledge from
the smaller model to the larger model during co-training improves the
performance of the larger model. 2) The enhanced performance of the larger
model further boosts the performance of the smaller model. The CTCD framework
shows promise as it can be combined with existing techniques like architecture
design or data augmentation, replacing one-way KD methods, to achieve further
performance improvement. Extensive ablation studies demonstrate the
effectiveness of CTCD, and the small model distilled by CTCD outperforms the
original larger model by a significant margin of 1.66 on the GLUE benchmark.
A novel co-training and co-distillation framework enhances performance and inference speed of smaller models through mutual knowledge transfer with larger models, outperforming traditional knowledge distillation methods.
knowledge distillation pre-trained language models co-training co-distillation mutual distillation GLUE benchmark