title: Co-training and Co-distillation for Quality Improvement and Compression
    of Language Models
summary: Knowledge Distillation (KD) compresses computationally expensive pre-trained
  language models (PLMs) by transferring their knowledge to smaller models,
  allowing their use in resource-constrained or real-time settings. However, most
  smaller models fail to surpass the performance of the original larger model,
  resulting in sacrificing performance to improve inference speed. To address
  this issue, we propose Co-Training and Co-Distillation (CTCD), a novel
  framework that improves performance and inference speed together by co-training
  two models while mutually distilling knowledge. The CTCD framework successfully
  achieves this based on two significant findings: 1) Distilling knowledge from
  the smaller model to the larger model during co-training improves the
  performance of the larger model. 2) The enhanced performance of the larger
  model further boosts the performance of the smaller model. The CTCD framework
  shows promise as it can be combined with existing techniques like architecture
  design or data augmentation, replacing one-way KD methods, to achieve further
  performance improvement. Extensive ablation studies demonstrate the
  effectiveness of CTCD, and the small model distilled by CTCD outperforms the
  original larger model by a significant margin of 1.66 on the GLUE benchmark.
paper_authors._id: 6549c7d81e5610f28bd56caa
paper_authors.name: Hayeon Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:16:28.248Z
paper_authors._id: 61a5a9009656217b5e171ac6
paper_authors.avatarUrl: /avatars/6cc949bfa7402fc4b2efd498b1ecdbde.svg
paper_authors.fullname: HayeonLee
paper_authors.type: user
paper_authors.user: HayeonLee
paper_authors._id: 6549c7d81e5610f28bd56cab
paper_authors.name: Rui Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:16:36.921Z
paper_authors._id: 6407d236a7bc7c3865adcc3a
paper_authors.avatarUrl: /avatars/8b75cc07fd049acde5363d268c5b88d4.svg
paper_authors.fullname: Rui Hou
paper_authors.type: user
paper_authors.user: rayhou
paper_authors._id: 6549c7d81e5610f28bd56cac
paper_authors.name: Jongpil Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:16:46.267Z
paper_authors._id: 64218cd42143035270d919ce
paper_authors.avatarUrl: /avatars/8dfb17d124ee957782b96d5daacbc1f1.svg
paper_authors.fullname: Kim
paper_authors.type: user
paper_authors.user: jongpill
paper_authors._id: 6549c7d81e5610f28bd56cad
paper_authors.name: Davis Liang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-07T05:15:35.567Z
paper_authors._id: 64b6e1a9f1603ffd84dd09ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b6e1a9f1603ffd84dd09ba/uL39d0Eojf0ADSCP0lfr4.png
paper_authors.fullname: Davis Liang
paper_authors.type: user
paper_authors.user: davisliang
paper_authors._id: 6549c7d81e5610f28bd56cae
paper_authors.name: Hongbo Zhang
paper_authors._id: 6549c7d81e5610f28bd56caf
paper_authors.name: Sung Ju Hwang
paper_authors._id: 6549c7d81e5610f28bd56cb0
paper_authors.name: Alexander Min
paper_title: Co-training and Co-distillation for Quality Improvement and Compression
    of Language Models
paper_summary: Knowledge Distillation (KD) compresses computationally expensive pre-trained
  language models (PLMs) by transferring their knowledge to smaller models,
  allowing their use in resource-constrained or real-time settings. However, most
  smaller models fail to surpass the performance of the original larger model,
  resulting in sacrificing performance to improve inference speed. To address
  this issue, we propose Co-Training and Co-Distillation (CTCD), a novel
  framework that improves performance and inference speed together by co-training
  two models while mutually distilling knowledge. The CTCD framework successfully
  achieves this based on two significant findings: 1) Distilling knowledge from
  the smaller model to the larger model during co-training improves the
  performance of the larger model. 2) The enhanced performance of the larger
  model further boosts the performance of the smaller model. The CTCD framework
  shows promise as it can be combined with existing techniques like architecture
  design or data augmentation, replacing one-way KD methods, to achieve further
  performance improvement. Extensive ablation studies demonstrate the
  effectiveness of CTCD, and the small model distilled by CTCD outperforms the
  original larger model by a significant margin of 1.66 on the GLUE benchmark.
paper_ai_summary: A novel co-training and co-distillation framework enhances performance and inference speed of smaller models through mutual knowledge transfer with larger models, outperforming traditional knowledge distillation methods.
paper_ai_keywords: knowledge distillation pre-trained language models co-training co-distillation mutual distillation GLUE benchmark