title: TAID: Temporally Adaptive Interpolated Distillation for Efficient
    Knowledge Transfer in Language Models
summary: Causal language models have demonstrated remarkable capabilities, but their
  size poses significant challenges for deployment in resource-constrained
  environments. Knowledge distillation, a widely-used technique for transferring
  knowledge from a large teacher model to a small student model, presents a
  promising approach for model compression. A significant remaining issue lies in
  the major differences between teacher and student models, namely the
  substantial capacity gap, mode averaging, and mode collapse, which pose
  barriers during distillation. To address these issues, we introduce
  Temporally Adaptive Interpolated Distillation (TAID), a novel
  knowledge distillation approach that dynamically interpolates student and
  teacher distributions through an adaptive intermediate distribution, gradually
  shifting from the student's initial distribution towards the teacher's
  distribution. We provide a theoretical analysis demonstrating TAID's ability to
  prevent mode collapse and empirically show its effectiveness in addressing the
  capacity gap while balancing mode averaging and mode collapse. Our
  comprehensive experiments demonstrate TAID's superior performance across
  various model sizes and architectures in both instruction tuning and
  pre-training scenarios. Furthermore, we showcase TAID's practical impact by
  developing two state-of-the-art compact foundation models:
  TAID-LLM-1.5B for language tasks and TAID-VLM-2B for
  vision-language tasks. These results demonstrate TAID's effectiveness in
  creating high-performing and efficient models, advancing the development of
  more accessible AI technologies.
paper_authors._id: 679990fa121155210e3ac4e0
paper_authors.name: Makoto Shing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T08:40:33.171Z
paper_authors._id: 60c2e7747a42b2edc5d2ccf7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60c2e7747a42b2edc5d2ccf7/5GdUBW1HYEy17orItOqfV.png
paper_authors.fullname: Makoto Shing
paper_authors.type: user
paper_authors.user: mkshing
paper_authors._id: 679990fa121155210e3ac4e1
paper_authors.name: Kou Misaki
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-30T09:36:28.498Z
paper_authors._id: 66601725a7167714d56e1f28
paper_authors.avatarUrl: /avatars/752324a350685027d125f95ef2eea665.svg
paper_authors.fullname: Kou Misaki
paper_authors.type: user
paper_authors.user: takkyu2
paper_authors._id: 679990fa121155210e3ac4e2
paper_authors.name: Han Bao
paper_authors._id: 679990fa121155210e3ac4e3
paper_authors.name: Sho Yokoi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-30T09:36:06.697Z
paper_authors._id: 66c5d59e55c196259b54e383
paper_authors.avatarUrl: /avatars/708f54d2888885f07d6db1505712a173.svg
paper_authors.fullname: Sho Yokoi
paper_authors.type: user
paper_authors.user: eumesy
paper_authors._id: 679990fa121155210e3ac4e4
paper_authors.name: Takuya Akiba
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-30T09:36:00.611Z
paper_authors._id: 6482810dba6c556892f6f257
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6482810dba6c556892f6f257/c7-wiVKenXiRtwnRpnjZN.jpeg
paper_authors.fullname: Takuya Akiba
paper_authors.type: user
paper_authors.user: iwiwi
paper_title: TAID: Temporally Adaptive Interpolated Distillation for Efficient
    Knowledge Transfer in Language Models
paper_summary: Causal language models have demonstrated remarkable capabilities, but their
  size poses significant challenges for deployment in resource-constrained
  environments. Knowledge distillation, a widely-used technique for transferring
  knowledge from a large teacher model to a small student model, presents a
  promising approach for model compression. A significant remaining issue lies in
  the major differences between teacher and student models, namely the
  substantial capacity gap, mode averaging, and mode collapse, which pose
  barriers during distillation. To address these issues, we introduce
  Temporally Adaptive Interpolated Distillation (TAID), a novel
  knowledge distillation approach that dynamically interpolates student and
  teacher distributions through an adaptive intermediate distribution, gradually
  shifting from the student's initial distribution towards the teacher's
  distribution. We provide a theoretical analysis demonstrating TAID's ability to
  prevent mode collapse and empirically show its effectiveness in addressing the
  capacity gap while balancing mode averaging and mode collapse. Our
  comprehensive experiments demonstrate TAID's superior performance across
  various model sizes and architectures in both instruction tuning and
  pre-training scenarios. Furthermore, we showcase TAID's practical impact by
  developing two state-of-the-art compact foundation models:
  TAID-LLM-1.5B for language tasks and TAID-VLM-2B for
  vision-language tasks. These results demonstrate TAID's effectiveness in
  creating high-performing and efficient models, advancing the development of
  more accessible AI technologies.
paper_ai_keywords: causal language models knowledge distillation teacher model student model temporal adaptive interpolated distillation (TAID) adaptive intermediate distribution mode collapse mode steering TAID-LLM-1.5B TAID-VLM-2B pre-training instruction tuning