TAID: Temporally Adaptive Interpolated Distillation for Efficient
  Knowledge Transfer in Language Models
Causal language models have demonstrated remarkable capabilities, but their
size poses significant challenges for deployment in resource-constrained
environments. Knowledge distillation, a widely-used technique for transferring
knowledge from a large teacher model to a small student model, presents a
promising approach for model compression. A significant remaining issue lies in
the major differences between teacher and student models, namely the
substantial capacity gap, mode averaging, and mode collapse, which pose
barriers during distillation. To address these issues, we introduce
Temporally Adaptive Interpolated Distillation (TAID), a novel
knowledge distillation approach that dynamically interpolates student and
teacher distributions through an adaptive intermediate distribution, gradually
shifting from the student's initial distribution towards the teacher's
distribution. We provide a theoretical analysis demonstrating TAID's ability to
prevent mode collapse and empirically show its effectiveness in addressing the
capacity gap while balancing mode averaging and mode collapse. Our
comprehensive experiments demonstrate TAID's superior performance across
various model sizes and architectures in both instruction tuning and
pre-training scenarios. Furthermore, we showcase TAID's practical impact by
developing two state-of-the-art compact foundation models:
TAID-LLM-1.5B for language tasks and TAID-VLM-2B for
vision-language tasks. These results demonstrate TAID's effectiveness in
creating high-performing and efficient models, advancing the development of
more accessible AI technologies.
679990fa121155210e3ac4e0
Makoto Shing
claimed_verified
2025-01-30T08:40:33.171Z
60c2e7747a42b2edc5d2ccf7
https://cdn-avatars.huggingface.co/v1/production/uploads/60c2e7747a42b2edc5d2ccf7/FGlJt3Zd6NEcE-DKC3-dL.jpeg
Makoto Shing
user
mkshing
679990fa121155210e3ac4e1
Kou Misaki
admin_assigned
2025-01-30T09:36:28.498Z
66601725a7167714d56e1f28
/avatars/752324a350685027d125f95ef2eea665.svg
Kou Misaki
user
takkyu2
679990fa121155210e3ac4e2
Han Bao
679990fa121155210e3ac4e3
Sho Yokoi
admin_assigned
2025-01-30T09:36:06.697Z
66c5d59e55c196259b54e383
/avatars/708f54d2888885f07d6db1505712a173.svg
Sho Yokoi
user
eumesy
679990fa121155210e3ac4e4
Takuya Akiba
admin_assigned
2025-01-30T09:36:00.611Z
6482810dba6c556892f6f257
https://cdn-avatars.huggingface.co/v1/production/uploads/6482810dba6c556892f6f257/c7-wiVKenXiRtwnRpnjZN.jpeg
Takuya Akiba
user
iwiwi
TAID: Temporally Adaptive Interpolated Distillation for Efficient
  Knowledge Transfer in Language Models
Causal language models have demonstrated remarkable capabilities, but their
size poses significant challenges for deployment in resource-constrained
environments. Knowledge distillation, a widely-used technique for transferring
knowledge from a large teacher model to a small student model, presents a
promising approach for model compression. A significant remaining issue lies in
the major differences between teacher and student models, namely the
substantial capacity gap, mode averaging, and mode collapse, which pose
barriers during distillation. To address these issues, we introduce
Temporally Adaptive Interpolated Distillation (TAID), a novel
knowledge distillation approach that dynamically interpolates student and
teacher distributions through an adaptive intermediate distribution, gradually
shifting from the student's initial distribution towards the teacher's
distribution. We provide a theoretical analysis demonstrating TAID's ability to
prevent mode collapse and empirically show its effectiveness in addressing the
capacity gap while balancing mode averaging and mode collapse. Our
comprehensive experiments demonstrate TAID's superior performance across
various model sizes and architectures in both instruction tuning and
pre-training scenarios. Furthermore, we showcase TAID's practical impact by
developing two state-of-the-art compact foundation models:
TAID-LLM-1.5B for language tasks and TAID-VLM-2B for
vision-language tasks. These results demonstrate TAID's effectiveness in
creating high-performing and efficient models, advancing the development of
more accessible AI technologies.
Temporally Adaptive Interpolated Distillation (TAID) addresses capacity gap, mode averaging, and mode collapse between large and small models, enhancing knowledge distillation and leading to efficient compact foundation models for language and vision-language tasks.
causal language models knowledge distillation teacher model student model capacity gap mode averaging mode collapse Temporally Adaptive Interpolated Distillation (TAID) adaptive intermediate distribution instruction tuning pre-training TAID-LLM-1.5B TAID-VLM-2B