title: Improving Knowledge Distillation Under Unknown Covariate Shift Through
    Confidence-Guided Data Augmentation
summary: Large foundation models trained on extensive datasets demonstrate strong
  zero-shot capabilities in various domains. To replicate their success when data
  and model size are constrained, knowledge distillation has become an
  established tool for transferring knowledge from foundation models to small
  student networks. However, the effectiveness of distillation is critically
  limited by the available training data. This work addresses the common
  practical issue of covariate shift in knowledge distillation, where spurious
  features appear during training but not at test time. We ask the question: when
  these spurious features are unknown, yet a robust teacher is available, is it
  possible for a student to also become robust to them? We address this problem
  by introducing a novel diffusion-based data augmentation strategy that
  generates images by maximizing the disagreement between the teacher and the
  student, effectively creating challenging samples that the student struggles
  with. Experiments demonstrate that our approach significantly improves worst
  group and mean group accuracy on CelebA and SpuCo Birds as well as the spurious
  mAUC on spurious ImageNet under covariate shift, outperforming state-of-the-art
  diffusion-based data augmentation baselines
paper_authors._id: 6840cd169241913d43af9d28
paper_authors.name: Niclas Popp
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T13:00:50.249Z
paper_authors._id: 655646baf8a2d3c020546ec8
paper_authors.avatarUrl: /avatars/4ca8de82745bb5a4fda511569bb6bd94.svg
paper_authors.fullname: Niclas P
paper_authors.type: user
paper_authors.user: NPBP26
paper_authors._id: 6840cd169241913d43af9d29
paper_authors.name: Kevin Alexander Laube
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T13:00:56.090Z
paper_authors._id: 662a1098fc86170808ac598d
paper_authors.avatarUrl: /avatars/91b4910c96f21468c97c97cdfac2ab53.svg
paper_authors.fullname: Kevin Alexander Laube
paper_authors.type: user
paper_authors.user: laubekebosch
paper_authors._id: 6840cd169241913d43af9d2a
paper_authors.name: Matthias Hein
paper_authors._id: 6840cd169241913d43af9d2b
paper_authors.name: Lukas Schott
paper_title: Improving Knowledge Distillation Under Unknown Covariate Shift Through
    Confidence-Guided Data Augmentation
paper_summary: Large foundation models trained on extensive datasets demonstrate strong
  zero-shot capabilities in various domains. To replicate their success when data
  and model size are constrained, knowledge distillation has become an
  established tool for transferring knowledge from foundation models to small
  student networks. However, the effectiveness of distillation is critically
  limited by the available training data. This work addresses the common
  practical issue of covariate shift in knowledge distillation, where spurious
  features appear during training but not at test time. We ask the question: when
  these spurious features are unknown, yet a robust teacher is available, is it
  possible for a student to also become robust to them? We address this problem
  by introducing a novel diffusion-based data augmentation strategy that
  generates images by maximizing the disagreement between the teacher and the
  student, effectively creating challenging samples that the student struggles
  with. Experiments demonstrate that our approach significantly improves worst
  group and mean group accuracy on CelebA and SpuCo Birds as well as the spurious
  mAUC on spurious ImageNet under covariate shift, outperforming state-of-the-art
  diffusion-based data augmentation baselines
paper_ai_summary: A diffusion-based data augmentation strategy improves robustness in knowledge distillation by generating challenging samples, enhancing accuracy and spurious feature resilience.
paper_ai_keywords: knowledge distillation diffusion-based data augmentation covariate shift teacher-student model CelebA SpuCo Birds spurious ImageNet mean group accuracy worst group accuracy spurious mAUC