Improving Knowledge Distillation Under Unknown Covariate Shift Through
  Confidence-Guided Data Augmentation
Large foundation models trained on extensive datasets demonstrate strong
zero-shot capabilities in various domains. To replicate their success when data
and model size are constrained, knowledge distillation has become an
established tool for transferring knowledge from foundation models to small
student networks. However, the effectiveness of distillation is critically
limited by the available training data. This work addresses the common
practical issue of covariate shift in knowledge distillation, where spurious
features appear during training but not at test time. We ask the question: when
these spurious features are unknown, yet a robust teacher is available, is it
possible for a student to also become robust to them? We address this problem
by introducing a novel diffusion-based data augmentation strategy that
generates images by maximizing the disagreement between the teacher and the
student, effectively creating challenging samples that the student struggles
with. Experiments demonstrate that our approach significantly improves worst
group and mean group accuracy on CelebA and SpuCo Birds as well as the spurious
mAUC on spurious ImageNet under covariate shift, outperforming state-of-the-art
diffusion-based data augmentation baselines
6840cd169241913d43af9d28
Niclas Popp
admin_assigned
2025-06-05T13:00:50.249Z
655646baf8a2d3c020546ec8
/avatars/4ca8de82745bb5a4fda511569bb6bd94.svg
Niclas P
user
NPBP26
6840cd169241913d43af9d29
Kevin Alexander Laube
admin_assigned
2025-06-05T13:00:56.090Z
662a1098fc86170808ac598d
/avatars/91b4910c96f21468c97c97cdfac2ab53.svg
Kevin Alexander Laube
user
laubekebosch
6840cd169241913d43af9d2a
Matthias Hein
6840cd169241913d43af9d2b
Lukas Schott
Improving Knowledge Distillation Under Unknown Covariate Shift Through
  Confidence-Guided Data Augmentation
Large foundation models trained on extensive datasets demonstrate strong
zero-shot capabilities in various domains. To replicate their success when data
and model size are constrained, knowledge distillation has become an
established tool for transferring knowledge from foundation models to small
student networks. However, the effectiveness of distillation is critically
limited by the available training data. This work addresses the common
practical issue of covariate shift in knowledge distillation, where spurious
features appear during training but not at test time. We ask the question: when
these spurious features are unknown, yet a robust teacher is available, is it
possible for a student to also become robust to them? We address this problem
by introducing a novel diffusion-based data augmentation strategy that
generates images by maximizing the disagreement between the teacher and the
student, effectively creating challenging samples that the student struggles
with. Experiments demonstrate that our approach significantly improves worst
group and mean group accuracy on CelebA and SpuCo Birds as well as the spurious
mAUC on spurious ImageNet under covariate shift, outperforming state-of-the-art
diffusion-based data augmentation baselines
A diffusion-based data augmentation strategy improves robustness in knowledge distillation by generating challenging samples, enhancing accuracy and spurious feature resilience.
knowledge distillation diffusion-based data augmentation covariate shift teacher-student model CelebA SpuCo Birds spurious ImageNet mean group accuracy worst group accuracy spurious mAUC