PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation
Various visual foundation models have distinct strengths and weaknesses, both
of which can be improved through heterogeneous multi-teacher knowledge
distillation without labels, termed "agglomerative models." We build upon this
body of work by studying the effect of the teachers' activation statistics,
particularly the impact of the loss function on the resulting student model
quality. We explore a standard toolkit of statistical normalization techniques
to better align the different distributions and assess their effects. Further,
we examine the impact on downstream teacher-matching metrics, which motivates
the use of Hadamard matrices. With these matrices, we demonstrate useful
properties, showing how they can be used for isotropic standardization, where
each dimension of a multivariate distribution is standardized using the same
scale. We call this technique "PHI Standardization" (PHI-S) and empirically
demonstrate that it produces the best student model across the suite of methods
studied.
66fede8bd5ede3018af8df2d
Mike Ranzinger
extracted_confirmed
2024-10-03T21:01:41.389Z
6424a4c9f1d18f46decb8a15
/avatars/f9cad32cc1672c7e2689ee51fa7a360d.svg
Mike Ranzinger
user
mranzinger
66fede8bd5ede3018af8df2e
Jon Barker
66fede8bd5ede3018af8df2f
Greg Heinrich
admin_assigned
2024-10-04T07:34:31.292Z
64c23e06203c836866ed0bdd
https://cdn-avatars.huggingface.co/v1/production/uploads/64c23e06203c836866ed0bdd/rPUrP8PqL1dhWNO6L3XwI.png
Greg Heinrich
user
gheinrich
66fede8bd5ede3018af8df30
Pavlo Molchanov
claimed_verified
2025-04-20T15:05:11.571Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
66fede8bd5ede3018af8df31
Bryan Catanzaro
admin_assigned
2024-10-04T07:34:40.519Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
66fede8bd5ede3018af8df32
Andrew Tao
PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation
Various visual foundation models have distinct strengths and weaknesses, both
of which can be improved through heterogeneous multi-teacher knowledge
distillation without labels, termed "agglomerative models." We build upon this
body of work by studying the effect of the teachers' activation statistics,
particularly the impact of the loss function on the resulting student model
quality. We explore a standard toolkit of statistical normalization techniques
to better align the different distributions and assess their effects. Further,
we examine the impact on downstream teacher-matching metrics, which motivates
the use of Hadamard matrices. With these matrices, we demonstrate useful
properties, showing how they can be used for isotropic standardization, where
each dimension of a multivariate distribution is standardized using the same
scale. We call this technique "PHI Standardization" (PHI-S) and empirically
demonstrate that it produces the best student model across the suite of methods
studied.
Heterogeneous multi-teacher knowledge distillation improves visual foundation models by optimizing activation statistics and alignment, with PHI Standardization shown to achieve the best student model quality.
heterogeneous multi-teacher knowledge distillation activation statistics loss function statistical normalization techniques Hadamard matrices isotropic standardization PHI Standardization