title: PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation
summary: Various visual foundation models have distinct strengths and weaknesses, both
  of which can be improved through heterogeneous multi-teacher knowledge
  distillation without labels, termed "agglomerative models." We build upon this
  body of work by studying the effect of the teachers' activation statistics,
  particularly the impact of the loss function on the resulting student model
  quality. We explore a standard toolkit of statistical normalization techniques
  to better align the different distributions and assess their effects. Further,
  we examine the impact on downstream teacher-matching metrics, which motivates
  the use of Hadamard matrices. With these matrices, we demonstrate useful
  properties, showing how they can be used for isotropic standardization, where
  each dimension of a multivariate distribution is standardized using the same
  scale. We call this technique "PHI Standardization" (PHI-S) and empirically
  demonstrate that it produces the best student model across the suite of methods
  studied.
paper_authors._id: 66fede8bd5ede3018af8df2d
paper_authors.name: Mike Ranzinger
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-03T21:01:41.389Z
paper_authors._id: 6424a4c9f1d18f46decb8a15
paper_authors.avatarUrl: /avatars/f9cad32cc1672c7e2689ee51fa7a360d.svg
paper_authors.fullname: Mike Ranzinger
paper_authors.type: user
paper_authors.user: mranzinger
paper_authors._id: 66fede8bd5ede3018af8df2e
paper_authors.name: Jon Barker
paper_authors._id: 66fede8bd5ede3018af8df2f
paper_authors.name: Greg Heinrich
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T07:34:31.292Z
paper_authors._id: 64c23e06203c836866ed0bdd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c23e06203c836866ed0bdd/rPUrP8PqL1dhWNO6L3XwI.png
paper_authors.fullname: Greg Heinrich
paper_authors.type: user
paper_authors.user: gheinrich
paper_authors._id: 66fede8bd5ede3018af8df30
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:05:11.571Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_authors._id: 66fede8bd5ede3018af8df31
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T07:34:40.519Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_authors._id: 66fede8bd5ede3018af8df32
paper_authors.name: Andrew Tao
paper_title: PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation
paper_summary: Various visual foundation models have distinct strengths and weaknesses, both
  of which can be improved through heterogeneous multi-teacher knowledge
  distillation without labels, termed "agglomerative models." We build upon this
  body of work by studying the effect of the teachers' activation statistics,
  particularly the impact of the loss function on the resulting student model
  quality. We explore a standard toolkit of statistical normalization techniques
  to better align the different distributions and assess their effects. Further,
  we examine the impact on downstream teacher-matching metrics, which motivates
  the use of Hadamard matrices. With these matrices, we demonstrate useful
  properties, showing how they can be used for isotropic standardization, where
  each dimension of a multivariate distribution is standardized using the same
  scale. We call this technique "PHI Standardization" (PHI-S) and empirically
  demonstrate that it produces the best student model across the suite of methods
  studied.
paper_ai_summary: Heterogeneous multi-teacher knowledge distillation improves visual foundation models by optimizing activation statistics and alignment, with PHI Standardization shown to achieve the best student model quality.
paper_ai_keywords: heterogeneous multi-teacher knowledge distillation activation statistics loss function statistical normalization techniques Hadamard matrices isotropic standardization PHI Standardization