Answers.Body: <p>There are multiple confounding factors here. Firstly, learning from the mistakes is not necessarily bad if encouraging the student model to make these mistakes improves its generalisation<sup>†</sup>. This can be a result of there being an insufficient sampling around these incorrectly predicted data points, incorrect human-labelling, or these points are just too hard to classify anyway.</p>
  <p>If you train a student network solely on the incorrect predictions of the teacher (no ground truth) it will generalise to some extent. This is because the distillation process is doing some form of function matching. Interpolating between these incorrect predictions is then a reasonable function for classifying the training data, hence the student will non-trivially generalises.</p>
  <p>Another perspective is that regardless of these two methods (hard or soft), the teacher is still transferring the translational equivariance to the student, which is a strong inductive bias for generalisation.</p>
  <p>In summary, although the correlation between classes of the teacher predictions is one important component attributed to the success of distillation, there are a lot more factors to consider which then explain why hard distillation works.</p>
  <p><sup>†</sup>: Alternatively, we can say the student may accurately predict more of the training data on average by making these few mistakes anyway.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-21T14:47:22.893Z
Answers.LastActivityDate: 2023-03-21T15:40:40.607Z
Answers.LastEditDate: 2023-03-21T15:40:40.607Z
Answers.UserDisplayName: Roy
Body: <p>In order to get a smaller model, one often uses larger model, that performs reasonably well on the data as a teacher, and uses the information from large model to train the smaller one.</p>
  <p>There are several strategies to do this:</p>
  <ul>
  <li><p>Soft distillation</p>
  <p>Given logits of the teacher one adds the KL-divergence between the student logits and teacher logits to  the loss:
  <span class="math-container">$$
  \mathcal{L}_{loss} = (1 - \alpha) \mathcal{L}_{BCE} (y_{student}, y_{true}) + 
  \lambda  \mathcal{L}_{KL} (y_{student}, y_{teacher})
  $$</span>
  Intuition behind this approach is clear - logits are more informative than a single target label and seemingly allow for faster training.</p>
  </li>
  <li><p>Hard distillation</p>
  <p>One adds the <code>BCE</code> between student logits and teacher model outputs as if they were true labels.
  <span class="math-container">$$
  \mathcal{L}_{loss} = (1 - \alpha) \mathcal{L}_{BCE} (y_{student}, y_{true}) + 
  \lambda  \mathcal{L}_{BCE} (y_{student}, y_{teacher})
  $$</span></p>
  </li>
  </ul>
  <p>And the benefit of the last approach is unclear to me. For the perfect model, one will have no difference with the vanilla training procedure, and for the case, where the teacher makes mistakes, we will optimize the wrong objective.</p>
  <p>Despite these concerns, it was shown experimentally in several papers, and in
  the <a href="https://arxiv.org/abs/2012.12877" rel="nofollow noreferrer">Deit</a>, that this objective can improve performance. Even more, it is better, than soft distillation.</p>
  <p><a href="https://i.sstatic.net/E3cKi.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/E3cKi.png" alt="enter image description here" /></a></p>
  <p>Why can this be the case?</p>
  
Tags: <image-recognition><transformer>
Title: What is the purpose of hard distillation?
UserDisplayName: spiridon_the_sun_rotator
ThreadText: # What is the purpose of hard distillation?
  
  (Asked by: username_0 on 2021-10-13)
  
  In order to get a smaller model, one often uses larger model, that performs reasonably well on the data as a teacher, and uses the information from large model to train the smaller one.
  
  There are several strategies to do this:
  
  * Soft distillation
  
    Given logits of the teacher one adds the KL-divergence between the student logits and teacher logits to the loss:
    $$
    \mathcal{L}\_{loss} = (1 - \alpha) \mathcal{L}\_{BCE} (y\_{student}, y\_{true}) +
    \lambda \mathcal{L}\_{KL} (y\_{student}, y\_{teacher})
    $$
    Intuition behind this approach is clear - logits are more informative than a single target label and seemingly allow for faster training.
  * Hard distillation
  
    One adds the `BCE` between student logits and teacher model outputs as if they were true labels.
    $$
    \mathcal{L}\_{loss} = (1 - \alpha) \mathcal{L}\_{BCE} (y\_{student}, y\_{true}) +
    \lambda \mathcal{L}\_{BCE} (y\_{student}, y\_{teacher})
    $$
  
  And the benefit of the last approach is unclear to me. For the perfect model, one will have no difference with the vanilla training procedure, and for the case, where the teacher makes mistakes, we will optimize the wrong objective.
  
  Despite these concerns, it was shown experimentally in several papers, and in
  the [Deit](https://arxiv.org/abs/2012.12877), that this objective can improve performance. Even more, it is better, than soft distillation.
  
  [![enter image description here](https://i.sstatic.net/E3cKi.png)](https://i.sstatic.net/E3cKi.png)
  
  Why can this be the case?
  
  **Tags:** image-recognition, transformer
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-21. Score: 1
  
  There are multiple confounding factors here. Firstly, learning from the mistakes is not necessarily bad if encouraging the student model to make these mistakes improves its generalisation†. This can be a result of there being an insufficient sampling around these incorrectly predicted data points, incorrect human-labelling, or these points are just too hard to classify anyway.
  
  If you train a student network solely on the incorrect predictions of the teacher (no ground truth) it will generalise to some extent. This is because the distillation process is doing some form of function matching. Interpolating between these incorrect predictions is then a reasonable function for classifying the training data, hence the student will non-trivially generalises.
  
  Another perspective is that regardless of these two methods (hard or soft), the teacher is still transferring the translational equivariance to the student, which is a strong inductive bias for generalisation.
  
  In summary, although the correlation between classes of the teacher predictions is one important component attributed to the success of distillation, there are a lot more factors to consider which then explain why hard distillation works.
  
  †: Alternatively, we can say the student may accurately predict more of the training data on average by making these few mistakes anyway.
  
  
  ---
  
  