Answers.Body: <p><strong>Knowledge Distillation</strong> refers to using a teacher model and distilling its knowlege to a student model, mostly done by the teacher providing soft labels for the student model to create loss. So basically it defines the <strong>action of using a teacher model to teach the student</strong>.</p>
  <p><strong>Mean Teacher</strong>, on the other hand, is one way of how one would define/train the teacher (especially in self supervised learning). Usually in supervised learning, one can train the teacher using existing labels, but in self supervised learning this is not possible. Mean Teacher defines the teacher as a model that utilizes the weighted average of the student's past weights. So basically it is <strong>one methodology of defining the teacher in self supervised knowledge distillation</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T16:06:15.937Z
Answers.Text: Thank you for your comment ... So in MT, there is no distilled knowledge (e.g : transfer set of soft labels) from the teacher to the student because the teacher also needs to learn? And both teacher and student are trained together via parameters update (from loss  backprogation for the student and EMA for the teacher)
Answers.UserDisplayName: Đặng Huy Hoàng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T16:06:52.200Z
Answers.Text: @ĐặngHuyHoàng correct
Answers.UserDisplayName: DKDK
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-18T15:15:48.090Z
Answers.Text: @ĐặngHuyHoàng this is an answer, and not a comment; please see [What should I do when someone answers my question?](https://ai.stackexchange.com/help/someone-answers)
Answers.UserDisplayName: desertnaut
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T06:53:47.633Z
Answers.LastActivityDate: 2022-01-06T06:53:47.633Z
Answers.UserDisplayName: DKDK
Body: <p>I recently read two papers:</p>
  <ul>
  <li>BYOL <a href="https://arxiv.org/abs/2006.07733" rel="nofollow noreferrer">Bootstrap your own latent: A new approach to self-supervised Learning</a></li>
  </ul>
  <p><a href="https://i.sstatic.net/6YUz8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6YUz8.png" alt="BYOL_arch" /></a></p>
  <ul>
  <li>DINO <a href="https://arxiv.org/abs/2104.14294" rel="nofollow noreferrer">Emerging Properties in Self-Supervised Vision Transformers</a>.</li>
  </ul>
  <p><a href="https://i.sstatic.net/AvYWU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AvYWU.png" alt="DINO_arch" /></a></p>
  <p>I am confused about the terms <strong>Mean Teacher</strong> in BYOL and <strong>Knowledge Distillation</strong> in DINO.</p>
  <p>Is KD the same as MT but using the cross-entropy loss instead of mean square error (since MT has preditor head while KD only has softmax head)?</p>
  
Tags: <deep-learning><comparison><vision-transformer><contrastive-learning>
Title: What is the difference between Mean Teacher and Knowledge Distillation?
UserDisplayName: Đặng Huy Hoàng
ThreadText: # What is the difference between Mean Teacher and Knowledge Distillation?
  
  (Asked by: username_0 on 2022-01-06)
  
  I recently read two papers:
  
  * BYOL [Bootstrap your own latent: A new approach to self-supervised Learning](https://arxiv.org/abs/2006.07733)
  
  [![BYOL_arch](https://i.sstatic.net/6YUz8.png)](https://i.sstatic.net/6YUz8.png)
  
  * DINO [Emerging Properties in Self-Supervised Vision Transformers](https://arxiv.org/abs/2104.14294).
  
  [![DINO_arch](https://i.sstatic.net/AvYWU.png)](https://i.sstatic.net/AvYWU.png)
  
  I am confused about the terms **Mean Teacher** in BYOL and **Knowledge Distillation** in DINO.
  
  Is KD the same as MT but using the cross-entropy loss instead of mean square error (since MT has preditor head while KD only has softmax head)?
  
  **Tags:** deep-learning, comparison, vision-transformer, contrastive-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-06. Score: 1
  
  **Knowledge Distillation** refers to using a teacher model and distilling its knowlege to a student model, mostly done by the teacher providing soft labels for the student model to create loss. So basically it defines the **action of using a teacher model to teach the student**.
  
  **Mean Teacher**, on the other hand, is one way of how one would define/train the teacher (especially in self supervised learning). Usually in supervised learning, one can train the teacher using existing labels, but in self supervised learning this is not possible. Mean Teacher defines the teacher as a model that utilizes the weighted average of the student's past weights. So basically it is **one methodology of defining the teacher in self supervised knowledge distillation**.
  
  > **Comments:**
  > **username_0**: Thank you for your comment ... So in MT, there is no distilled knowledge (e.g : transfer set of soft labels) from the teacher to the student because the teacher also needs to learn? And both teacher and student are trained together via parameters update (from loss  backprogation for the student and EMA for the teacher)  
  > **username_1**: @ĐặngHuyHoàng correct  
  > **username_2**: @ĐặngHuyHoàng this is an answer, and not a comment; please see [What should I do when someone answers my question?](https://ai.stackexchange.com/help/someone-answers)  
  
  ---
  
  