Answers.Body: <p>Short answer: yes, the teacher model should always train with noise.</p>
  <p>The noise here is just the augmentation and regularization that you may feel familiar with, the author used three kinds of noises:</p>
  <ul>
  <li>Data augmentation</li>
  <li>Dropout</li>
  <li>Stochastic Depth (drop out for residual block).</li>
  </ul>
  <p>In step 2, the authors said that only the sample which has a high score (high confidence)</p>
  <blockquote>
  <p>Specifically, we filter images that the
  teacher model has low confidences on since they are usually out-of-domain images.</p>
  </blockquote>
  <p>And they even add or remove samples to ensure the distribution of the dataset</p>
  <blockquote>
  <p>we duplicate images in classes where there are not enough images. For classes where we have too many images, we take the images with the highest confidence.</p>
  </blockquote>
  <p>And the most important thing, they use the bigger (more capacity, more parameters) model as a student</p>
  <blockquote>
  <p>we want the student to be better than the teacher by giving the student model enough capacity</p>
  </blockquote>
  <p>Now, just think simple, those methods help to enhance the model, which makes the model stronger so teachers, students, or any model should apply those to the training phase and remove them to predict at the test phase (teacher predicts labels).</p>
  <p>In my opinion, this method is just like <a href="https://paperswithcode.com/method/label-smoothing" rel="nofollow noreferrer">label smoothing</a>  but replaces the hyper-parameters with the learning one (the teacher).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-01T06:54:19.940Z
Answers.LastActivityDate: 2022-04-01T06:54:19.940Z
Answers.UserDisplayName: CuCaRot
Body: <p>Reading through the <a href="https://arxiv.org/abs/1911.04252" rel="nofollow noreferrer">paper</a> on the Noisy Student algorithm, I have a quick question about how the initial teacher model is built.</p>
  <p>In step 1 of the algorithm, the loss function is defined such that it looks like the initial teacher model is trained using noise. But then when you get to step 2 it says the teacher model you use to generate your labels for the unlabeled data should not be noised.</p>
  <p>So, should you be adding noise or not for the first teacher model that you train?</p>
  
Tags: <neural-networks><papers><semi-supervised-learning><image-net>
Title: Is the initial teacher model in the Noisy Student algorithm noised?
UserDisplayName: lamyvista
ThreadText: # Is the initial teacher model in the Noisy Student algorithm noised?
  
  (Asked by: username_0 on 2022-03-28)
  
  Reading through the [paper](https://arxiv.org/abs/1911.04252) on the Noisy Student algorithm, I have a quick question about how the initial teacher model is built.
  
  In step 1 of the algorithm, the loss function is defined such that it looks like the initial teacher model is trained using noise. But then when you get to step 2 it says the teacher model you use to generate your labels for the unlabeled data should not be noised.
  
  So, should you be adding noise or not for the first teacher model that you train?
  
  **Tags:** neural-networks, papers, semi-supervised-learning, image-net
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-04-01. Score: 0 | Accepted answer
  
  Short answer: yes, the teacher model should always train with noise.
  
  The noise here is just the augmentation and regularization that you may feel familiar with, the author used three kinds of noises:
  
  * Data augmentation
  * Dropout
  * Stochastic Depth (drop out for residual block).
  
  In step 2, the authors said that only the sample which has a high score (high confidence)
  
  > Specifically, we filter images that the
  > teacher model has low confidences on since they are usually out-of-domain images.
  
  And they even add or remove samples to ensure the distribution of the dataset
  
  > we duplicate images in classes where there are not enough images. For classes where we have too many images, we take the images with the highest confidence.
  
  And the most important thing, they use the bigger (more capacity, more parameters) model as a student
  
  > we want the student to be better than the teacher by giving the student model enough capacity
  
  Now, just think simple, those methods help to enhance the model, which makes the model stronger so teachers, students, or any model should apply those to the training phase and remove them to predict at the test phase (teacher predicts labels).
  
  In my opinion, this method is just like [label smoothing](https://paperswithcode.com/method/label-smoothing) but replaces the hyper-parameters with the learning one (the teacher).
  
  
  ---
  
  