Answers.Body: <p>Suppose you have two tokens that are equally likely. In your original training data, your next-token predictions are</p>
  <p><span class="math-container">$$[1, 0]\quad\text{and}\quad [0, 1]$$</span></p>
  <p>an equal number of times. The entropy of each of these samples is zero. However, a fully trained network should converge to</p>
  <p><span class="math-container">$$[0.5, 0.5],$$</span></p>
  <p>which has an entropy of one bit. A second network trained on these higher-entropy samples will see less variance in its KL-loss. Also, because the learned network is essentially collating a whole lot of training samples together, we get more information per sample.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-23T17:15:48.610Z
Answers.LastActivityDate: 2024-04-23T17:15:48.610Z
Answers.UserDisplayName: programjames
Answers.Body: <p>Since it is a trained network already, when you run an example through it, the gradient will not have a very high variance.</p>
  <p>The gradient varies a lot when you are training a network from the scratch but then it stops varying much since it understands the pattern.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-06T02:18:56.143Z
Answers.LastActivityDate: 2021-04-09T03:23:14.240Z
Answers.LastEditDate: 2021-04-09T03:23:14.240Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>I've been trying to understand the <a href="https://arxiv.org/pdf/1503.02531.pdf" rel="nofollow noreferrer">Distilling the Knowledge in a Neural Network</a> paper by Hinton et al. But I cannot fully understand this:</p>
  <blockquote>
  <p>When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases [...]</p>
  </blockquote>
  <p>The information part is very clear, but how does high entropy correlate to less variance between training cases?</p>
  
Tags: <deep-learning><papers><variance><entropy>
Title: How does high entropy targets relate to less variance of the gradient between training cases?
UserDisplayName: Userr2232
ThreadText: # How does high entropy targets relate to less variance of the gradient between training cases?
  
  (Asked by: username_0 on 2021-04-05)
  
  I've been trying to understand the [Distilling the Knowledge in a Neural Network](https://arxiv.org/pdf/1503.02531.pdf) paper by Hinton et al. But I cannot fully understand this:
  
  > When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases [...]
  
  The information part is very clear, but how does high entropy correlate to less variance between training cases?
  
  **Tags:** deep-learning, papers, variance, entropy
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-04-23. Score: 0
  
  Suppose you have two tokens that are equally likely. In your original training data, your next-token predictions are
  
  $$[1, 0]\quad\text{and}\quad [0, 1]$$
  
  an equal number of times. The entropy of each of these samples is zero. However, a fully trained network should converge to
  
  $$[0.5, 0.5],$$
  
  which has an entropy of one bit. A second network trained on these higher-entropy samples will see less variance in its KL-loss. Also, because the learned network is essentially collating a whole lot of training samples together, we get more information per sample.
  
  
  ---
  
  ### Answer by username_2 on 2021-04-06. Score: 0
  
  Since it is a trained network already, when you run an example through it, the gradient will not have a very high variance.
  
  The gradient varies a lot when you are training a network from the scratch but then it stops varying much since it understands the pattern.
  
  
  ---
  
  