<p>Suppose you have two tokens that are equally likely. In your original training data, your next-token predictions are</p>
<p><span class="math-container">$$[1, 0]\quad\text{and}\quad [0, 1]$$</span></p>
<p>an equal number of times. The entropy of each of these samples is zero. However, a fully trained network should converge to</p>
<p><span class="math-container">$$[0.5, 0.5],$$</span></p>
<p>which has an entropy of one bit. A second network trained on these higher-entropy samples will see less variance in its KL-loss. Also, because the learned network is essentially collating a whole lot of training samples together, we get more information per sample.</p>

CC BY-SA 4.0
2024-04-23T17:15:48.610Z
2024-04-23T17:15:48.610Z
programjames
<p>Since it is a trained network already, when you run an example through it, the gradient will not have a very high variance.</p>
<p>The gradient varies a lot when you are training a network from the scratch but then it stops varying much since it understands the pattern.</p>

CC BY-SA 4.0
2021-04-06T02:18:56.143Z
2021-04-09T03:23:14.240Z
2021-04-09T03:23:14.240Z
Abhishek Verma
<p>I've been trying to understand the <a href="https://arxiv.org/pdf/1503.02531.pdf" rel="nofollow noreferrer">Distilling the Knowledge in a Neural Network</a> paper by Hinton et al. But I cannot fully understand this:</p>
<blockquote>
<p>When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases [...]</p>
</blockquote>
<p>The information part is very clear, but how does high entropy correlate to less variance between training cases?</p>

<deep-learning><papers><variance><entropy>
How does high entropy targets relate to less variance of the gradient between training cases?
Userr2232
# How does high entropy targets relate to less variance of the gradient between training cases?

(Asked by: username_0 on 2021-04-05)

I've been trying to understand the [Distilling the Knowledge in a Neural Network](https://arxiv.org/pdf/1503.02531.pdf) paper by Hinton et al. But I cannot fully understand this:

> When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases [...]

The information part is very clear, but how does high entropy correlate to less variance between training cases?

**Tags:** deep-learning, papers, variance, entropy


---

## 2 answers

### Answer by username_1 on 2024-04-23. Score: 0

Suppose you have two tokens that are equally likely. In your original training data, your next-token predictions are

$$[1, 0]\quad\text{and}\quad [0, 1]$$

an equal number of times. The entropy of each of these samples is zero. However, a fully trained network should converge to

$$[0.5, 0.5],$$

which has an entropy of one bit. A second network trained on these higher-entropy samples will see less variance in its KL-loss. Also, because the learned network is essentially collating a whole lot of training samples together, we get more information per sample.


---

### Answer by username_2 on 2021-04-06. Score: 0

Since it is a trained network already, when you run an example through it, the gradient will not have a very high variance.

The gradient varies a lot when you are training a network from the scratch but then it stops varying much since it understands the pattern.


---

