Answers.Body: <p>This scaling factor is derived in Section 2.1 of the paper. <span class="math-container">$\frac{1}{T^2}$</span> is a coefficient for both equations (3) and (4) when calculating the derivative of the cross-entropy loss <span class="math-container">$C$</span> wrt a logit <span class="math-container">$z_i$</span>. These two equations are made under some assumptions, though: the temperature <span class="math-container">$T$</span> has to be high compared to the logit magnitudes (for equation (3)) and that the logits need to be zero-meaned (for equation (4)).</p>
  <p>The way you get from (2) to (3) using the temperature magnitude assumption is with the approximation: <span class="math-container">$\log(1+x) \approx x$</span> for small <span class="math-container">$x$</span>. Let <span class="math-container">$x$</span> be <span class="math-container">$\frac{z_i}{T}$</span> which means:
  <span class="math-container">$$\log(1 + \frac{z_i}{T}) \approx \frac{z_i}{T}$$</span>
  <span class="math-container">$$1 + \frac{z_i}{T} \approx e^{\frac{z_i}{T}}$$</span>
  Apply this to the numerators and denominators to complete the derivation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-21T09:52:44.040Z
Answers.LastActivityDate: 2023-07-21T09:57:51.193Z
Answers.LastEditDate: 2023-07-21T09:57:51.193Z
Answers.UserDisplayName: Alexander Wan
Body: <p>In the paper &quot;Distilling the knowledge in a neural network&quot;, it mentioned &quot;the magnitudes of the gradients produced by the soft targets scale as 1/(T^2) &quot;, but it has no mathematical derivation. So how the term 1/(T^2) is calculated?</p>
  
Tags: <deep-learning><gradient>
Title: Why the gradients produced by the soft targets scale as 1/T^2 in knowledge distillation?
UserDisplayName: Xiong
ThreadText: # Why the gradients produced by the soft targets scale as 1/T^2 in knowledge distillation?
  
  (Asked by: username_0 on 2023-07-21)
  
  In the paper "Distilling the knowledge in a neural network", it mentioned "the magnitudes of the gradients produced by the soft targets scale as 1/(T^2) ", but it has no mathematical derivation. So how the term 1/(T^2) is calculated?
  
  **Tags:** deep-learning, gradient
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-21. Score: 1
  
  This scaling factor is derived in Section 2.1 of the paper. $\frac{1}{T^2}$ is a coefficient for both equations (3) and (4) when calculating the derivative of the cross-entropy loss $C$ wrt a logit $z\_i$. These two equations are made under some assumptions, though: the temperature $T$ has to be high compared to the logit magnitudes (for equation (3)) and that the logits need to be zero-meaned (for equation (4)).
  
  The way you get from (2) to (3) using the temperature magnitude assumption is with the approximation: $\log(1+x) \approx x$ for small $x$. Let $x$ be $\frac{z\_i}{T}$ which means:
  $$\log(1 + \frac{z\_i}{T}) \approx \frac{z\_i}{T}$$
  $$1 + \frac{z\_i}{T} \approx e^{\frac{z\_i}{T}}$$
  Apply this to the numerators and denominators to complete the derivation.
  
  
  ---
  
  