Body: <p>Taking out the weighting factor we can define focal loss as<br>
  <span class="math-container">$$FL(p) = -(1-p)^\gamma log(p) $$</span>  </p>
  
  <p>Where <span class="math-container">$p$</span> is the target probability. The idea being that single stage object detectors have a huge class imbalance between foreground and background (several orders of magnitude of difference), and this loss will down-scale all results that are positively classified compared to normal cross entropy (<span class="math-container">$CE(p) = -log(p)$</span>) so that the optimization can then focus on the rest.  </p>
  
  <p>On the other hand, the general optimization scheme uses the gradient to find the direction with the steepest descent. There exists methodologies for adaption, momentum and etc but that is the general gist.<br>
  <span class="math-container">$$ \theta \leftarrow \theta - \eta \nabla_\theta L $$</span>  </p>
  
  <p>Focal losses gradient follows as so<br>
  <span class="math-container">$$\dot {FL}(p) = \dot p [\gamma(1-p)^{\gamma -1} log(p) -\frac{(1-p)^\gamma}{p}]$$</span> compared to the normal crossentropies loss of<br>
  <span class="math-container">$$ \dot{CE}(p) = -\frac{\dot p}{p}$$</span>  </p>
  
  <p>So we can now rewrite these as  </p>
  
  <p><span class="math-container">$$\dot{FL}(p) = (1-p)^\gamma \dot{CE}(p) + \gamma \dot p (1-p)^{\gamma -1} log(p)$$</span>  </p>
  
  <p>The initial term, given our optimization scheme will do what we (and the authors of <a href="https://arxiv.org/abs/1708.02002" rel="nofollow noreferrer">the retinanet paper</a>) want which is downscale the effect of the labels that are already well classified but the second term is slightly less interpretative and in parameter space and may cause an unwanted result. So my question is why not remove it and only use the gradient<br>
  <span class="math-container">$$\dot L =  (1-p)^\gamma \dot{CE}(p)$$</span>  </p>
  
  <p>Which given a <span class="math-container">$\gamma \in \mathbb{N}$</span> produces a loss function<br>
  <span class="math-container">$$ L(p) = -log(p) - \sum_{i=1}^\gamma {\gamma \choose i}\frac{(-p)^i}{i}$$</span>  </p>
  
  <p><strong>Summary:</strong> Is there a reason we make the loss adaptive and not the gradient in cases like focal loss? Does that second term add something useful?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-09T01:31:17.820Z
Comments.Text: would you add details of "slightly less interpretative (and) in parameter space and may cause an unwanted result.", thanks.
Comments.UserDisplayName: Cloud Cho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-09T01:39:49.170Z
Comments.Text: Yes, the second term in the gradient doesn't really have a clear cut meaning. the first is the gradient for weighted CE so its impact will be just that, but this one is difficult to completely interpret, its a weighted copy of log(p)*dp, but what does that mean. I wrote a small blog post a couple years back talking about a possible fix to this: https://mshlis.github.io/FocalGradLoss/
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-09T17:41:38.757Z
Comments.Text: thanks for your link. Have you tried to train neural network with only the second term?
Comments.UserDisplayName: Cloud Cho
Tags: <neural-networks><computer-vision><optimization><gradient-descent><object-detection>
Title: Does Retina-net's focal loss accomplish its goal?
UserDisplayName: mshlis
ThreadText: # Does Retina-net's focal loss accomplish its goal?
  
  (Asked by: username_0 on 2019-08-03)
  
  Taking out the weighting factor we can define focal loss as  
  $$FL(p) = -(1-p)^\gamma log(p) $$
  
  Where $p$ is the target probability. The idea being that single stage object detectors have a huge class imbalance between foreground and background (several orders of magnitude of difference), and this loss will down-scale all results that are positively classified compared to normal cross entropy ($CE(p) = -log(p)$) so that the optimization can then focus on the rest.
  
  On the other hand, the general optimization scheme uses the gradient to find the direction with the steepest descent. There exists methodologies for adaption, momentum and etc but that is the general gist.  
  $$ \theta \leftarrow \theta - \eta \nabla\_\theta L $$
  
  Focal losses gradient follows as so  
  $$\dot {FL}(p) = \dot p [\gamma(1-p)^{\gamma -1} log(p) -\frac{(1-p)^\gamma}{p}]$$ compared to the normal crossentropies loss of  
  $$ \dot{CE}(p) = -\frac{\dot p}{p}$$
  
  So we can now rewrite these as
  
  $$\dot{FL}(p) = (1-p)^\gamma \dot{CE}(p) + \gamma \dot p (1-p)^{\gamma -1} log(p)$$
  
  The initial term, given our optimization scheme will do what we (and the authors of [the retinanet paper](https://arxiv.org/abs/1708.02002)) want which is downscale the effect of the labels that are already well classified but the second term is slightly less interpretative and in parameter space and may cause an unwanted result. So my question is why not remove it and only use the gradient  
  $$\dot L = (1-p)^\gamma \dot{CE}(p)$$
  
  Which given a $\gamma \in \mathbb{N}$ produces a loss function  
  $$ L(p) = -log(p) - \sum\_{i=1}^\gamma {\gamma \choose i}\frac{(-p)^i}{i}$$
  
  **Summary:** Is there a reason we make the loss adaptive and not the gradient in cases like focal loss? Does that second term add something useful?
  
  **Tags:** neural-networks, computer-vision, optimization, gradient-descent, object-detection
  
  > **Comments:**
  > **username_1**: would you add details of "slightly less interpretative (and) in parameter space and may cause an unwanted result.", thanks.  
  > **username_0**: Yes, the second term in the gradient doesn't really have a clear cut meaning. the first is the gradient for weighted CE so its impact will be just that, but this one is difficult to completely interpret, its a weighted copy of log(p)*dp, but what does that mean. I wrote a small blog post a couple years back talking about a possible fix to this: https://username_0.github.io/FocalGradLoss/  
  > **username_1**: thanks for your link. Have you tried to train neural network with only the second term?  
  
  ---
  
  ## 0 answers
  
  