Answers.Body: <p>From the previous paragraph:</p>
  <blockquote>
  <p>&quot;This, however, can cause two problems. First, it may result in
  over-fitting: if the model learns to assign full probability to the
  groundtruth label for each training example, it is not guaranteed to
  generalize. Second, it encourages the differences between the largest
  logit and all others to become large, and this, combined with the
  bounded gradient ∂` ∂zk , reduces the ability of the model to adapt.
  Intuitively, this happens because the model becomes too confident
  about its predictions.&quot;</p>
  </blockquote>
  <p>In other words, label smoothing increases your model's entropy in its output logits, which, in turn, makes it less confident in its predictions. As the authors point out, this may lead to a decrease in training performance w.r.t. the log-likelihood of the training labels. In particular, the model's capacity may be high enough to find a complex function to map over all input examples <em>without</em> label smoothing. This function, however, may not generalise well, making it less <em>adaptable</em> to unseen data. In the GAN space, there are other benefits to label smoothing as well, such as preventing the discriminator from relaying a too high gradient signal to the generator (see here: <a href="https://www.kth.se/social/files/59086d09f2765460c378ca73/GANs.pdf" rel="nofollow noreferrer">https://www.kth.se/social/files/59086d09f2765460c378ca73/GANs.pdf</a>). Large gradients are known to make the training process more instable and may prevent convergence. For a deeper understanding of label-smoothing and entropy, I recommend this paper:
  <a href="https://arxiv.org/pdf/2005.00820.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2005.00820.pdf</a>
  Also, check out this related question:
  <a href="https://datascience.stackexchange.com/questions/28764/one-sided-label-smoothing-in-gans">https://datascience.stackexchange.com/questions/28764/one-sided-label-smoothing-in-gans</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T14:10:44.583Z
Answers.LastActivityDate: 2023-04-26T14:10:44.583Z
Answers.UserDisplayName: postnubilaphoebus
Body: <p>I was learning about GANs when the term &quot;Label Smoothing&quot; surfaced. In the video tutorial that I watched, they use the term &quot;label smoothing&quot; to change the binary labels when calculating the loss of the discriminator network. Instead of using 1, they use 0.9 for the label. What is the main purpose of this label smoothing?</p>
  <p>I've skimmed through the <a href="https://arxiv.org/pdf/1512.00567.pdf" rel="nofollow noreferrer">original paper</a>, and there is a lot of maths that, honestly, I have difficulty understanding. But I notice this paragraph in there:</p>
  <blockquote>
  <p>We propose a mechanism for encouraging the model to be less confident. While this may not be desired if the goal is to maximize the log-likelihood of training labels, it does regularize  the  model  and  makes  it  more  adaptable</p>
  </blockquote>
  <p>And it gives me another question:</p>
  <ul>
  <li><p>why &quot;this may not be desired if the goal is to maximize the log-likelihood of training labels&quot;?</p>
  </li>
  <li><p>what do they mean by &quot;adaptable&quot;?</p>
  </li>
  </ul>
  
Tags: <generative-adversarial-networks><generative-model><regularization><labels>
Title: What is the intuition behind the Label Smoothing in GANs?
UserDisplayName: malioboro
ThreadText: # What is the intuition behind the Label Smoothing in GANs?
  
  (Asked by: username_0 on 2019-05-04)
  
  I was learning about GANs when the term "Label Smoothing" surfaced. In the video tutorial that I watched, they use the term "label smoothing" to change the binary labels when calculating the loss of the discriminator network. Instead of using 1, they use 0.9 for the label. What is the main purpose of this label smoothing?
  
  I've skimmed through the [original paper](https://arxiv.org/pdf/1512.00567.pdf), and there is a lot of maths that, honestly, I have difficulty understanding. But I notice this paragraph in there:
  
  > We propose a mechanism for encouraging the model to be less confident. While this may not be desired if the goal is to maximize the log-likelihood of training labels, it does regularize the model and makes it more adaptable
  
  And it gives me another question:
  
  * why "this may not be desired if the goal is to maximize the log-likelihood of training labels"?
  * what do they mean by "adaptable"?
  
  **Tags:** generative-adversarial-networks, generative-model, regularization, labels
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-26. Score: 1
  
  From the previous paragraph:
  
  > "This, however, can cause two problems. First, it may result in
  > over-fitting: if the model learns to assign full probability to the
  > groundtruth label for each training example, it is not guaranteed to
  > generalize. Second, it encourages the differences between the largest
  > logit and all others to become large, and this, combined with the
  > bounded gradient ∂` ∂zk , reduces the ability of the model to adapt.
  > Intuitively, this happens because the model becomes too confident
  > about its predictions."
  
  In other words, label smoothing increases your model's entropy in its output logits, which, in turn, makes it less confident in its predictions. As the authors point out, this may lead to a decrease in training performance w.r.t. the log-likelihood of the training labels. In particular, the model's capacity may be high enough to find a complex function to map over all input examples *without* label smoothing. This function, however, may not generalise well, making it less *adaptable* to unseen data. In the GAN space, there are other benefits to label smoothing as well, such as preventing the discriminator from relaying a too high gradient signal to the generator (see here: <https://www.kth.se/social/files/59086d09f2765460c378ca73/GANs.pdf>). Large gradients are known to make the training process more instable and may prevent convergence. For a deeper understanding of label-smoothing and entropy, I recommend this paper:
  <https://arxiv.org/pdf/2005.00820.pdf>
  Also, check out this related question:
  <https://datascience.stackexchange.com/questions/28764/one-sided-label-smoothing-in-gans>
  
  
  ---
  
  