Answers.Body: <p><a href="https://i.sstatic.net/p9daq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/p9daq.png" alt="enter image description here"></a></p>
  
  <p>Derivative of the sigmoid curve is 0 when the output is 0 or 1 as you can see from the image above. The technique you are referring to is called label-smoothing which is used in various applications (e.g. GANs) but I can see how it would be applicable here also, by helping to avoid 0-gradients saturating learning.</p>
  
  <p>To answer your third question, sigmoids are rarely used in the intermediate layers of networks these days. They have been replaced by ReLUs (or variants of ReLU) for this exact reason - sigmoids are prone to causing 'vanishing gradients' where the gradient becomes 0 and therefore does not get backpropagated any further. ReLUs alleviate this problem by always providing a gradient of 1 for positive input values.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-23T10:38:05.843Z
Answers.LastActivityDate: 2018-08-23T10:38:05.843Z
Answers.UserDisplayName: Mark
Body: <p>I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "saturation effects". I only understood is halfway, and was hoping someone could clarify a few things for me:</p>
  
  <ol>
  <li><p>Is this only the case when the output is boolean (0 or 1), or will it also be the case for continual values in the range between 0 and 1. If so, should all values be scaled to the interval [0.1, 0.9]?</p></li>
  <li><p>What exactly is the problem of output 0 or 1? Does it have something to do with the derivative of sigmoid being 0 when it's value is 0 or 1? As I understood it weights could end up approaching infinity, but I didn't understand why.</p></li>
  <li><p>Is this the case only when sigmoid is used in the output layer (which it rarely is, I believe), or is it also the case when sigmoid is used in hidden layers only?</p></li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-22T12:18:39.083Z
Comments.Text: Can you link to the article you mention? Out of context, it's not completely clear what is meant.
Comments.UserDisplayName: John Doucette
Tags: <neural-networks><deep-neural-networks><activation-functions><sigmoid>
Title: Target values of 0.1 for 0 and 0.9 for 1 for sigmoid
UserDisplayName: Mr. Eivind
ThreadText: # Target values of 0.1 for 0 and 0.9 for 1 for sigmoid
  
  (Asked by: username_0 on 2018-08-21)
  
  I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "saturation effects". I only understood is halfway, and was hoping someone could clarify a few things for me:
  
  1. Is this only the case when the output is boolean (0 or 1), or will it also be the case for continual values in the range between 0 and 1. If so, should all values be scaled to the interval [0.1, 0.9]?
  2. What exactly is the problem of output 0 or 1? Does it have something to do with the derivative of sigmoid being 0 when it's value is 0 or 1? As I understood it weights could end up approaching infinity, but I didn't understand why.
  3. Is this the case only when sigmoid is used in the output layer (which it rarely is, I believe), or is it also the case when sigmoid is used in hidden layers only?
  
  **Tags:** neural-networks, deep-neural-networks, activation-functions, sigmoid
  > **Comments:**
  > **username_1**: Can you link to the article you mention? Out of context, it's not completely clear what is meant.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-08-23. Score: 2 | Accepted answer
  
  [![enter image description here](https://i.sstatic.net/p9daq.png)](https://i.sstatic.net/p9daq.png)
  
  Derivative of the sigmoid curve is 0 when the output is 0 or 1 as you can see from the image above. The technique you are referring to is called label-smoothing which is used in various applications (e.g. GANs) but I can see how it would be applicable here also, by helping to avoid 0-gradients saturating learning.
  
  To answer your third question, sigmoids are rarely used in the intermediate layers of networks these days. They have been replaced by ReLUs (or variants of ReLU) for this exact reason - sigmoids are prone to causing 'vanishing gradients' where the gradient becomes 0 and therefore does not get backpropagated any further. ReLUs alleviate this problem by always providing a gradient of 1 for positive input values.
  
  
  ---
  
  