Answers.Body: <p>The normalization makes the signal small enough to remain in the region of the sigmoid that can be well approximated by a straight line.</p>
  <p>The idea is the same as in electronics: <a href="https://en.wikipedia.org/wiki/Small-signal_model" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Small-signal_model</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-29T04:45:28.543Z
Answers.Text: Oh so near the range of around [-1, 1] on the horizontal axis where the slope is close to being a straight line?
Answers.UserDisplayName: rkuang25
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-29T06:30:37.410Z
Answers.Text: Yes, that's it. The sigmoid is also almost linear, flat, for large absolute values. The hard sigmoid is a piecewise linear approximation to the sigmoid based on these three regions: large negative, around zero, large positive.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-28T15:12:15.693Z
Answers.LastActivityDate: 2022-12-28T15:12:15.693Z
Answers.UserDisplayName: Jaume Oliver Lafont
Body: <p>In section 3 paragraph 2 of <em>Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift</em> paper (<a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">https://arxiv.org/abs/1502.03167</a>) they say that normalizing a layer's input may change what it represents, I understand this. But what do they mean in the bolded part?</p>
  <blockquote>
  <p>For instance, normalizing the inputs of a sigmoid would constrain them
  to the <strong>linear regime of the nonlinearity</strong>.</p>
  </blockquote>
  
Tags: <neural-networks><batch-normalization><normalisation>
Title: What does linear regime of nonlinearity mean in normalisation?
UserDisplayName: rkuang25
ThreadText: # What does linear regime of nonlinearity mean in normalisation?
  
  (Asked by: username_0 on 2022-12-28)
  
  In section 3 paragraph 2 of *Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift* paper (<https://arxiv.org/abs/1502.03167>) they say that normalizing a layer's input may change what it represents, I understand this. But what do they mean in the bolded part?
  
  > For instance, normalizing the inputs of a sigmoid would constrain them
  > to the **linear regime of the nonlinearity**.
  
  **Tags:** neural-networks, batch-normalization, normalisation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-12-28. Score: 2
  
  The normalization makes the signal small enough to remain in the region of the sigmoid that can be well approximated by a straight line.
  
  The idea is the same as in electronics: <https://en.wikipedia.org/wiki/Small-signal_model>
  
  > **Comments:**
  > **username_0**: Oh so near the range of around [-1, 1] on the horizontal axis where the slope is close to being a straight line?  
  > **username_1**: Yes, that's it. The sigmoid is also almost linear, flat, for large absolute values. The hard sigmoid is a piecewise linear approximation to the sigmoid based on these three regions: large negative, around zero, large positive.  
  
  ---
  
  