<p>The normalization makes the signal small enough to remain in the region of the sigmoid that can be well approximated by a straight line.</p>
<p>The idea is the same as in electronics: <a href="https://en.wikipedia.org/wiki/Small-signal_model" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Small-signal_model</a></p>

CC BY-SA 4.0
2022-12-29T04:45:28.543Z
Oh so near the range of around [-1, 1] on the horizontal axis where the slope is close to being a straight line?
rkuang25
CC BY-SA 4.0
2022-12-29T06:30:37.410Z
Yes, that's it. The sigmoid is also almost linear, flat, for large absolute values. The hard sigmoid is a piecewise linear approximation to the sigmoid based on these three regions: large negative, around zero, large positive.
Jaume Oliver Lafont
CC BY-SA 4.0
2022-12-28T15:12:15.693Z
2022-12-28T15:12:15.693Z
Jaume Oliver Lafont
<p>In section 3 paragraph 2 of <em>Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift</em> paper (<a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">https://arxiv.org/abs/1502.03167</a>) they say that normalizing a layer's input may change what it represents, I understand this. But what do they mean in the bolded part?</p>
<blockquote>
<p>For instance, normalizing the inputs of a sigmoid would constrain them
to the <strong>linear regime of the nonlinearity</strong>.</p>
</blockquote>

<neural-networks><batch-normalization><normalisation>
What does linear regime of nonlinearity mean in normalisation?
rkuang25
# What does linear regime of nonlinearity mean in normalisation?

(Asked by: username_0 on 2022-12-28)

In section 3 paragraph 2 of *Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift* paper (<https://arxiv.org/abs/1502.03167>) they say that normalizing a layer's input may change what it represents, I understand this. But what do they mean in the bolded part?

> For instance, normalizing the inputs of a sigmoid would constrain them
> to the **linear regime of the nonlinearity**.

**Tags:** neural-networks, batch-normalization, normalisation


---

## 1 answers

### Answer by username_1 on 2022-12-28. Score: 2

The normalization makes the signal small enough to remain in the region of the sigmoid that can be well approximated by a straight line.

The idea is the same as in electronics: <https://en.wikipedia.org/wiki/Small-signal_model>

> **Comments:**
> **username_0**: Oh so near the range of around [-1, 1] on the horizontal axis where the slope is close to being a straight line?  
> **username_1**: Yes, that's it. The sigmoid is also almost linear, flat, for large absolute values. The hard sigmoid is a piecewise linear approximation to the sigmoid based on these three regions: large negative, around zero, large positive.  

---

