<p>You can find a definition for &quot;numerical stability&quot; in <a href="https://mathworld.wolfram.com/NumericalStability.html" rel="nofollow noreferrer">mathworld wolframe</a>:</p>
<blockquote>
<p>Numerical stability refers to how a malformed input affects the execution of an algorithm. In a numerically stable algorithm, errors in the input lessen in significance as the algorithm executes, having little effect on the final output.
On the other hand, in a numerically unstable algorithm, errors in the input cause a considerably larger error in the final output.</p>
</blockquote>
<p>In your example, suppose <span class="math-container">$Var(x)$</span> is little and truncated to zero in the corresponding computing machine. In that case, you will get <code>INF</code> as a result and it can be problematic to continue the computation.</p>
<p>Therefore, they add a small value such as <span class="math-container">$\epsilon$</span> to the division to prevent such a case. As <span class="math-container">$Var(x) \geqslant 0$</span>, <span class="math-container">$Var(x) + \epsilon$</span> will be always greater than zero for any machine with higher precision than the <span class="math-container">$\epsilon$</span>. So, here, an interpretation of the numerical stability can be like this!</p>

CC BY-SA 4.0
2021-08-03T14:11:34.160Z
2021-08-03T14:11:34.160Z
OmG
<p>I came across the phrase &quot;numerical stability&quot; several times. But almost in the same context.</p>
<p>I encountered this word mostly in the analytical formula for <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="nofollow noreferrer">batch normalization</a>.</p>
<blockquote>
<p><span class="math-container">$$y = \dfrac{x - \mathbb{E}[x]}{\sqrt{Var[x]+\epsilon}}* \gamma +
 \beta$$</span></p>
<p><strong>eps</strong> – a value added to the denominator for <strong>numerical stability</strong>. Default: 1e-5</p>
</blockquote>
<p>Is the phenomenon of &quot;numerical instability&quot; happens during the training of neural networks? Or is it a general one in other models also? What is the reason for its occurrence?</p>

CC BY-SA 4.0
2021-08-17T02:00:14.120Z
Related: 4.2 [Poor Conditioning](https://www.deeplearningbook.org/contents/numerical.html)
hanugm
<neural-networks><machine-learning><terminology><numerical-algorithms>
What is numerical stability?
hanugm
# What is numerical stability?

(Asked by: username_0 on 2021-08-03)

I came across the phrase "numerical stability" several times. But almost in the same context.

I encountered this word mostly in the analytical formula for [batch normalization](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html).

> $$y = \dfrac{x - \mathbb{E}[x]}{\sqrt{Var[x]+\epsilon}}\* \gamma +
> \beta$$
>
> **eps** – a value added to the denominator for **numerical stability**. Default: 1e-5

Is the phenomenon of "numerical instability" happens during the training of neural networks? Or is it a general one in other models also? What is the reason for its occurrence?

**Tags:** neural-networks, machine-learning, terminology, numerical-algorithms

> **Comments:**
> **username_0**: Related: 4.2 [Poor Conditioning](https://www.deeplearningbook.org/contents/numerical.html)  

---

## 1 answers

### Answer by username_1 on 2021-08-03. Score: 4 | Accepted answer

You can find a definition for "numerical stability" in [mathworld wolframe](https://mathworld.wolfram.com/NumericalStability.html):

> Numerical stability refers to how a malformed input affects the execution of an algorithm. In a numerically stable algorithm, errors in the input lessen in significance as the algorithm executes, having little effect on the final output.
> On the other hand, in a numerically unstable algorithm, errors in the input cause a considerably larger error in the final output.

In your example, suppose $Var(x)$ is little and truncated to zero in the corresponding computing machine. In that case, you will get `INF` as a result and it can be problematic to continue the computation.

Therefore, they add a small value such as $\epsilon$ to the division to prevent such a case. As $Var(x) \geqslant 0$, $Var(x) + \epsilon$ will be always greater than zero for any machine with higher precision than the $\epsilon$. So, here, an interpretation of the numerical stability can be like this!


---

