Answers.Body: <p>You can find a definition for &quot;numerical stability&quot; in <a href="https://mathworld.wolfram.com/NumericalStability.html" rel="nofollow noreferrer">mathworld wolframe</a>:</p>
  <blockquote>
  <p>Numerical stability refers to how a malformed input affects the execution of an algorithm. In a numerically stable algorithm, errors in the input lessen in significance as the algorithm executes, having little effect on the final output.
  On the other hand, in a numerically unstable algorithm, errors in the input cause a considerably larger error in the final output.</p>
  </blockquote>
  <p>In your example, suppose <span class="math-container">$Var(x)$</span> is little and truncated to zero in the corresponding computing machine. In that case, you will get <code>INF</code> as a result and it can be problematic to continue the computation.</p>
  <p>Therefore, they add a small value such as <span class="math-container">$\epsilon$</span> to the division to prevent such a case. As <span class="math-container">$Var(x) \geqslant 0$</span>, <span class="math-container">$Var(x) + \epsilon$</span> will be always greater than zero for any machine with higher precision than the <span class="math-container">$\epsilon$</span>. So, here, an interpretation of the numerical stability can be like this!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T14:11:34.160Z
Answers.LastActivityDate: 2021-08-03T14:11:34.160Z
Answers.UserDisplayName: OmG
Body: <p>I came across the phrase &quot;numerical stability&quot; several times. But almost in the same context.</p>
  <p>I encountered this word mostly in the analytical formula for <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="nofollow noreferrer">batch normalization</a>.</p>
  <blockquote>
  <p><span class="math-container">$$y = \dfrac{x - \mathbb{E}[x]}{\sqrt{Var[x]+\epsilon}}* \gamma +
   \beta$$</span></p>
  <p><strong>eps</strong> – a value added to the denominator for <strong>numerical stability</strong>. Default: 1e-5</p>
  </blockquote>
  <p>Is the phenomenon of &quot;numerical instability&quot; happens during the training of neural networks? Or is it a general one in other models also? What is the reason for its occurrence?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-17T02:00:14.120Z
Comments.Text: Related: 4.2 [Poor Conditioning](https://www.deeplearningbook.org/contents/numerical.html)
Comments.UserDisplayName: hanugm
Tags: <neural-networks><machine-learning><terminology><numerical-algorithms>
Title: What is numerical stability?
UserDisplayName: hanugm
ThreadText: # What is numerical stability?
  
  (Asked by: username_0 on 2021-08-03)
  
  I came across the phrase "numerical stability" several times. But almost in the same context.
  
  I encountered this word mostly in the analytical formula for [batch normalization](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html).
  
  > $$y = \dfrac{x - \mathbb{E}[x]}{\sqrt{Var[x]+\epsilon}}\* \gamma +
  > \beta$$
  >
  > **eps** – a value added to the denominator for **numerical stability**. Default: 1e-5
  
  Is the phenomenon of "numerical instability" happens during the training of neural networks? Or is it a general one in other models also? What is the reason for its occurrence?
  
  **Tags:** neural-networks, machine-learning, terminology, numerical-algorithms
  > **Comments:**
  > **username_0**: Related: 4.2 [Poor Conditioning](https://www.deeplearningbook.org/contents/numerical.html)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-03. Score: 4 | Accepted answer
  
  You can find a definition for "numerical stability" in [mathworld wolframe](https://mathworld.wolfram.com/NumericalStability.html):
  
  > Numerical stability refers to how a malformed input affects the execution of an algorithm. In a numerically stable algorithm, errors in the input lessen in significance as the algorithm executes, having little effect on the final output.
  > On the other hand, in a numerically unstable algorithm, errors in the input cause a considerably larger error in the final output.
  
  In your example, suppose $Var(x)$ is little and truncated to zero in the corresponding computing machine. In that case, you will get `INF` as a result and it can be problematic to continue the computation.
  
  Therefore, they add a small value such as $\epsilon$ to the division to prevent such a case. As $Var(x) \geqslant 0$, $Var(x) + \epsilon$ will be always greater than zero for any machine with higher precision than the $\epsilon$. So, here, an interpretation of the numerical stability can be like this!
  
  
  ---
  
  