Answers.Body: <p>A critical point of a function <span class="math-container">$f$</span> can be</p>
  <ol>
  <li>a stationary point (i.e. <span class="math-container">$f'(x) = 0$</span>), or</li>
  <li>a point where the derivative is undefined (for example, in the case of the <a href="https://math.stackexchange.com/q/991475/168764">absolute value function</a> <span class="math-container">$f(x)$</span>, <span class="math-container">$x=0$</span> is a critical point, as <span class="math-container">$f$</span> is not differentiable at <span class="math-container">$x=0$</span>).</li>
  </ol>
  <p>So, all stationary points are critical points.</p>
  <p><a href="https://tutorial.math.lamar.edu/classes/calci/criticalpoints.aspx" rel="nofollow noreferrer">These notes</a> provide more examples of how to find critical points of a function, so they could be useful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-21T00:51:51.603Z
Answers.Text: Is your writing completed? Please address, if possible, the **Is there no need to address the points where f′(x) does not exist?** part, which is causing issue.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-21T01:02:27.960Z
Answers.Text: @hanugm Are you asking if we can ignore the critical points that are not stationary points? If yes, in which context?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-21T01:13:08.950Z
Answers.Text: yea true, else they may not interchangeably use them.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-21T10:11:55.427Z
Answers.Text: @hanugm The point is: you can always refer to a stationary point as a "critical point" , but you cannot always refer to a critical point as a "stationary point". Some cases, where the derivative is undefined, may require a little bit of care, so you don't have to ignore all critical points. I've just read the answer by Taw, and it seems to give an example of where you need to pay attention to functions that are not differentiable at some points (e.g. ReLU).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-21T00:45:32.427Z
Answers.LastActivityDate: 2021-08-21T00:45:32.427Z
Answers.UserDisplayName: nbro
Answers.Body: <p>From reading the text, it's clear that the authors are using critical point to mean the same thing as stationary point, so they are not using the proper mathematical definition.</p>
  <p>More generally, automatic differentiation will return a gradient at nondifferentiable points. Ask tensorflow or pytorch to take the gradient of <span class="math-container">$\text{ReLU}(x)\big|_{x=0}$</span>. They return 0, even though it should technically return NaN, since it doesn't exist. Theoretically, the question of how to deal with nonsmooth functions is well understood - see <a href="https://en.wikipedia.org/wiki/Subderivative" rel="nofollow noreferrer">subdifferential</a> for example. In the relu example it actually returns an element of the subdifferential but <a href="https://proceedings.neurips.cc/paper/2018/file/142c65e00f4f7cf2e6c4c996e34005df-Paper.pdf" rel="nofollow noreferrer">there are pathological examples</a> where this is not true.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-21T04:03:24.037Z
Answers.LastActivityDate: 2021-08-21T04:03:24.037Z
Answers.UserDisplayName: Taw
Body: <p>Consider the following paragraph from <a href="https://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">Numerical Computation</a> of the deep learning book.</p>
  <blockquote>
  <p>When <span class="math-container">$f'(x) = 0$</span>, the derivative provides no information about which
  direction to move. Points where <span class="math-container">$f'(x)$</span> = 0 are known as <strong>critical
  points, or stationary points.</strong> A local minimum is a point where
  <span class="math-container">$f(x)$</span> is lower than at all neighboring points, so it is no longer
  possible to decrease <span class="math-container">$f(x)$</span> by making inﬁnitesimal steps. A local
  maximum is a point where <span class="math-container">$f(x)$</span> is higher than at all neighboring
  points so it is not possible to increase <span class="math-container">$f(x)$</span> by making inﬁnitesimal
  steps. Some critical points are neither maxima nor minima. These are
  known as saddle points.</p>
  </blockquote>
  <p>In short, points where <span class="math-container">$f'(x) =0 $</span> are called critical points, or stationary points.</p>
  <p>But, according to mathematical terminology, the definitions are as follows:</p>
  <p>#1: <a href="https://mathworld.wolfram.com/CriticalPoint.html" rel="nofollow noreferrer">Critical point</a></p>
  <blockquote>
  <p>A function <span class="math-container">$y=f(x)$</span> has critical points at all points <span class="math-container">$x_0$</span> where
  <span class="math-container">$f'(x_0)=0$</span> or <strong><span class="math-container">$f(x)$</span> is not differentiable</strong>.</p>
  </blockquote>
  <p>#2: <a href="https://mathworld.wolfram.com/StationaryPoint.html" rel="nofollow noreferrer">Stationary point</a></p>
  <blockquote>
  <p>A point <span class="math-container">$x_0$</span> at which the derivative of a function <span class="math-container">$f(x)$</span> vanishes,
  <span class="math-container">$f'(x_0)=0$</span>. A stationary point may be a minimum, maximum, or
  inflection point.</p>
  </blockquote>
  <p>It can be noticed that the definitions that are given in the deep learning book do match exactly with stationary points since the only premise is <span class="math-container">$f'(x)=0$</span>. The definition for critical point is not apt since a critical point can also be a point where <span class="math-container">$f'(x)$</span> is nonexistent.</p>
  <p>Is there any reason for using the terms critical points and stationary points interchangeably? Is there no need to address the points where <span class="math-container">$f'(x)$</span> does not exist?</p>
  
Tags: <terminology><books><derivative>
Title: Why does critical points and stationary points are used interchangeably?
UserDisplayName: hanugm
ThreadText: # Why does critical points and stationary points are used interchangeably?
  
  (Asked by: username_0 on 2021-08-20)
  
  Consider the following paragraph from [Numerical Computation](https://www.deeplearningbook.org/contents/numerical.html) of the deep learning book.
  
  > When $f'(x) = 0$, the derivative provides no information about which
  > direction to move. Points where $f'(x)$ = 0 are known as **critical
  > points, or stationary points.** A local minimum is a point where
  > $f(x)$ is lower than at all neighboring points, so it is no longer
  > possible to decrease $f(x)$ by making inﬁnitesimal steps. A local
  > maximum is a point where $f(x)$ is higher than at all neighboring
  > points so it is not possible to increase $f(x)$ by making inﬁnitesimal
  > steps. Some critical points are neither maxima nor minima. These are
  > known as saddle points.
  
  In short, points where $f'(x) =0 $ are called critical points, or stationary points.
  
  But, according to mathematical terminology, the definitions are as follows:
  
  #1: [Critical point](https://mathworld.wolfram.com/CriticalPoint.html)
  
  > A function $y=f(x)$ has critical points at all points $x\_0$ where
  > $f'(x\_0)=0$ or **$f(x)$ is not differentiable**.
  
  #2: [Stationary point](https://mathworld.wolfram.com/StationaryPoint.html)
  
  > A point $x\_0$ at which the derivative of a function $f(x)$ vanishes,
  > $f'(x\_0)=0$. A stationary point may be a minimum, maximum, or
  > inflection point.
  
  It can be noticed that the definitions that are given in the deep learning book do match exactly with stationary points since the only premise is $f'(x)=0$. The definition for critical point is not apt since a critical point can also be a point where $f'(x)$ is nonexistent.
  
  Is there any reason for using the terms critical points and stationary points interchangeably? Is there no need to address the points where $f'(x)$ does not exist?
  
  **Tags:** terminology, books, derivative
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-08-21. Score: 1
  
  A critical point of a function $f$ can be
  
  1. a stationary point (i.e. $f'(x) = 0$), or
  2. a point where the derivative is undefined (for example, in the case of the [absolute value function](https://math.stackexchange.com/q/991475/168764) $f(x)$, $x=0$ is a critical point, as $f$ is not differentiable at $x=0$).
  
  So, all stationary points are critical points.
  
  [These notes](https://tutorial.math.lamar.edu/classes/calci/criticalpoints.aspx) provide more examples of how to find critical points of a function, so they could be useful.
  
  > **Comments:**
  > **username_0**: Is your writing completed? Please address, if possible, the **Is there no need to address the points where f′(x) does not exist?** part, which is causing issue.  
  > **username_1**: @hanugm Are you asking if we can ignore the critical points that are not stationary points? If yes, in which context?  
  > **username_0**: yea true, else they may not interchangeably use them.  
  > **username_1**: @hanugm The point is: you can always refer to a stationary point as a "critical point" , but you cannot always refer to a critical point as a "stationary point". Some cases, where the derivative is undefined, may require a little bit of care, so you don't have to ignore all critical points. I've just read the answer by Taw, and it seems to give an example of where you need to pay attention to functions that are not differentiable at some points (e.g. ReLU).  
  
  ---
  
  ### Answer by username_2 on 2021-08-21. Score: 1
  
  From reading the text, it's clear that the authors are using critical point to mean the same thing as stationary point, so they are not using the proper mathematical definition.
  
  More generally, automatic differentiation will return a gradient at nondifferentiable points. Ask tensorflow or pytorch to take the gradient of $\text{ReLU}(x)\big|\_{x=0}$. They return 0, even though it should technically return NaN, since it doesn't exist. Theoretically, the question of how to deal with nonsmooth functions is well understood - see [subdifferential](https://en.wikipedia.org/wiki/Subderivative) for example. In the relu example it actually returns an element of the subdifferential but [there are pathological examples](https://proceedings.neurips.cc/paper/2018/file/142c65e00f4f7cf2e6c4c996e34005df-Paper.pdf) where this is not true.
  
  
  ---
  
  