Answers.Body: <p>A <a href="https://mathworld.wolfram.com/SmoothFunction.html" rel="nofollow noreferrer">smooth function</a> is usually defined to be a function that is <span class="math-container">$n$</span>-times <em>continuously differentiable</em>, which means that <span class="math-container">$f$</span>, <span class="math-container">$f'$</span>, <span class="math-container">$\dots$</span>, <span class="math-container">$f^{(n - 1)}$</span> are all differentiable and <span class="math-container">$f^{(n)}$</span> is continuous. Such functions are also called <span class="math-container">$C^n$</span> functions.</p>
  <p>It can be a bit of a vague term; some people might even stretch the definition and say any continuous function is smooth (though I'd be a little surprised if I saw that in use, personally). Other people write smooth to mean <em>infinitely differentiable</em>: for example <span class="math-container">$f(x) = e^x$</span> can be differentiated as many times as you like.</p>
  <p>I guess what the author is trying to point out is that the ReLU rectifier function isn't differentiable. Even if you use the &quot;trick&quot;<sup>1</sup> of treating ReLU as differentiable everywhere, you would still get a derivative that is <strong>discontinuous</strong>:</p>
  <p><span class="math-container">$$\mathrm{ReLU}'(x) = \begin{cases} 1 &amp; x \ge 0 \\ 0 &amp; \text{otherwise.} \end{cases}$$</span></p>
  <p>So, it's fair to say that ReLU isn't smooth in the same sense of the <span class="math-container">$\tanh$</span> function, which has a continuous derivative (and, in fact, you could carry on and consider the higher derivatives).</p>
  <hr />
  <p><sup>1</sup> If this doesn't sound familiar, see p. 188 of <a href="https://www.deeplearningbook.org/contents/mlp.html" rel="nofollow noreferrer">Deep Learning</a> by Bengio et al. We can get around the fact that ReLU functions aren't differentiable at zero by just pretending it has a well-defined derivative of zero or one. A little dishonest, perhaps, but it works very well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-08T13:38:52.737Z
Answers.LastActivityDate: 2021-07-08T13:38:52.737Z
Answers.UserDisplayName: htl
Body: <p>The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature.</p>
  <p>The following excerpt taken from p4 of <a href="https://web.stanford.edu/%7Ejurafsky/slp3/7.pdf" rel="nofollow noreferrer">Neural Networks and Neural Language Models</a> says that <code>tanh</code> has a couple of interesting properties.</p>
  <blockquote>
  <p>For example, the tanh function has the nice properties of being
  <strong>smoothly differentiable</strong> and mapping outlier values toward the mean.</p>
  </blockquote>
  <p>A function is said to be differentiable if it is differentiable at every point in the domain of function. The domain of <code>tanh</code> is <span class="math-container">$\mathbb{R}$</span> and <span class="math-container">$ \dfrac{e^x-e^{-x}}{e^x+e^{-x}}$</span> is differentiable in <span class="math-container">$\mathbb{R}$</span>.</p>
  <p>But what is meant by &quot;<strong>smoothly</strong> differentiable&quot; in the case of <code>tanh</code> activation function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-08T09:30:39.827Z
Comments.Text: That sounds like a pretty subjective description. I would argue sigmoid is also smoothly differentiable because the gradient function is predictable and symmetric. It could maybe be referring to how the tanh derivative exists in one of 3 states: Essentially stationary, linear, or a transition between the two.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-08T10:51:05.187Z
Comments.Text: In math, as far as I know, "smooth" often refers to the fact that the function can be differentiated multiple times. See [this](https://en.wikipedia.org/wiki/Smoothness).  To be honest, I never really thought about the properties of the hyperbolic tangent, so I don't know if the author of that excerpt is referring to something else.
Comments.UserDisplayName: nbro
Tags: <neural-networks><terminology><math><activation-functions><tanh>
Title: Why is tanh a "smoothly" differentiable function?
UserDisplayName: hanugm
ThreadText: # Why is tanh a "smoothly" differentiable function?
  
  (Asked by: username_0 on 2021-07-07)
  
  The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature.
  
  The following excerpt taken from p4 of [Neural Networks and Neural Language Models](https://web.stanford.edu/%7Ejurafsky/slp3/7.pdf) says that `tanh` has a couple of interesting properties.
  
  > For example, the tanh function has the nice properties of being
  > **smoothly differentiable** and mapping outlier values toward the mean.
  
  A function is said to be differentiable if it is differentiable at every point in the domain of function. The domain of `tanh` is $\mathbb{R}$ and $ \dfrac{e^x-e^{-x}}{e^x+e^{-x}}$ is differentiable in $\mathbb{R}$.
  
  But what is meant by "**smoothly** differentiable" in the case of `tanh` activation function?
  
  **Tags:** neural-networks, terminology, math, activation-functions, tanh
  > **Comments:**
  > **username_1**: That sounds like a pretty subjective description. I would argue sigmoid is also smoothly differentiable because the gradient function is predictable and symmetric. It could maybe be referring to how the tanh derivative exists in one of 3 states: Essentially stationary, linear, or a transition between the two.  
  > **username_2**: In math, as far as I know, "smooth" often refers to the fact that the function can be differentiated multiple times. See [this](https://en.wikipedia.org/wiki/Smoothness).  To be honest, I never really thought about the properties of the hyperbolic tangent, so I don't know if the author of that excerpt is referring to something else.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-07-08. Score: 4 | Accepted answer
  
  A [smooth function](https://mathworld.wolfram.com/SmoothFunction.html) is usually defined to be a function that is $n$-times *continuously differentiable*, which means that $f$, $f'$, $\dots$, $f^{(n - 1)}$ are all differentiable and $f^{(n)}$ is continuous. Such functions are also called $C^n$ functions.
  
  It can be a bit of a vague term; some people might even stretch the definition and say any continuous function is smooth (though I'd be a little surprised if I saw that in use, personally). Other people write smooth to mean *infinitely differentiable*: for example $f(x) = e^x$ can be differentiated as many times as you like.
  
  I guess what the author is trying to point out is that the ReLU rectifier function isn't differentiable. Even if you use the "trick"1 of treating ReLU as differentiable everywhere, you would still get a derivative that is **discontinuous**:
  
  $$\mathrm{ReLU}'(x) = \begin{cases} 1 & x \ge 0 \\ 0 & \text{otherwise.} \end{cases}$$
  
  So, it's fair to say that ReLU isn't smooth in the same sense of the $\tanh$ function, which has a continuous derivative (and, in fact, you could carry on and consider the higher derivatives).
  
  ---
  
  1 If this doesn't sound familiar, see p. 188 of [Deep Learning](https://www.deeplearningbook.org/contents/mlp.html) by Bengio et al. We can get around the fact that ReLU functions aren't differentiable at zero by just pretending it has a well-defined derivative of zero or one. A little dishonest, perhaps, but it works very well.
  
  
  ---
  
  