Answers.Body: <p>Your interpretation is definitely correct. As you correctly pointed out, the derivative of <code>softplus</code> is continuous and <span class="math-container">$n$</span>-times differentiable, that makes the function smooth, which is not the case for <code>ReLU</code>.</p>
  <p>What is quite interesting here is <em>why</em> <code>softplus</code> can be called an approximation to <code>ReLU</code>.</p>
  <p>If we break down the definition of <code>softplus</code>, we note that the parameter <span class="math-container">$\beta$</span> controls the <em>smoothness</em> of the function. In fact, we can define a <em>temperature</em> <span class="math-container">$T = \frac{1}{\beta}$</span> (much like the <em>softmax</em> function).
  Let's denote with <span class="math-container">$f_T$</span> a softplus function with temperature <span class="math-container">$T$</span>. We can easily see that as <span class="math-container">$T \rightarrow 0$</span> (or equivalently <span class="math-container">$\beta \rightarrow \infty$</span>), <span class="math-container">$f_T(x) \rightarrow \max(0, x)$</span>. In fact, let's compute the limit of the softplus for all <span class="math-container">$x &gt; 0$</span>:</p>
  <p><span class="math-container">$\lim_{\beta \rightarrow \infty} \frac{1}{\beta} ln(1 + e^{\beta x})$</span></p>
  <p>Rewrite <span class="math-container">$ln(1+e^{\beta x})$</span> as <span class="math-container">$ln(e^{-\beta x} + 1) +ln(e^{\beta x})$</span> we get:</p>
  <p><span class="math-container">$\lim_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(e^{-\beta x} + 1) +ln(e^{\beta x})] 
  = \\ = 
  \lim_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(0 + 1) + ln(e^{\beta x})]
  = \\ = 
  \lim_{\beta \rightarrow \infty} \frac{1}{\beta}[0 + ln(e^{\beta x})] 
  = \\ = 
  \lim_{\beta \rightarrow \infty} \frac{1}{\beta}\beta x = \\ = x$</span></p>
  <p>Computing the limit for <span class="math-container">$x \leq 0 $</span> is straightforward and the result is <span class="math-container">$0$</span>.
  So we see that for high value of <span class="math-container">$\beta$</span> (or low value of <span class="math-container">$T$</span>), the softplus is actually the <span class="math-container">$\max(0, x)$</span> function!</p>
  <p>Therefore, softplus is a <em>family</em> of functions <span class="math-container">$f_T$</span> which approximate ReLU with high precision as <span class="math-container">$T$</span> gets to 0 (less smoothness), and low precision with high <span class="math-container">$T$</span> (more smoothness).</p>
  <p>Among all, we usually use softplus with <span class="math-container">$beta = 1$</span> (i.e, <span class="math-container">$f_1$</span>), and this is why we call it a <em>smooth approximation</em> of the ReLU. Because <span class="math-container">$f_1(x)$</span> is smooth (<span class="math-container">$T=1$</span>) and approximates <span class="math-container">$f_\infty(x) = \max(0, x) = \text{ReLU}(x)$</span>.</p>
  <p>Hope this was useful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-30T13:08:14.133Z
Answers.LastActivityDate: 2021-07-30T13:19:06.657Z
Answers.LastEditDate: 2021-07-30T13:19:06.657Z
Answers.UserDisplayName: danin
Body: <p>Consider the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Softplus.html" rel="nofollow noreferrer">following details</a> regarding Softplus activation function</p>
  <blockquote>
  <p><span class="math-container">$$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$</span></p>
  <p>SoftPlus is a <strong>smooth approximation</strong> to the ReLU function and can be
  used to constrain the output of a machine to always be positive.</p>
  </blockquote>
  <p>It says that Softplus is a smooth approximation to the ReLU function. Let us consider the analytical form and plot of the RELU function.</p>
  <p><span class="math-container">$$\text{ReLU}(x)=(x)^+=\max(0,x)$$</span></p>
  <p><a href="https://i.sstatic.net/ruAwY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ruAwY.png" alt="enter image description here" /></a></p>
  <p>The plot of Softplus function is</p>
  <p><a href="https://i.sstatic.net/GJUAM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GJUAM.png" alt="enter image description here" /></a></p>
  <p>If we observe both plots, we can see the Softplus is almost similar to ReLU. There is a property for Softplus that ReLU does not have. <a href="https://ai.stackexchange.com/a/28588/18758">ReLU is not differentiable at zero and the derivative of ReLU is also not continuous</a>.</p>
  <p>If we observe the behavior of Softplus, it is <span class="math-container">$n-$</span>times continuously differentiable and hence a smooth function.</p>
  <p>Since Softplus is both a smooth function and approximates ReLU, it is considered as a smooth approximation of ReLU.</p>
  <p>Is my interpretation correct? if no, then what is meant by &quot;smooth approximation&quot; here?</p>
  
Tags: <comparison><terminology><definitions><activation-functions>
Title: Is my understanding on "smooth approximation" correct?
UserDisplayName: hanugm
ThreadText: # Is my understanding on "smooth approximation" correct?
  
  (Asked by: username_0 on 2021-07-29)
  
  Consider the [following details](https://pytorch.org/docs/stable/generated/torch.nn.Softplus.html) regarding Softplus activation function
  
  > $$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$
  >
  > SoftPlus is a **smooth approximation** to the ReLU function and can be
  > used to constrain the output of a machine to always be positive.
  
  It says that Softplus is a smooth approximation to the ReLU function. Let us consider the analytical form and plot of the RELU function.
  
  $$\text{ReLU}(x)=(x)^+=\max(0,x)$$
  
  [![enter image description here](https://i.sstatic.net/ruAwY.png)](https://i.sstatic.net/ruAwY.png)
  
  The plot of Softplus function is
  
  [![enter image description here](https://i.sstatic.net/GJUAM.png)](https://i.sstatic.net/GJUAM.png)
  
  If we observe both plots, we can see the Softplus is almost similar to ReLU. There is a property for Softplus that ReLU does not have. [ReLU is not differentiable at zero and the derivative of ReLU is also not continuous](https://ai.stackexchange.com/a/28588/18758).
  
  If we observe the behavior of Softplus, it is $n-$times continuously differentiable and hence a smooth function.
  
  Since Softplus is both a smooth function and approximates ReLU, it is considered as a smooth approximation of ReLU.
  
  Is my interpretation correct? if no, then what is meant by "smooth approximation" here?
  
  **Tags:** comparison, terminology, definitions, activation-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-30. Score: 1 | Accepted answer
  
  Your interpretation is definitely correct. As you correctly pointed out, the derivative of `softplus` is continuous and $n$-times differentiable, that makes the function smooth, which is not the case for `ReLU`.
  
  What is quite interesting here is *why* `softplus` can be called an approximation to `ReLU`.
  
  If we break down the definition of `softplus`, we note that the parameter $\beta$ controls the *smoothness* of the function. In fact, we can define a *temperature* $T = \frac{1}{\beta}$ (much like the *softmax* function).
  Let's denote with $f\_T$ a softplus function with temperature $T$. We can easily see that as $T \rightarrow 0$ (or equivalently $\beta \rightarrow \infty$), $f\_T(x) \rightarrow \max(0, x)$. In fact, let's compute the limit of the softplus for all $x > 0$:
  
  $\lim\_{\beta \rightarrow \infty} \frac{1}{\beta} ln(1 + e^{\beta x})$
  
  Rewrite $ln(1+e^{\beta x})$ as $ln(e^{-\beta x} + 1) +ln(e^{\beta x})$ we get:
  
  $\lim\_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(e^{-\beta x} + 1) +ln(e^{\beta x})]
  = \\ =
  \lim\_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(0 + 1) + ln(e^{\beta x})]
  = \\ =
  \lim\_{\beta \rightarrow \infty} \frac{1}{\beta}[0 + ln(e^{\beta x})]
  = \\ =
  \lim\_{\beta \rightarrow \infty} \frac{1}{\beta}\beta x = \\ = x$
  
  Computing the limit for $x \leq 0 $ is straightforward and the result is $0$.
  So we see that for high value of $\beta$ (or low value of $T$), the softplus is actually the $\max(0, x)$ function!
  
  Therefore, softplus is a *family* of functions $f\_T$ which approximate ReLU with high precision as $T$ gets to 0 (less smoothness), and low precision with high $T$ (more smoothness).
  
  Among all, we usually use softplus with $beta = 1$ (i.e, $f\_1$), and this is why we call it a *smooth approximation* of the ReLU. Because $f\_1(x)$ is smooth ($T=1$) and approximates $f\_\infty(x) = \max(0, x) = \text{ReLU}(x)$.
  
  Hope this was useful.
  
  
  ---
  
  