<p>The question must be split in two:</p>
<p>Use in softmax: softmax is based in the concept of cross entropy, logistic regression, logistic function ... . All these concepts includes exponential or logarithmic in its formulation.</p>
<p>Use in activation functions:</p>
<ul>
<li>monotonic: a non-monotonic function will give same output to different inputs, disturbing the concept of distance/error.</li>
<li>non-linear: a linear activation function collapses the layer with the next one, thus, it is useless.</li>
<li>differentiable: necessary to use back-propagation.</li>
</ul>

CC BY-SA 4.0
2022-02-13T17:08:25.967Z
2022-02-13T17:08:25.967Z
pasaba por aqui
<p>While all the properties you list are true, the essential parts are the <strong>Differentiability</strong> and the fact that you build a <strong>nice probability distribution from it</strong>.</p>
<p>The unnormalized scores the neural network puts out <em>before</em> we apply softmax to them may very well contain negative values. Applying the exponential function is a great way to build a probability distribution from a given vector, disregarding the sign of the values.</p>

CC BY-SA 4.0
2021-09-16T10:29:56.393Z
2021-09-16T10:29:56.393Z
tnfru
<p>In mathematics, there is a proof that the following infinite series converges to a constant irrational number, denoted by <span class="math-container">$e$</span>, called as Euler's number or Euler's constant or Napier's constant. The value of <span class="math-container">$e$</span> lies between 2 and 3.</p>
<p><span class="math-container">$$1 + \dfrac{1}{1!} +  \dfrac{1}{2!} +  \dfrac{1}{3!} + \cdots$$</span></p>
<p>The natural exponential function, defined as follows, has some interesting properties</p>
<p><span class="math-container">$$f: \mathbb{R} \rightarrow  \mathbb{R}$$</span>
<span class="math-container">$$f(x) = e^x$$</span></p>
<p>It is used in several algorithms and in the definitions of functions like <a href="https://ai.stackexchange.com/questions/22426/which-paper-introduced-the-term-softmax">SoftMax</a>.
I am interested in knowing the possible mathematical characteristics that lead this function useful in artificial intelligence.</p>
<p>The following are the properties I am aware of. But, I am not sure about how some of them will be useful</p>
<ol>
<li><p><strong>Non-linearlity</strong>: Activation functions are intended to provide non-linearity. So, it is a candidate for activation functions due to this property. You can check its graph <a href="https://www.desmos.com/calculator/1cxgy5ry6k" rel="nofollow noreferrer">here</a>.</p>
</li>
<li><p><strong>Differentiability</strong>:  Loss functions used for in back-propagation algorithm <a href="https://ai.stackexchange.com/a/28884/18758">need to be differentiable</a>. So, it can be a candidate for usage in loss functions too.</p>
</li>
</ol>
<p><span class="math-container">$$\dfrac{d}{dx} e^x = e^x \text{ for all } x \in \mathbb{R}$$</span></p>
<ol start="3">
<li><p><strong>Continuity</strong>: I am not sure how this property is useful in algorithms. Intuitively, you can check from graph provided above that it is continuous.</p>
</li>
<li><p><strong>Smoothness</strong>: I am not sure how this property is useful in algorithms. But <a href="https://ai.stackexchange.com/q/29896/18758">seems useful</a>. The natural exponential function <a href="https://ai.stackexchange.com/a/28588/18758">has the smoothness property</a>.</p>
</li>
</ol>
<p><span class="math-container">$$\dfrac{d^n}{d^nx} e^x = e^x \text{ for all } x \in \mathbb{R} \text{ and } n \in \mathbb{N}$$</span>.</p>
<p>Are there any other properties like non-linearily, differentiability, smoothness etc., for the natural exponential function that make it superior to use in AI algorithms?</p>

CC BY-SA 4.0
2022-03-02T14:48:43.490Z
I think this question is a bit broad because a function may be used in different ways, in different contexts, and for different purposes. I think you're interested in ML (and maybe specifically in neural networks) and not the whole AI. If that's the case, I would start by replacing "AI" with "ML" or be more specific, then I would try to motivate a little more why you're asking this question. For example, in logic-based AI, maybe the characteristics that you mention are not even useful. Where would a function be used in those contexts? These are all issues that I see with this post.
nbro
CC BY-SA 4.0
2022-03-03T10:13:07.807Z
@nbro But I am not sure there are many branches in AI that use $e^x$. And it is true that there are plenty of branches, but, if we view from the perspective of function only, there are finite properties for $e^x$. Am I wrong anywhere?
hanugm
<math>
What are the mathematical properties of natural exponential function that lead to its usefulness in artificial intelligence?
hanugm
# What are the mathematical properties of natural exponential function that lead to its usefulness in artificial intelligence?

(Asked by: username_0 on 2021-08-17)

In mathematics, there is a proof that the following infinite series converges to a constant irrational number, denoted by $e$, called as Euler's number or Euler's constant or Napier's constant. The value of $e$ lies between 2 and 3.

$$1 + \dfrac{1}{1!} + \dfrac{1}{2!} + \dfrac{1}{3!} + \cdots$$

The natural exponential function, defined as follows, has some interesting properties

$$f: \mathbb{R} \rightarrow \mathbb{R}$$
$$f(x) = e^x$$

It is used in several algorithms and in the definitions of functions like [SoftMax](https://ai.stackexchange.com/questions/22426/which-paper-introduced-the-term-softmax).
I am interested in knowing the possible mathematical characteristics that lead this function useful in artificial intelligence.

The following are the properties I am aware of. But, I am not sure about how some of them will be useful

1. **Non-linearlity**: Activation functions are intended to provide non-linearity. So, it is a candidate for activation functions due to this property. You can check its graph [here](https://www.desmos.com/calculator/1cxgy5ry6k).
2. **Differentiability**: Loss functions used for in back-propagation algorithm [need to be differentiable](https://ai.stackexchange.com/a/28884/18758). So, it can be a candidate for usage in loss functions too.

$$\dfrac{d}{dx} e^x = e^x \text{ for all } x \in \mathbb{R}$$

3. **Continuity**: I am not sure how this property is useful in algorithms. Intuitively, you can check from graph provided above that it is continuous.
4. **Smoothness**: I am not sure how this property is useful in algorithms. But [seems useful](https://ai.stackexchange.com/q/29896/18758). The natural exponential function [has the smoothness property](https://ai.stackexchange.com/a/28588/18758).

$$\dfrac{d^n}{d^nx} e^x = e^x \text{ for all } x \in \mathbb{R} \text{ and } n \in \mathbb{N}$$.

Are there any other properties like non-linearily, differentiability, smoothness etc., for the natural exponential function that make it superior to use in AI algorithms?

**Tags:** math

> **Comments:**
> **username_1**: I think this question is a bit broad because a function may be used in different ways, in different contexts, and for different purposes. I think you're interested in ML (and maybe specifically in neural networks) and not the whole AI. If that's the case, I would start by replacing "AI" with "ML" or be more specific, then I would try to motivate a little more why you're asking this question. For example, in logic-based AI, maybe the characteristics that you mention are not even useful. Where would a function be used in those contexts? These are all issues that I see with this post.  
> **username_0**: @username_1 But I am not sure there are many branches in AI that use $e^x$. And it is true that there are plenty of branches, but, if we view from the perspective of function only, there are finite properties for $e^x$. Am I wrong anywhere?  

---

## 2 answers

### Answer by username_2 on 2022-02-13. Score: 2

The question must be split in two:

Use in softmax: softmax is based in the concept of cross entropy, logistic regression, logistic function ... . All these concepts includes exponential or logarithmic in its formulation.

Use in activation functions:

* monotonic: a non-monotonic function will give same output to different inputs, disturbing the concept of distance/error.
* non-linear: a linear activation function collapses the layer with the next one, thus, it is useless.
* differentiable: necessary to use back-propagation.


---

### Answer by username_3 on 2021-09-16. Score: 1

While all the properties you list are true, the essential parts are the **Differentiability** and the fact that you build a **nice probability distribution from it**.

The unnormalized scores the neural network puts out *before* we apply softmax to them may very well contain negative values. Applying the exponential function is a great way to build a probability distribution from a given vector, disregarding the sign of the values.


---

