Answers.Body: <p>The question must be split in two:</p>
  <p>Use in softmax: softmax is based in the concept of cross entropy, logistic regression, logistic function ... . All these concepts includes exponential or logarithmic in its formulation.</p>
  <p>Use in activation functions:</p>
  <ul>
  <li>monotonic: a non-monotonic function will give same output to different inputs, disturbing the concept of distance/error.</li>
  <li>non-linear: a linear activation function collapses the layer with the next one, thus, it is useless.</li>
  <li>differentiable: necessary to use back-propagation.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T17:08:25.967Z
Answers.LastActivityDate: 2022-02-13T17:08:25.967Z
Answers.UserDisplayName: pasaba por aqui
Answers.Body: <p>While all the properties you list are true, the essential parts are the <strong>Differentiability</strong> and the fact that you build a <strong>nice probability distribution from it</strong>.</p>
  <p>The unnormalized scores the neural network puts out <em>before</em> we apply softmax to them may very well contain negative values. Applying the exponential function is a great way to build a probability distribution from a given vector, disregarding the sign of the values.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-16T10:29:56.393Z
Answers.LastActivityDate: 2021-09-16T10:29:56.393Z
Answers.UserDisplayName: tnfru
Body: <p>In mathematics, there is a proof that the following infinite series converges to a constant irrational number, denoted by <span class="math-container">$e$</span>, called as Euler's number or Euler's constant or Napier's constant. The value of <span class="math-container">$e$</span> lies between 2 and 3.</p>
  <p><span class="math-container">$$1 + \dfrac{1}{1!} +  \dfrac{1}{2!} +  \dfrac{1}{3!} + \cdots$$</span></p>
  <p>The natural exponential function, defined as follows, has some interesting properties</p>
  <p><span class="math-container">$$f: \mathbb{R} \rightarrow  \mathbb{R}$$</span>
  <span class="math-container">$$f(x) = e^x$$</span></p>
  <p>It is used in several algorithms and in the definitions of functions like <a href="https://ai.stackexchange.com/questions/22426/which-paper-introduced-the-term-softmax">SoftMax</a>.
  I am interested in knowing the possible mathematical characteristics that lead this function useful in artificial intelligence.</p>
  <p>The following are the properties I am aware of. But, I am not sure about how some of them will be useful</p>
  <ol>
  <li><p><strong>Non-linearlity</strong>: Activation functions are intended to provide non-linearity. So, it is a candidate for activation functions due to this property. You can check its graph <a href="https://www.desmos.com/calculator/1cxgy5ry6k" rel="nofollow noreferrer">here</a>.</p>
  </li>
  <li><p><strong>Differentiability</strong>:  Loss functions used for in back-propagation algorithm <a href="https://ai.stackexchange.com/a/28884/18758">need to be differentiable</a>. So, it can be a candidate for usage in loss functions too.</p>
  </li>
  </ol>
  <p><span class="math-container">$$\dfrac{d}{dx} e^x = e^x \text{ for all } x \in \mathbb{R}$$</span></p>
  <ol start="3">
  <li><p><strong>Continuity</strong>: I am not sure how this property is useful in algorithms. Intuitively, you can check from graph provided above that it is continuous.</p>
  </li>
  <li><p><strong>Smoothness</strong>: I am not sure how this property is useful in algorithms. But <a href="https://ai.stackexchange.com/q/29896/18758">seems useful</a>. The natural exponential function <a href="https://ai.stackexchange.com/a/28588/18758">has the smoothness property</a>.</p>
  </li>
  </ol>
  <p><span class="math-container">$$\dfrac{d^n}{d^nx} e^x = e^x \text{ for all } x \in \mathbb{R} \text{ and } n \in \mathbb{N}$$</span>.</p>
  <p>Are there any other properties like non-linearily, differentiability, smoothness etc., for the natural exponential function that make it superior to use in AI algorithms?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-02T14:48:43.490Z
Comments.Text: I think this question is a bit broad because a function may be used in different ways, in different contexts, and for different purposes. I think you're interested in ML (and maybe specifically in neural networks) and not the whole AI. If that's the case, I would start by replacing "AI" with "ML" or be more specific, then I would try to motivate a little more why you're asking this question. For example, in logic-based AI, maybe the characteristics that you mention are not even useful. Where would a function be used in those contexts? These are all issues that I see with this post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-03T10:13:07.807Z
Comments.Text: @nbro But I am not sure there are many branches in AI that use $e^x$. And it is true that there are plenty of branches, but, if we view from the perspective of function only, there are finite properties for $e^x$. Am I wrong anywhere?
Comments.UserDisplayName: hanugm
Tags: <math>
Title: What are the mathematical properties of natural exponential function that lead to its usefulness in artificial intelligence?
UserDisplayName: hanugm
ThreadText: # What are the mathematical properties of natural exponential function that lead to its usefulness in artificial intelligence?
  
  (Asked by: username_0 on 2021-08-17)
  
  In mathematics, there is a proof that the following infinite series converges to a constant irrational number, denoted by $e$, called as Euler's number or Euler's constant or Napier's constant. The value of $e$ lies between 2 and 3.
  
  $$1 + \dfrac{1}{1!} + \dfrac{1}{2!} + \dfrac{1}{3!} + \cdots$$
  
  The natural exponential function, defined as follows, has some interesting properties
  
  $$f: \mathbb{R} \rightarrow \mathbb{R}$$
  $$f(x) = e^x$$
  
  It is used in several algorithms and in the definitions of functions like [SoftMax](https://ai.stackexchange.com/questions/22426/which-paper-introduced-the-term-softmax).
  I am interested in knowing the possible mathematical characteristics that lead this function useful in artificial intelligence.
  
  The following are the properties I am aware of. But, I am not sure about how some of them will be useful
  
  1. **Non-linearlity**: Activation functions are intended to provide non-linearity. So, it is a candidate for activation functions due to this property. You can check its graph [here](https://www.desmos.com/calculator/1cxgy5ry6k).
  2. **Differentiability**: Loss functions used for in back-propagation algorithm [need to be differentiable](https://ai.stackexchange.com/a/28884/18758). So, it can be a candidate for usage in loss functions too.
  
  $$\dfrac{d}{dx} e^x = e^x \text{ for all } x \in \mathbb{R}$$
  
  3. **Continuity**: I am not sure how this property is useful in algorithms. Intuitively, you can check from graph provided above that it is continuous.
  4. **Smoothness**: I am not sure how this property is useful in algorithms. But [seems useful](https://ai.stackexchange.com/q/29896/18758). The natural exponential function [has the smoothness property](https://ai.stackexchange.com/a/28588/18758).
  
  $$\dfrac{d^n}{d^nx} e^x = e^x \text{ for all } x \in \mathbb{R} \text{ and } n \in \mathbb{N}$$.
  
  Are there any other properties like non-linearily, differentiability, smoothness etc., for the natural exponential function that make it superior to use in AI algorithms?
  
  **Tags:** math
  > **Comments:**
  > **username_1**: I think this question is a bit broad because a function may be used in different ways, in different contexts, and for different purposes. I think you're interested in ML (and maybe specifically in neural networks) and not the whole AI. If that's the case, I would start by replacing "AI" with "ML" or be more specific, then I would try to motivate a little more why you're asking this question. For example, in logic-based AI, maybe the characteristics that you mention are not even useful. Where would a function be used in those contexts? These are all issues that I see with this post.  
  > **username_0**: @nbro But I am not sure there are many branches in AI that use $e^x$. And it is true that there are plenty of branches, but, if we view from the perspective of function only, there are finite properties for $e^x$. Am I wrong anywhere?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-02-13. Score: 2
  
  The question must be split in two:
  
  Use in softmax: softmax is based in the concept of cross entropy, logistic regression, logistic function ... . All these concepts includes exponential or logarithmic in its formulation.
  
  Use in activation functions:
  
  * monotonic: a non-monotonic function will give same output to different inputs, disturbing the concept of distance/error.
  * non-linear: a linear activation function collapses the layer with the next one, thus, it is useless.
  * differentiable: necessary to use back-propagation.
  
  
  ---
  
  ### Answer by username_3 on 2021-09-16. Score: 1
  
  While all the properties you list are true, the essential parts are the **Differentiability** and the fact that you build a **nice probability distribution from it**.
  
  The unnormalized scores the neural network puts out *before* we apply softmax to them may very well contain negative values. Applying the exponential function is a great way to build a probability distribution from a given vector, disregarding the sign of the values.
  
  
  ---
  
  