Answers.Body: <p>If <span class="math-container">$d$</span> is a positive real number different from <span class="math-container">$1$</span>, then</p>
  <p><span class="math-container">$$d^{-x}=e^{-x\ln(d)}$$</span></p>
  <p>So <span class="math-container">$d^{-x}$</span> is obtained from <span class="math-container">$e^{-x}$</span> by a horizontal shrink (when <span class="math-container">$\ln(d)&gt;1$</span>, that is <span class="math-container">$d&gt;e$</span>) or by a horizontal stretch (when <span class="math-container">$\ln(d)&lt;1$</span>, that is <span class="math-container">$0&lt;d&lt;e$</span>).</p>
  <p>The general shape of the graph is the same but it is raising faster from (close to) <span class="math-container">$0$</span> to (close to) <span class="math-container">$1$</span> when <span class="math-container">$d$</span> is large.</p>
  <p>The choice of <span class="math-container">$e$</span> is convenient as the derivative of <span class="math-container">$e^x$</span> is slightly simpler than <span class="math-container">$d^x$</span> (as explained by @Sycorax), making it the default choice in the mathematical literature.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-14T06:30:18.163Z
Answers.Text: Worth noting that the horizontal scaling is learnable by the network by scaling the weights. So no generalisation is lost.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-14T03:40:51.253Z
Answers.LastActivityDate: 2023-08-14T03:40:51.253Z
Answers.UserDisplayName: Taladris
Answers.Body: <p>To add to other answers: Note that the usefulness of <span class="math-container">$e$</span> as the base is not limited to this particular case of sigmoid activation function. It is <strong>the go-to base</strong> in so many areas of mathematics because of many nice properties (including the reasons given in other answers); see e.g. the <a href="https://en.wikipedia.org/wiki/Exponential_function" rel="noreferrer">exponential function</a> in Wikipedia.</p>
  <p>In fact, the choice is so natural (pun intended) that if any other base was chosen, people would ask the question &quot;<span class="math-container">$d$</span> is unnecessary, why not just use <span class="math-container">$e$</span> as the base?&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-14T11:52:53.017Z
Answers.LastActivityDate: 2023-08-14T11:52:53.017Z
Answers.UserDisplayName: JiK
Answers.Body: <p><span class="math-container">$f$</span> is the unique function such that <span class="math-container">$f(0) = \frac{1}{2}$</span> and <span class="math-container">$f'(x) = f(x)(1 - f(x))$</span>. Using <span class="math-container">$e$</span> is necessary to make sure the derivative takes this very simple form.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-15T15:47:58.077Z
Answers.Text: This combines with binary cross entropy loss, to make a simple combined gradient with respect to the logits in logistic regression and binary classifiers.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-15T02:44:53.207Z
Answers.LastActivityDate: 2023-08-15T02:44:53.207Z
Answers.UserDisplayName: Charles Hudgins
Answers.Body: <p>The choice of <span class="math-container">$e$</span> is convenient when taking derivatives.</p>
  <p>Compare <span class="math-container">$\frac{d}{dx} \exp(x)$</span> to <span class="math-container">$\frac{d}{dx} a^x$</span> for any other <span class="math-container">$a &gt; 0$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-13T22:40:29.027Z
Answers.LastActivityDate: 2023-08-15T02:21:23.463Z
Answers.LastEditDate: 2023-08-15T02:21:23.463Z
Answers.UserDisplayName: Sycorax
Body: <p>Within the Sigmoid Squishification function,</p>
  <pre><code>f(x) = 1/(1 + e^(-x))
  </code></pre>
  <p>&quot;e&quot; is unnecessary, as it can be replaced by any other value that is not <code>0</code> or <code>1</code>. Why is &quot;e&quot; used here?</p>
  <p>As shown below, the function is working well without that, and in replacement, any other number that's greater than 1. All of them</p>
  <ul>
  <li>Squish the number between 0 and 1</li>
  <li>Reach (0, 0.5)</li>
  <li>Make an &quot;S&quot; curve</li>
  <li>Has a working derivative</li>
  <li>Have similar derivatives, with Maximas varying on the replacement of Euler's number</li>
  </ul>
  <p>The function and derivative with &quot;d&quot; as the parameter replacement can be written as:</p>
  <pre><code>const sigmoid = (x, d) =&gt; 1/(1 + d**(-x));
  const sigmoid_derivative = (x, d) =&gt; (d**x) * Math.log(d) / ((d**(x)) + 1)**2;
  </code></pre>
  <p><a href="https://www.desmos.com/calculator/xpkhdijt3v" rel="noreferrer">https://www.desmos.com/calculator/xpkhdijt3v</a></p>
  <p><a href="https://i.sstatic.net/6Og3l.png" rel="noreferrer"><img src="https://i.sstatic.net/6Og3l.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-14T06:20:55.980Z
Comments.Text: Using $e^{-x}$ is a writing convention, what you really use is the standard library function $\exp(-x)$. The implementation of $\exp(x)$ is optimized, with guaranteed error rate. The power function for $d^x$ is composite of other primitive library functions, with much overhead to guarantee similar error rates.
Comments.UserDisplayName: Lutz Lehmann
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-14T14:48:56.780Z
Comments.Text: Remember that $a^x=e^{x log a}$. So yeah, they are all the same, its just easier to work with $e$
Comments.UserDisplayName: Ander Biguri
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-15T02:47:22.397Z
Comments.Text: The point of using the sigmoid with $e$ is that you can simply calculate the derivative using the equation $f'(x) = f(x)(1 - f(x))$. Using a built-in method to calculate the derivative defeats the purpose of using the sigmoid function, which is that it has a derivative with a simple algebraic form in terms of $f$.
Comments.UserDisplayName: Charles Hudgins
Tags: <sigmoid>
Title: What does "e" do in the Sigmoid Activation Function?
UserDisplayName: Jake
ThreadText: # What does "e" do in the Sigmoid Activation Function?
  
  (Asked by: username_0 on 2023-08-13)
  
  Within the Sigmoid Squishification function,
  
  ```
  f(x) = 1/(1 + e^(-x))
  
  ```
  
  "e" is unnecessary, as it can be replaced by any other value that is not `0` or `1`. Why is "e" used here?
  
  As shown below, the function is working well without that, and in replacement, any other number that's greater than 1. All of them
  
  * Squish the number between 0 and 1
  * Reach (0, 0.5)
  * Make an "S" curve
  * Has a working derivative
  * Have similar derivatives, with Maximas varying on the replacement of Euler's number
  
  The function and derivative with "d" as the parameter replacement can be written as:
  
  ```
  const sigmoid = (x, d) => 1/(1 + d**(-x));
  const sigmoid_derivative = (x, d) => (d**x) * Math.log(d) / ((d**(x)) + 1)**2;
  
  ```
  
  <https://www.desmos.com/calculator/xpkhdijt3v>
  
  [![enter image description here](https://i.sstatic.net/6Og3l.png)](https://i.sstatic.net/6Og3l.png)
  
  **Tags:** sigmoid
  
  > **Comments:**
  > **username_1**: Using $e^{-x}$ is a writing convention, what you really use is the standard library function $\exp(-x)$. The implementation of $\exp(x)$ is optimized, with guaranteed error rate. The power function for $d^x$ is composite of other primitive library functions, with much overhead to guarantee similar error rates.  
  > **username_2**: Remember that $a^x=e^{x log a}$. So yeah, they are all the same, its just easier to work with $e$  
  > **username_3**: The point of using the sigmoid with $e$ is that you can simply calculate the derivative using the equation $f'(x) = f(x)(1 - f(x))$. Using a built-in method to calculate the derivative defeats the purpose of using the sigmoid function, which is that it has a derivative with a simple algebraic form in terms of $f$.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_7 on 2023-08-13. Score: 18 | Accepted answer
  
  The choice of $e$ is convenient when taking derivatives.
  
  Compare $\frac{d}{dx} \exp(x)$ to $\frac{d}{dx} a^x$ for any other $a > 0$.
  
  
  ---
  
  ### Answer by username_4 on 2023-08-14. Score: 5
  
  If $d$ is a positive real number different from $1$, then
  
  $$d^{-x}=e^{-x\ln(d)}$$
  
  So $d^{-x}$ is obtained from $e^{-x}$ by a horizontal shrink (when $\ln(d)>1$, that is $d>e$) or by a horizontal stretch (when $\ln(d)<1$, that is $0<d<e$).
  
  The general shape of the graph is the same but it is raising faster from (close to) $0$ to (close to) $1$ when $d$ is large.
  
  The choice of $e$ is convenient as the derivative of $e^x$ is slightly simpler than $d^x$ (as explained by @username_7), making it the default choice in the mathematical literature.
  
  > **Comments:**
  > **username_5**: Worth noting that the horizontal scaling is learnable by the network by scaling the weights. So no generalisation is lost.  
  
  ---
  
  ### Answer by username_6 on 2023-08-14. Score: 5
  
  To add to other answers: Note that the usefulness of $e$ as the base is not limited to this particular case of sigmoid activation function. It is **the go-to base** in so many areas of mathematics because of many nice properties (including the reasons given in other answers); see e.g. the [exponential function](https://en.wikipedia.org/wiki/Exponential_function) in Wikipedia.
  
  In fact, the choice is so natural (pun intended) that if any other base was chosen, people would ask the question "$d$ is unnecessary, why not just use $e$ as the base?"
  
  
  ---
  
  ### Answer by username_3 on 2023-08-15. Score: 2
  
  $f$ is the unique function such that $f(0) = \frac{1}{2}$ and $f'(x) = f(x)(1 - f(x))$. Using $e$ is necessary to make sure the derivative takes this very simple form.
  
  > **Comments:**
  > **username_5**: This combines with binary cross entropy loss, to make a simple combined gradient with respect to the logits in logistic regression and binary classifiers.  
  
  ---
  
  