Answers.Body: <p>If <span class="math-container">$d$</span> is a positive real number different from <span class="math-container">$1$</span>, then</p>
  <p><span class="math-container">$$d^{-x}=e^{-x\ln(d)}$$</span></p>
  <p>So <span class="math-container">$d^{-x}$</span> is obtained from <span class="math-container">$e^{-x}$</span> by a horizontal shrink (when <span class="math-container">$\ln(d)&gt;1$</span>, that is <span class="math-container">$d&gt;e$</span>) or by a horizontal stretch (when <span class="math-container">$\ln(d)&lt;1$</span>, that is <span class="math-container">$0&lt;d&lt;e$</span>).</p>
  <p>The general shape of the graph is the same but it is raising faster from (close to) <span class="math-container">$0$</span> to (close to) <span class="math-container">$1$</span> when <span class="math-container">$d$</span> is large.</p>
  <p>The choice of <span class="math-container">$e$</span> is convenient as the derivative of <span class="math-container">$e^x$</span> is slightly simpler than <span class="math-container">$d^x$</span> (as explained by @Sycorax), making it the default choice in the mathematical literature.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-14T06:30:18.163Z
Answers.Text: Worth noting that the horizontal scaling is learnable by the network by scaling the weights. So no generalisation is lost.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-14T03:40:51.253Z
Answers.LastActivityDate: 2023-08-14T03:40:51.253Z
Answers.UserDisplayName: Taladris
Answers.Body: <p>To add to other answers: Note that the usefulness of <span class="math-container">$e$</span> as the base is not limited to this particular case of sigmoid activation function. It is <strong>the go-to base</strong> in so many areas of mathematics because of many nice properties (including the reasons given in other answers); see e.g. the <a href="https://en.wikipedia.org/wiki/Exponential_function" rel="noreferrer">exponential function</a> in Wikipedia.</p>
  <p>In fact, the choice is so natural (pun intended) that if any other base was chosen, people would ask the question &quot;<span class="math-container">$d$</span> is unnecessary, why not just use <span class="math-container">$e$</span> as the base?&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-14T11:52:53.017Z
Answers.LastActivityDate: 2023-08-14T11:52:53.017Z
Answers.UserDisplayName: JiK
Answers.Body: <p><span class="math-container">$f$</span> is the unique function such that <span class="math-container">$f(0) = \frac{1}{2}$</span> and <span class="math-container">$f'(x) = f(x)(1 - f(x))$</span>. Using <span class="math-container">$e$</span> is necessary to make sure the derivative takes this very simple form.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-15T15:47:58.077Z
Answers.Text: This combines with binary cross entropy loss, to make a simple combined gradient with respect to the logits in logistic regression and binary classifiers.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-15T02:44:53.207Z
Answers.LastActivityDate: 2023-08-15T02:44:53.207Z
Answers.UserDisplayName: Charles Hudgins
Answers.Body: <p>The choice of <span class="math-container">$e$</span> is convenient when taking derivatives.</p>
  <p>Compare <span class="math-container">$\frac{d}{dx} \exp(x)$</span> to <span class="math-container">$\frac{d}{dx} a^x$</span> for any other <span class="math-container">$a &gt; 0$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-13T22:40:29.027Z
Answers.LastActivityDate: 2023-08-15T02:21:23.463Z
Answers.LastEditDate: 2023-08-15T02:21:23.463Z
Answers.UserDisplayName: Sycorax
Body: <p>Within the Sigmoid Squishification function,</p>
  <pre><code>f(x) = 1/(1 + e^(-x))
  </code></pre>
  <p>&quot;e&quot; is unnecessary, as it can be replaced by any other value that is not <code>0</code> or <code>1</code>. Why is &quot;e&quot; used here?</p>
  <p>As shown below, the function is working well without that, and in replacement, any other number that's greater than 1. All of them</p>
  <ul>
  <li>Squish the number between 0 and 1</li>
  <li>Reach (0, 0.5)</li>
  <li>Make an &quot;S&quot; curve</li>
  <li>Has a working derivative</li>
  <li>Have similar derivatives, with Maximas varying on the replacement of Euler's number</li>
  </ul>
  <p>The function and derivative with &quot;d&quot; as the parameter replacement can be written as:</p>
  <pre><code>const sigmoid = (x, d) =&gt; 1/(1 + d**(-x));
  const sigmoid_derivative = (x, d) =&gt; (d**x) * Math.log(d) / ((d**(x)) + 1)**2;
  </code></pre>
  <p><a href="https://www.desmos.com/calculator/xpkhdijt3v" rel="noreferrer">https://www.desmos.com/calculator/xpkhdijt3v</a></p>
  <p><a href="https://i.sstatic.net/6Og3l.png" rel="noreferrer"><img src="https://i.sstatic.net/6Og3l.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-14T06:20:55.980Z
Comments.Text: Using $e^{-x}$ is a writing convention, what you really use is the standard library function $\exp(-x)$. The implementation of $\exp(x)$ is optimized, with guaranteed error rate. The power function for $d^x$ is composite of other primitive library functions, with much overhead to guarantee similar error rates.
Comments.UserDisplayName: Lutz Lehmann
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-14T14:48:56.780Z
Comments.Text: Remember that $a^x=e^{x log a}$. So yeah, they are all the same, its just easier to work with $e$
Comments.UserDisplayName: Ander Biguri
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-15T02:47:22.397Z
Comments.Text: The point of using the sigmoid with $e$ is that you can simply calculate the derivative using the equation $f'(x) = f(x)(1 - f(x))$. Using a built-in method to calculate the derivative defeats the purpose of using the sigmoid function, which is that it has a derivative with a simple algebraic form in terms of $f$.
Comments.UserDisplayName: Charles Hudgins
Tags: <sigmoid>
Title: What does "e" do in the Sigmoid Activation Function?
UserDisplayName: Jake
ThreadText: # What does "e" do in the Sigmoid Activation Function?
  
  (Asked by: username_0 on 2023-08-13)
  
  Within the Sigmoid Squishification function,
  f(x) = 1/(1 + e^(-x))
  
  "e" is unnecessary, as it can be replaced by any other value that is not 0 or 1. Why is "e" used here?
  As shown below, the function is working well without that, and in replacement, any other number that's greater than 1. All of them
  
  Squish the number between 0 and 1
  Reach (0, 0.5)
  Make an "S" curve
  Has a working derivative
  Have similar derivatives, with Maximas varying on the replacement of Euler's number
  
  The function and derivative with "d" as the parameter replacement can be written as:
  const sigmoid = (x, d) => 1/(1 + d**(-x));
  const sigmoid_derivative = (x, d) => (d**x) * Math.log(d) / ((d**(x)) + 1)**2;
  
  https://www.desmos.com/calculator/xpkhdijt3v
  
  **Tags:** sigmoid
  > **Comments:**
  > **username_1**: Using $e^{-x}$ is a writing convention, what you really use is the standard library function $\exp(-x)$. The implementation of $\exp(x)$ is optimized, with guaranteed error rate. The power function for $d^x$ is composite of other primitive library functions, with much overhead to guarantee similar error rates.  
  > **username_2**: Remember that $a^x=e^{x log a}$. So yeah, they are all the same, its just easier to work with $e$  
  > **username_3**: The point of using the sigmoid with $e$ is that you can simply calculate the derivative using the equation $f'(x) = f(x)(1 - f(x))$. Using a built-in method to calculate the derivative defeats the purpose of using the sigmoid function, which is that it has a derivative with a simple algebraic form in terms of $f$.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_7 on 2023-08-13. Score: 18 | Accepted answer
  
  The choice of $e$ is convenient when taking derivatives.
  
  Compare $\frac{d}{dx} \exp(x)$ to $\frac{d}{dx} a^x$ for any other $a > 0$.
  
  
  ---
  
  ### Answer by username_4 on 2023-08-14. Score: 5
  
  If $d$ is a positive real number different from $1$, then
  
  $$d^{-x}=e^{-x\ln(d)}$$
  
  So $d^{-x}$ is obtained from $e^{-x}$ by a horizontal shrink (when $\ln(d)>1$, that is $d>e$) or by a horizontal stretch (when $\ln(d)<1$, that is $0).
  
  The general shape of the graph is the same but it is raising faster from (close to) $0$ to (close to) $1$ when $d$ is large.
  
  The choice of $e$ is convenient as the derivative of $e^x$ is slightly simpler than $d^x$ (as explained by @Sycorax), making it the default choice in the mathematical literature.
  
  > **Comments:**
  > **username_5**: Worth noting that the horizontal scaling is learnable by the network by scaling the weights. So no generalisation is lost.  
  
  ---
  
  ### Answer by username_6 on 2023-08-14. Score: 5
  
  To add to other answers: Note that the usefulness of $e$ as the base is not limited to this particular case of sigmoid activation function. It is **the go-to base** in so many areas of mathematics because of many nice properties (including the reasons given in other answers); see e.g. the [exponential function](https://en.wikipedia.org/wiki/Exponential_function) in Wikipedia.
  
  In fact, the choice is so natural (pun intended) that if any other base was chosen, people would ask the question "$d$ is unnecessary, why not just use $e$ as the base?"
  
  
  ---
  
  ### Answer by username_3 on 2023-08-15. Score: 2
  
  $f$ is the unique function such that $f(0) = \frac{1}{2}$ and $f'(x) = f(x)(1 - f(x))$. Using $e$ is necessary to make sure the derivative takes this very simple form.
  
  > **Comments:**
  > **username_5**: This combines with binary cross entropy loss, to make a simple combined gradient with respect to the logits in logistic regression and binary classifiers.  
  
  ---
  
  