Answers.Body: <p>IEEE 754 provides sufficient precision for most ML tasks even with base <span class="math-container">$e$</span>, and the choice in softmax and sigmoid functions is rooted in mathematical and practical considerations.</p>
  <p>Notice the function <span class="math-container">$e^x$</span> has a unique property that its derivative is itself, therefore this simplifies gradient calculations during backpropagation of the common CE loss. Changing the base <span class="math-container">$2$</span> doesn't gain much precision but complicates existing math formulations in machine learning field and optimization computations potentially across many places. Therefore both softmax and sigmoid functions often used in probability and information theory typically use natural logarithm and exponential function based on <span class="math-container">$e$</span> in practice, such as the CE loss implemented in PyTorch. Often in information-theoretic discussions the base <span class="math-container">$2$</span> for the unit of bits is preferred such as Shanon's original paper about entropy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-19T01:52:30.527Z
Answers.LastActivityDate: 2024-12-19T21:04:48.907Z
Answers.LastEditDate: 2024-12-19T21:04:48.907Z
Answers.UserDisplayName: cinch
Answers.Body: <p>The base of the logarithm change the range, but it not necessarily affect the precision.</p>
  <p>There exists the following formula:</p>
  <p><span class="math-container">$\log_b(x) = \log(x) \cdot \frac{1}{\log(b)}$</span></p>
  <p>In which b is the base.</p>
  <p>Then, the selection of the base affect the range. Larger bases generate smaller ranges.</p>
  <p>Also, computing gradients with log with base e is more easy that when using other bases. Gradients are needed for backpropagation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-18T19:26:46.133Z
Answers.LastActivityDate: 2024-12-19T12:49:09.703Z
Answers.LastEditDate: 2024-12-19T12:49:09.703Z
Answers.UserDisplayName: Patricio Loncomilla
Answers.Body: <p><span class="math-container">$\log_2$</span> results in larger numbers than <span class="math-container">$\ln$</span>, but that doesn't make them more precise. Floating point arithmetic has the same relative precision for numbers of any size.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-20T08:56:43.830Z
Answers.LastActivityDate: 2024-12-20T08:56:43.830Z
Answers.UserDisplayName: Tomek Czajka
Body: <p>Performing <code>-ln(ε)</code> in NumPy returns relatively small values like this:</p>
  <pre><code>print(-np.log(np.finfo(np.float32).eps))
  print(-np.log(np.finfo(np.float64).eps))
  </code></pre>
  <p>Output:</p>
  <pre><code>15.942385
  36.04365338911715
  </code></pre>
  <p>Compared to <code>-log2(ε)</code> which has a greater range compared to base <code>e</code>:</p>
  <pre><code>print(-np.log2(np.finfo(np.float16).eps))
  print(-np.log2(np.finfo(np.float32).eps))
  print(-np.log2(np.finfo(np.float64).eps))
  </code></pre>
  <p>Output:</p>
  <pre><code>10.0
  23.0
  52.0
  </code></pre>
  <p>So, why don't softmax/sigmoid functions use <code>2</code> instead of <code>e</code> as the base for the exponential function? Using base <code>2</code> for the logarithm when converting back to logits would have more precision compared to base <code>e</code> in IEEE 754 representation.</p>
  
Tags: <python><softmax><sigmoid>
Title: Why softmax/sigmoid use base e instead of 2?
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # Why softmax/sigmoid use base e instead of 2?
  
  (Asked by: username_0 on 2024-12-18)
  
  Performing -ln(ε) in NumPy returns relatively small values like this:
  print(-np.log(np.finfo(np.float32).eps))
  print(-np.log(np.finfo(np.float64).eps))
  
  Output:
  15.942385
  36.04365338911715
  
  Compared to -log2(ε) which has a greater range compared to base e:
  print(-np.log2(np.finfo(np.float16).eps))
  print(-np.log2(np.finfo(np.float32).eps))
  print(-np.log2(np.finfo(np.float64).eps))
  
  Output:
  10.0
  23.0
  52.0
  
  So, why don't softmax/sigmoid functions use 2 instead of e as the base for the exponential function? Using base 2 for the logarithm when converting back to logits would have more precision compared to base e in IEEE 754 representation.
  
  **Tags:** python, softmax, sigmoid
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2024-12-19. Score: 2 | Accepted answer
  
  IEEE 754 provides sufficient precision for most ML tasks even with base $e$, and the choice in softmax and sigmoid functions is rooted in mathematical and practical considerations.
  
  Notice the function $e^x$ has a unique property that its derivative is itself, therefore this simplifies gradient calculations during backpropagation of the common CE loss. Changing the base $2$ doesn't gain much precision but complicates existing math formulations in machine learning field and optimization computations potentially across many places. Therefore both softmax and sigmoid functions often used in probability and information theory typically use natural logarithm and exponential function based on $e$ in practice, such as the CE loss implemented in PyTorch. Often in information-theoretic discussions the base $2$ for the unit of bits is preferred such as Shanon's original paper about entropy.
  
  
  ---
  
  ### Answer by username_2 on 2024-12-18. Score: 2
  
  The base of the logarithm change the range, but it not necessarily affect the precision.
  
  There exists the following formula:
  
  $\log\_b(x) = \log(x) \cdot \frac{1}{\log(b)}$
  
  In which b is the base.
  
  Then, the selection of the base affect the range. Larger bases generate smaller ranges.
  
  Also, computing gradients with log with base e is more easy that when using other bases. Gradients are needed for backpropagation.
  
  
  ---
  
  ### Answer by username_3 on 2024-12-20. Score: 1
  
  $\log\_2$ results in larger numbers than $\ln$, but that doesn't make them more precise. Floating point arithmetic has the same relative precision for numbers of any size.
  
  
  ---
  
  