Answers.Body: <p><em>Probably not worth the hassle.</em></p>
  
  <p>I guess that simply nobody cares. I'm all for having my function be differentiable at all points, but I don't think that having a single jump in the derivative matters much. The neuron basically either repeats its input (x>0) or outputs nearly nothing (x&lt;0) and the point in between is not that important.</p>
  
  <p>Replacing</p>
  
  <pre><code>ELU(x) = x &gt; 0 ? x : α * (exp(x) - 1)
  </code></pre>
  
  <p>by your proposal</p>
  
  <pre><code>ELU(x) = x &gt; 0 ? α * x : α * (exp(x) - 1)
  </code></pre>
  
  <p>would have far more consequences. For example, after <code>n</code> layers, the output would get multiplied by <code>α**n</code>, assuming all <code>x</code> on the path being positive. This could be countered by dividing each weight by <code>α</code>, but the derivatives would change. You surely could adjust everything (initial rates, learn rate and whatever), so it'd work as usual, but it could laborious, especially when testing multiple learning algorithms. </p>
  
  <p>I'd suggest to use</p>
  
  <pre><code>ELU(x) = x &gt; 0 ? x : α * (exp(x/α) - 1)
  </code></pre>
  
  <p>instead, as it just changes the less important part of the curve.</p>
  
  <blockquote>
    <p>Or did they not bother, because α=1 is definitely the hyperparameter to use?</p>
  </blockquote>
  
  <p>I can imagine that everyone tried α=1 and was happy with it. It prevented the gradient from being zero and that's it.</p>
  
  <p>I don't have any support of my claim (having never read about something doesn't mean that it doesn't get written about).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-17T18:02:50.993Z
Answers.LastActivityDate: 2018-04-17T18:02:50.993Z
Answers.UserDisplayName: maaartinus
Body: <p>An exponential linear unit (as proposed by <a href="https://arxiv.org/pdf/1511.07289.pdf" rel="nofollow noreferrer">Clevert et al.</a>) uses the function:</p>
  <p><span class="math-container">\begin{align}
  \text{ELU}_\alpha(x) 
  =
  \begin{cases}
  \alpha(e^x - 1), &amp;\text{if } x &lt; 0\\
  x, \text{if} &amp;\text{if } x \geq 0\\
  \end{cases}
  \end{align}</span></p>
  <p>Here's a picture.</p>
  <p><a href="https://i.sstatic.net/Ko4R5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ko4R5.png" alt="ELU with α=5" /></a></p>
  <p>Now, this is continuous at <span class="math-container">$x=0$</span>, which is great. It's differentiable there too if <span class="math-container">$\alpha=1$</span>, which is the value that the paper used to test ELU units.</p>
  <p>But if <span class="math-container">$\alpha \neq $</span> (as in the above diagram), then it's no longer differentiable at <span class="math-container">$x=0$</span>. It has a crook in it, which seems weird to me. Having your function be differentiable at all points seems advantageous. Further, it seems that, if you just make the linear portion evaluate to <span class="math-container">$\alpha x$</span> rather than <span class="math-container">$x$</span>, it <em>would</em> be differentiable there.</p>
  <p>Is there a reason that the function wasn't defined to do this? Or did they not bother, because <span class="math-container">$\alpha = 1$</span> is definitely the hyperparameter to use?</p>
  
Tags: <neural-networks><deep-learning><definitions><activation-functions><elu>
Title: Why don't ELUs multiply the linear portion by $\alpha$?
UserDisplayName: Adam Smith
ThreadText: # Why don't ELUs multiply the linear portion by $\alpha$?
  
  (Asked by: username_0 on 2018-03-28)
  
  An exponential linear unit (as proposed by [Clevert et al.](https://arxiv.org/pdf/1511.07289.pdf)) uses the function:
  
  \begin{align}
  \text{ELU}\_\alpha(x)
  =
  \begin{cases}
  \alpha(e^x - 1), &\text{if } x < 0\\
  x, \text{if} &\text{if } x \geq 0\\
  \end{cases}
  \end{align}
  
  Here's a picture.
  
  [![ELU with α=5](https://i.sstatic.net/Ko4R5.png)](https://i.sstatic.net/Ko4R5.png)
  
  Now, this is continuous at $x=0$, which is great. It's differentiable there too if $\alpha=1$, which is the value that the paper used to test ELU units.
  
  But if $\alpha \neq $ (as in the above diagram), then it's no longer differentiable at $x=0$. It has a crook in it, which seems weird to me. Having your function be differentiable at all points seems advantageous. Further, it seems that, if you just make the linear portion evaluate to $\alpha x$ rather than $x$, it *would* be differentiable there.
  
  Is there a reason that the function wasn't defined to do this? Or did they not bother, because $\alpha = 1$ is definitely the hyperparameter to use?
  
  **Tags:** neural-networks, deep-learning, definitions, activation-functions, elu
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-04-17. Score: 1 | Accepted answer
  
  *Probably not worth the hassle.*
  
  I guess that simply nobody cares. I'm all for having my function be differentiable at all points, but I don't think that having a single jump in the derivative matters much. The neuron basically either repeats its input (x>0) or outputs nearly nothing (x<0) and the point in between is not that important.
  
  Replacing
  
  ```
  ELU(x) = x > 0 ? x : α * (exp(x) - 1)
  
  ```
  
  by your proposal
  
  ```
  ELU(x) = x > 0 ? α * x : α * (exp(x) - 1)
  
  ```
  
  would have far more consequences. For example, after `n` layers, the output would get multiplied by `α**n`, assuming all `x` on the path being positive. This could be countered by dividing each weight by `α`, but the derivatives would change. You surely could adjust everything (initial rates, learn rate and whatever), so it'd work as usual, but it could laborious, especially when testing multiple learning algorithms.
  
  I'd suggest to use
  
  ```
  ELU(x) = x > 0 ? x : α * (exp(x/α) - 1)
  
  ```
  
  instead, as it just changes the less important part of the curve.
  
  > Or did they not bother, because α=1 is definitely the hyperparameter to use?
  
  I can imagine that everyone tried α=1 and was happy with it. It prevented the gradient from being zero and that's it.
  
  I don't have any support of my claim (having never read about something doesn't mean that it doesn't get written about).
  
  
  ---
  
  