Answers.Body: <p>Have a look at these graphics showing popular linear units (image taken from <a href="https://arxiv.org/pdf/1511.07289v5.pdf" rel="noreferrer">Clevert et al. 2016</a>):
  <a href="https://i.sstatic.net/gQVus.png" rel="noreferrer"><img src="https://i.sstatic.net/gQVus.png" alt="enter image description here" /></a></p>
  <p>You can see that these functions are linear functions for <span class="math-container">$x &gt; 0$</span>, that's why they are called <em>Linear Units</em>.</p>
  <p>For example, the ELU is defined as</p>
  <p><span class="math-container">$$ ELU(x) = \begin{cases} 
  x &amp;\text{if } x &gt; 0\\
  \alpha (\exp(x)-1) &amp; \text{if } x \leq 0.
  \end{cases} $$</span>
  These functions introduce the nonlinearity around zero, each in its own way, which can be used for different problems.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-29T13:27:07.627Z
Answers.LastActivityDate: 2021-07-29T13:27:07.627Z
Answers.UserDisplayName: e.Fro
Body: <p>Activation functions, in neural networks, are used to introduce non-linearity. Many activation functions that are used in neural networks have the term &quot;Linear Unit&quot; in their full form. &quot;Linear unit&quot; can be abbreviated as <strong>LU</strong>.</p>
  <p>For example, consider some activation functions</p>
  <blockquote>
  <p>ELU  - Exponential Linear Unit</p>
  <p>ReLU - Rectified Linear Unit</p>
  <p>................................................</p>
  </blockquote>
  <p>Why does the function name contain the term &quot;Linear Unit&quot;? What is meant by <strong>Linear Unit</strong> here? Is it saying anything about the nature of function under consideration?</p>
  
Tags: <neural-networks><deep-learning><terminology><activation-functions>
Title: What does "linear unit" mean in the names of activation functions?
UserDisplayName: hanugm
ThreadText: # What does "linear unit" mean in the names of activation functions?
  
  (Asked by: username_0 on 2021-07-29)
  
  Activation functions, in neural networks, are used to introduce non-linearity. Many activation functions that are used in neural networks have the term "Linear Unit" in their full form. "Linear unit" can be abbreviated as **LU**.
  
  For example, consider some activation functions
  
  > ELU - Exponential Linear Unit
  >
  > ReLU - Rectified Linear Unit
  >
  > ................................................
  
  Why does the function name contain the term "Linear Unit"? What is meant by **Linear Unit** here? Is it saying anything about the nature of function under consideration?
  
  **Tags:** neural-networks, deep-learning, terminology, activation-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-29. Score: 6 | Accepted answer
  
  Have a look at these graphics showing popular linear units (image taken from [Clevert et al. 2016](https://arxiv.org/pdf/1511.07289v5.pdf)):
  [![enter image description here](https://i.sstatic.net/gQVus.png)](https://i.sstatic.net/gQVus.png)
  
  You can see that these functions are linear functions for $x > 0$, that's why they are called *Linear Units*.
  
  For example, the ELU is defined as
  
  $$ ELU(x) = \begin{cases}
  x &\text{if } x > 0\\
  \alpha (\exp(x)-1) & \text{if } x \leq 0.
  \end{cases} $$
  These functions introduce the nonlinearity around zero, each in its own way, which can be used for different problems.
  
  
  ---
  
  