Answers.Body: <p>It seems <a href="https://medium.com/@danqing/a-practical-guide-to-relu-b83ca804f1f7#:~:text=Concatenated%20ReLU%20has%20two%20outputs,CReLU%20doubles%20the%20output%20dimension" rel="nofollow noreferrer">I have found it</a>. It is called <strong>concatenated ReLU</strong> (CReLU).</p>
  
  <blockquote>
    <p>Concatenated ReLU has two outputs, one ReLU and one negative ReLU, concatenated together. In other words, for positive x it produces [x, 0], and for negative x it produces [0, x]. Because it has two outputs, CReLU doubles the output dimension.</p>
  </blockquote>
  
  <ul>
  <li><a href="https://arxiv.org/abs/1603.05201" rel="nofollow noreferrer">Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units</a> (the original paper that introduced CReLU).</li>
  <li><a href="https://www.tensorflow.org/api_docs/python/tf/nn/crelu" rel="nofollow noreferrer">A TensorFlow implementation</a></li>
  </ul>
  
  <p>There is also Negative CReLU. It seems that the difference is only the sign.</p>
  
  <p><span class="math-container">$$\text{NCReLU}(x) = (\rho(x) , −\rho(−x) )$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-07T13:13:09.497Z
Answers.LastActivityDate: 2020-06-04T15:11:43.230Z
Answers.LastEditDate: 2020-06-04T15:11:43.230Z
Answers.UserDisplayName: Andrew Matiuk
Body: <p>Is there a ReLU-like activation function that concatenates positive and negative values? What is its name? Apparently, it doubles the output dimension.</p>
  
Tags: <convolutional-neural-networks><relu>
Title: Is there a ReLU-like activation function that concatenates positive and negative values?
UserDisplayName: Andrew Matiuk
ThreadText: # Is there a ReLU-like activation function that concatenates positive and negative values?
  
  (Asked by: username_0 on 2019-01-06)
  
  Is there a ReLU-like activation function that concatenates positive and negative values? What is its name? Apparently, it doubles the output dimension.
  
  **Tags:** convolutional-neural-networks, relu
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-01-07. Score: 0 | Accepted answer
  
  It seems [I have found it](https://medium.com/@danqing/a-practical-guide-to-relu-b83ca804f1f7#:~:text=Concatenated%20ReLU%20has%20two%20outputs,CReLU%20doubles%20the%20output%20dimension). It is called **concatenated ReLU** (CReLU).
  
  > Concatenated ReLU has two outputs, one ReLU and one negative ReLU, concatenated together. In other words, for positive x it produces [x, 0], and for negative x it produces [0, x]. Because it has two outputs, CReLU doubles the output dimension.
  
  * [Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units](https://arxiv.org/abs/1603.05201) (the original paper that introduced CReLU).
  * [A TensorFlow implementation](https://www.tensorflow.org/api_docs/python/tf/nn/crelu)
  
  There is also Negative CReLU. It seems that the difference is only the sign.
  
  $$\text{NCReLU}(x) = (\rho(x) , −\rho(−x) )$$
  
  
  ---
  
  