<p>It seems <a href="https://medium.com/@danqing/a-practical-guide-to-relu-b83ca804f1f7#:~:text=Concatenated%20ReLU%20has%20two%20outputs,CReLU%20doubles%20the%20output%20dimension" rel="nofollow noreferrer">I have found it</a>. It is called <strong>concatenated ReLU</strong> (CReLU).</p>

<blockquote>
  <p>Concatenated ReLU has two outputs, one ReLU and one negative ReLU, concatenated together. In other words, for positive x it produces [x, 0], and for negative x it produces [0, x]. Because it has two outputs, CReLU doubles the output dimension.</p>
</blockquote>

<ul>
<li><a href="https://arxiv.org/abs/1603.05201" rel="nofollow noreferrer">Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units</a> (the original paper that introduced CReLU).</li>
<li><a href="https://www.tensorflow.org/api_docs/python/tf/nn/crelu" rel="nofollow noreferrer">A TensorFlow implementation</a></li>
</ul>

<p>There is also Negative CReLU. It seems that the difference is only the sign.</p>

<p><span class="math-container">$$\text{NCReLU}(x) = (\rho(x) , −\rho(−x) )$$</span></p>

CC BY-SA 4.0
2019-01-07T13:13:09.497Z
2020-06-04T15:11:43.230Z
2020-06-04T15:11:43.230Z
Andrew Matiuk
<p>Is there a ReLU-like activation function that concatenates positive and negative values? What is its name? Apparently, it doubles the output dimension.</p>

<convolutional-neural-networks><relu>
Is there a ReLU-like activation function that concatenates positive and negative values?
Andrew Matiuk
# Is there a ReLU-like activation function that concatenates positive and negative values?

(Asked by: username_0 on 2019-01-06)

Is there a ReLU-like activation function that concatenates positive and negative values? What is its name? Apparently, it doubles the output dimension.

**Tags:** convolutional-neural-networks, relu


---

## 1 answers

### Answer by username_0 on 2019-01-07. Score: 0 | Accepted answer

It seems [I have found it](https://medium.com/@danqing/a-practical-guide-to-relu-b83ca804f1f7#:~:text=Concatenated%20ReLU%20has%20two%20outputs,CReLU%20doubles%20the%20output%20dimension). It is called **concatenated ReLU** (CReLU).

> Concatenated ReLU has two outputs, one ReLU and one negative ReLU, concatenated together. In other words, for positive x it produces [x, 0], and for negative x it produces [0, x]. Because it has two outputs, CReLU doubles the output dimension.

* [Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units](https://arxiv.org/abs/1603.05201) (the original paper that introduced CReLU).
* [A TensorFlow implementation](https://www.tensorflow.org/api_docs/python/tf/nn/crelu)

There is also Negative CReLU. It seems that the difference is only the sign.

$$\text{NCReLU}(x) = (\rho(x) , −\rho(−x) )$$


---

