Answers.Body: <p>I suppose, the situation is as follows - <code>PReLU</code> increases the expressiveness of a model for a bit at a small cost, but the gain is almost negligible as well (<a href="https://medium.com/@shoray.goel/prelu-activation-e294bb21fefa" rel="nofollow noreferrer">according to this post</a>).</p>
  <p>There is, indeed, a noticeable difference between <code>ReLU</code> and <code>PReLU</code>, since the former takes the same value for all <span class="math-container">$\mathbb{R}_{\leq 0}$</span>.</p>
  <p>However, compared with a <code>LeakyReLU</code>, note that this activation is accompanied with a linear operation, like <code>Dense</code> or <code>Convolution</code> layer:
  <span class="math-container">$$
  y \sim f \left(\sum w \cdot x \right)
  $$</span>
  And the slope <span class="math-container">$\alpha$</span> can be absorbed in the weights of neural network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-09T07:33:00.310Z
Answers.Text: It's unclear what your point is in the last two paragraphs. Surely $ f(\alpha b) \neq \alpha f(b) $ if the activation function f is non-linear.
Answers.UserDisplayName: Tim Kuipers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-10T14:40:50.250Z
Answers.LastActivityDate: 2021-09-11T13:14:11.323Z
Answers.LastEditDate: 2021-09-11T13:14:11.323Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to adjust). Only if we want the network to output positive values it makes sense to use it in the output layer. Other than that, I don't see why a priori I would decide to choose ReLU over PReLU. However, most architectures I came across use ReLU activations. Why?  Am I missing something?</p>
  
Tags: <deep-learning><comparison><activation-functions><relu><prelu>
Title: Why should one ever use ReLU instead of PReLU?
UserDisplayName: algebruh
ThreadText: # Why should one ever use ReLU instead of PReLU?
  
  (Asked by: username_0 on 2021-09-10)
  
  To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to adjust). Only if we want the network to output positive values it makes sense to use it in the output layer. Other than that, I don't see why a priori I would decide to choose ReLU over PReLU. However, most architectures I came across use ReLU activations. Why? Am I missing something?
  
  **Tags:** deep-learning, comparison, activation-functions, relu, prelu
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-10. Score: 2
  
  I suppose, the situation is as follows - `PReLU` increases the expressiveness of a model for a bit at a small cost, but the gain is almost negligible as well ([according to this post](https://medium.com/@shoray.goel/prelu-activation-e294bb21fefa)).
  
  There is, indeed, a noticeable difference between `ReLU` and `PReLU`, since the former takes the same value for all $\mathbb{R}\_{\leq 0}$.
  
  However, compared with a `LeakyReLU`, note that this activation is accompanied with a linear operation, like `Dense` or `Convolution` layer:
  $$
  y \sim f \left(\sum w \cdot x \right)
  $$
  And the slope $\alpha$ can be absorbed in the weights of neural network.
  
  > **Comments:**
  > **username_2**: It's unclear what your point is in the last two paragraphs. Surely $ f(\alpha b) \neq \alpha f(b) $ if the activation function f is non-linear.  
  
  ---
  
  