Answers.Body: <p>Lets assume we have 3 Dense layers, where the activations are <span class="math-container">$x^0 \rightarrow x^1 \rightarrow x^2$</span>, such that <span class="math-container">$x^2 = \psi PReLU(x^1) + \gamma$</span> and <span class="math-container">$x^1 = PReLU(Ax^0 + b)$</span>   </p>
  
  <p>Now lets see what it would take to conform the PReLU into a ReLU    </p>
  
  <p><span class="math-container">$\begin{align*} 
  PReLU(x^1) &amp;= ReLU(x^1) + ReLU(p \odot x^1)\\
  &amp;= ReLU(Ax^0+b) + ReLU(p\odot(Ax^0+b))\\
  &amp;= ReLU(Ax^0+b) + ReLU((eye(p)A + eye(p)b)x^0)\\ 
  &amp;= ReLU(Ax^0+b) + ReLU(Qx^0+c) \quad s.t. \quad Q = eye(p)A, \ \ c = eye(p)b\\
  &amp;= [I, I]^T[ReLU(Ax^0+b), ReLU(Qx^0+c)]\\
  \implies x^2 &amp;= [\psi, \psi][ReLU(Ax^0+b), ReLU(Qx^0+c)]\\
  &amp;= V*ReLU(Sx^0 + d) \quad V=[\psi, \psi], \ \ S=[A, Q] \ \ d=[b, c]
  \end{align*}$</span>  </p>
  
  <p>So as you said it is possible to break the form of the intermiediary <span class="math-container">$PReLU$</span> into a pure <span class="math-container">$ReLU$</span> while keeping it as a linear model, but if you take a second look at the parameters of the model, the size increase drastically. The hidden units of S <strong>doubled</strong> meaning to keep <span class="math-container">$x^2$</span> the same size <span class="math-container">$V$</span> also doubles in size. So this means if you dont want to use the <span class="math-container">$PReLU$</span> you are learning double the parameters to achieve the same capability (granted it allows you to learn a wider span of functions as well), and if you enforce the constraints on <span class="math-container">$V,S$</span> set by the <span class="math-container">$PReLU$</span> the number of paramaters is the same but you are still using more memory and more operations!  </p>
  
  <p>I hope this example convinces you of the difference</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-23T15:40:44.203Z
Answers.Text: ok thanks, this sounds like a matter of efficiency and probably better learning/convergence capabilities. what does eye(p) stand for?
Answers.UserDisplayName: Robert Nowak
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-23T15:48:05.777Z
Answers.Text: eye(p), is taking the vector p and making a diagonal matrix, where the elements of p is the diagonal (same functionality as like numpys np.eye).
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-23T15:15:18.240Z
Answers.LastActivityDate: 2019-07-23T15:15:18.240Z
Answers.UserDisplayName: mshlis
Body: <p>Why do people use the <span class="math-container">$PReLU$</span> activation?</p>
  
  <p><span class="math-container">$PReLU[x] = ReLU[x] + ReLU[p*x]$</span></p>
  
  <p>with the parameter <span class="math-container">$p$</span> typically being a small negative number.</p>
  
  <p>If a fully connected layer is followed by a at least two element <span class="math-container">$ReLU$</span> layer then the combined layers together are capable of emulating exactly the <span class="math-container">$PReLU$</span>, so why is it necessary?</p>
  
  <p>Am I missing something?</p>
  
Tags: <neural-networks><machine-learning><activation-functions><relu>
Title: Is PReLU superfluous with respect to ReLU?
UserDisplayName: Robert Nowak
ThreadText: # Is PReLU superfluous with respect to ReLU?
  
  (Asked by: username_0 on 2019-07-23)
  
  Why do people use the $PReLU$ activation?
  
  $PReLU[x] = ReLU[x] + ReLU[p\*x]$
  
  with the parameter $p$ typically being a small negative number.
  
  If a fully connected layer is followed by a at least two element $ReLU$ layer then the combined layers together are capable of emulating exactly the $PReLU$, so why is it necessary?
  
  Am I missing something?
  
  **Tags:** neural-networks, machine-learning, activation-functions, relu
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-23. Score: 1 | Accepted answer
  
  Lets assume we have 3 Dense layers, where the activations are $x^0 \rightarrow x^1 \rightarrow x^2$, such that $x^2 = \psi PReLU(x^1) + \gamma$ and $x^1 = PReLU(Ax^0 + b)$
  
  Now lets see what it would take to conform the PReLU into a ReLU
  
  $\begin{align\*}
  PReLU(x^1) &= ReLU(x^1) + ReLU(p \odot x^1)\\
  &= ReLU(Ax^0+b) + ReLU(p\odot(Ax^0+b))\\
  &= ReLU(Ax^0+b) + ReLU((eye(p)A + eye(p)b)x^0)\\
  &= ReLU(Ax^0+b) + ReLU(Qx^0+c) \quad s.t. \quad Q = eye(p)A, \ \ c = eye(p)b\\
  &= [I, I]^T[ReLU(Ax^0+b), ReLU(Qx^0+c)]\\
  \implies x^2 &= [\psi, \psi][ReLU(Ax^0+b), ReLU(Qx^0+c)]\\
  &= V\*ReLU(Sx^0 + d) \quad V=[\psi, \psi], \ \ S=[A, Q] \ \ d=[b, c]
  \end{align\*}$
  
  So as you said it is possible to break the form of the intermiediary $PReLU$ into a pure $ReLU$ while keeping it as a linear model, but if you take a second look at the parameters of the model, the size increase drastically. The hidden units of S **doubled** meaning to keep $x^2$ the same size $V$ also doubles in size. So this means if you dont want to use the $PReLU$ you are learning double the parameters to achieve the same capability (granted it allows you to learn a wider span of functions as well), and if you enforce the constraints on $V,S$ set by the $PReLU$ the number of paramaters is the same but you are still using more memory and more operations!
  
  I hope this example convinces you of the difference
  
  > **Comments:**
  > **username_0**: ok thanks, this sounds like a matter of efficiency and probably better learning/convergence capabilities. what does eye(p) stand for?  
  > **username_1**: eye(p), is taking the vector p and making a diagonal matrix, where the elements of p is the diagonal (same functionality as like numpys np.eye).  
  
  ---
  
  