Answers.Body: <p>The ReLU is the simplest nonlinear function that has shown remarkable performance when used as activation function in NNs. Note that the derivative is binary, either zero or one, just depending on the sign of the input. This makes ReLU very fast and convenient to use.</p>
  <p>Since its first usage, loads of other ReLU like functions have been proposed, for your specific case, pytorch has the <a href="https://pytorch.org/docs/stable/generated/torch.nn.PReLU.html#torch.nn.PReLU" rel="nofollow noreferrer">PReLU</a> function already implemented, which has the learnable parameter <span class="math-container">$a$</span> like in your function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-03T19:22:51.637Z
Answers.LastActivityDate: 2023-05-03T19:22:51.637Z
Answers.UserDisplayName: N. Kiefer
Body: <p>Why can we not parametrize and learn the non-linear activations? For example, if we look at leaky ReLu which equals to <span class="math-container">$f(y)=y$</span> for <span class="math-container">$y&gt;0$</span> and <span class="math-container">$f(y)=\alpha y$</span> for <span class="math-container">$y&lt;0$</span>, it seems that we can differentiate the parameter <span class="math-container">$\alpha$</span> with respect to the loss and learn it, why is it not done?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-03T23:39:50.257Z
Comments.Text: Saying "differentiate the parameters wrt the loss" makes no sense to me. What makes sense is differentiate a function (e.g. the loss function) with respect to some parameter.
Comments.UserDisplayName: nbro
Tags: <deep-learning><optimization><neural-architecture-search>
Title: Why are the non-linear activations in deep nets not learned?
UserDisplayName: Gilad Deutsch
ThreadText: # Why are the non-linear activations in deep nets not learned?
  
  (Asked by: username_0 on 2023-05-03)
  
  Why can we not parametrize and learn the non-linear activations? For example, if we look at leaky ReLu which equals to $f(y)=y$ for $y>0$ and $f(y)=\alpha y$ for $y<0$, it seems that we can differentiate the parameter $\alpha$ with respect to the loss and learn it, why is it not done?
  
  **Tags:** deep-learning, optimization, neural-architecture-search
  > **Comments:**
  > **username_1**: Saying "differentiate the parameters wrt the loss" makes no sense to me. What makes sense is differentiate a function (e.g. the loss function) with respect to some parameter.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-03. Score: 3 | Accepted answer
  
  The ReLU is the simplest nonlinear function that has shown remarkable performance when used as activation function in NNs. Note that the derivative is binary, either zero or one, just depending on the sign of the input. This makes ReLU very fast and convenient to use.
  
  Since its first usage, loads of other ReLU like functions have been proposed, for your specific case, pytorch has the [PReLU](https://pytorch.org/docs/stable/generated/torch.nn.PReLU.html#torch.nn.PReLU) function already implemented, which has the learnable parameter $a$ like in your function.
  
  
  ---
  
  