Answers.Body: <ul>
  <li><p>No you can't prune dead ReLUs without accuracy loss because even if you find that some ReLUs are dead for a small batch of inputs, you don't know if it's dead for other inputs. If you wanted to prune like this you'd need to do more finetuning.  Pruning dead ReLUs will not make your network shallower obviously.</p>
  </li>
  <li><p>I'm pretty sure they're not mathematically equivalent.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T01:06:39.617Z
Answers.LastActivityDate: 2024-03-02T17:28:53.107Z
Answers.LastEditDate: 2024-03-02T17:28:53.107Z
Answers.UserDisplayName: JobHunter69
Body: <p>From <em>Deep Learning</em> (Courville, Goodfellow, Bengio), a ReLU activation often &quot;dies&quot; because</p>
  <blockquote>
  <p>One drawback to rectified linear units is that they cannot learn via gradient based methods on examples for which their activation is zero.</p>
  </blockquote>
  <p>Similarly, L1 regularization (as opposed to L2) results in a sparse network</p>
  <blockquote>
  <p>This demonstrates that L2 regularization does not cause the parameters
  to become sparse, while L1 regularization may do so for large enough α. The sparsity property induced by L1 regularization has been used extensively
  as a feature selection mechanism.</p>
  </blockquote>
  <p>A couple questions about these topics:</p>
  <ol>
  <li>In practice, is there any way/use to prune these &quot;dead&quot; ReLU-activated neurons? And if our trained network performs well with lots of dead neurons, would that imply that a shallower network is a sufficient representation?</li>
  <li>Since ReLU activations also result in a sparse network, does it have the same &quot;feature selection&quot; property as L1 regularization? If it does, does this then imply that sigmoid/tanh activations don't have this property?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-21T12:01:50.727Z
Comments.Text: Please, select one question and remove the others, which you can ask in a separate post!
Comments.UserDisplayName: nbro
Tags: <machine-learning><relu><l1-regularization>
Title: Since ReLU activations also result in a sparse network, does it have the same "feature selection" property as L1 regularization?
UserDisplayName: rac.coon
ThreadText: # Since ReLU activations also result in a sparse network, does it have the same "feature selection" property as L1 regularization?
  
  (Asked by: username_0 on 2023-01-21)
  
  From *Deep Learning* (Courville, Goodfellow, Bengio), a ReLU activation often "dies" because
  
  > One drawback to rectified linear units is that they cannot learn via gradient based methods on examples for which their activation is zero.
  
  Similarly, L1 regularization (as opposed to L2) results in a sparse network
  
  > This demonstrates that L2 regularization does not cause the parameters
  > to become sparse, while L1 regularization may do so for large enough α. The sparsity property induced by L1 regularization has been used extensively
  > as a feature selection mechanism.
  
  A couple questions about these topics:
  
  1. In practice, is there any way/use to prune these "dead" ReLU-activated neurons? And if our trained network performs well with lots of dead neurons, would that imply that a shallower network is a sufficient representation?
  2. Since ReLU activations also result in a sparse network, does it have the same "feature selection" property as L1 regularization? If it does, does this then imply that sigmoid/tanh activations don't have this property?
  
  **Tags:** machine-learning, relu, l1-regularization
  
  > **Comments:**
  > **username_1**: Please, select one question and remove the others, which you can ask in a separate post!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-02-23. Score: 0
  
  * No you can't prune dead ReLUs without accuracy loss because even if you find that some ReLUs are dead for a small batch of inputs, you don't know if it's dead for other inputs. If you wanted to prune like this you'd need to do more finetuning. Pruning dead ReLUs will not make your network shallower obviously.
  * I'm pretty sure they're not mathematically equivalent.
  
  
  ---
  
  