Answers.Body: <p>I've read all the papers about PReLU, LeakyReLU (...) and all the claims how it improves this and that but the little dirty secret is: most of the time it doesn't matter at all and you can't go much wrong with ReLU - empirically proven. I've personally tried all of them in many different problems (from training small networks from scratch through changing activations in large pretrained models) My guess is that gradient doesn't die much in any of them and the rest is pretty much irrelevant.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-14T14:35:09.737Z
Answers.LastActivityDate: 2017-04-14T14:35:09.737Z
Answers.UserDisplayName: FunkyKowal
Body: <p>What rectifier is better in general case of Convolutional Neural Network and how about empirical rules to use each type?</p>
  
  <ul>
  <li>ReLU</li>
  <li>PReLU</li>
  <li>RReLU</li>
  <li>ELU</li>
  <li>Leacky ReLU</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-11-09T14:42:58.450Z
Comments.Text: FYI [Comprehensive list of activation functions in neural networks with pros/cons](http://stats.stackexchange.com/q/115258/12359)
Comments.UserDisplayName: Franck Dernoncourt
Tags: <neural-networks><convolutional-neural-networks><architecture>
Title: What linear rectifier is better?
UserDisplayName: UndeadDragon
ThreadText: # What linear rectifier is better?
  
  (Asked by: username_0 on 2016-11-09)
  
  What rectifier is better in general case of Convolutional Neural Network and how about empirical rules to use each type?
  
  * ReLU
  * PReLU
  * RReLU
  * ELU
  * Leacky ReLU
  
  **Tags:** neural-networks, convolutional-neural-networks, architecture
  
  > **Comments:**
  > **username_1**: FYI [Comprehensive list of activation functions in neural networks with pros/cons](http://stats.stackexchange.com/q/115258/12359)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-04-14. Score: 4 | Accepted answer
  
  I've read all the papers about PReLU, LeakyReLU (...) and all the claims how it improves this and that but the little dirty secret is: most of the time it doesn't matter at all and you can't go much wrong with ReLU - empirically proven. I've personally tried all of them in many different problems (from training small networks from scratch through changing activations in large pretrained models) My guess is that gradient doesn't die much in any of them and the rest is pretty much irrelevant.
  
  
  ---
  
  