Body: <p>I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically <code>if z &gt; 0 tanh(z) else 0</code>. I have implemented it and I compared with ReLu on odd MNIST. They both achieved about 94% success rate in 10 epochs. My logic was that usually humans tend to stop feeling more confident once they have learnt something. Similarly, I thought a Convolutional layer neuron should not feel much more confident (higher activation) with growing evidence (higher weighted input). So is there any evidence of perhaps such a rectified tanh being more successful ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-16T05:34:12.753Z
Comments.Text: I don't have any evidence, but I would say probably not. Ultimately you're trying to fit a function, and a ReLU does it so fast because of how easy it is to combine ReLU functions and also how simple it is to compute (in the forward and backward pass). I don't think you'll have much luck getting much more accuracy using a rectified tanh, but if you compare convergence time or computation time I think you'll find that a simple ReLU outperforms by quite a margin.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-17T03:12:48.867Z
Comments.Text: Yes I agree. I should mention that mathematically, rectified tanh would make it harder for the neuron to change its parameters as they grow larger (if they start out positive). This appears a useful characteristics as long you have small parameter initialization. I think it would allow you to get the extra percent or two in accuracy. For computation, perhaps a rectified arctan would perform better since its derivative is simpler than tanh.
Comments.UserDisplayName: Physics
Tags: <convolutional-neural-networks><activation-functions>
Title: Using a rectified Tanh to train a CNN?
UserDisplayName: Physics
ThreadText: # Using a rectified Tanh to train a CNN?
  
  (Asked by: username_0 on 2021-07-16)
  
  I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically `if z > 0 tanh(z) else 0`. I have implemented it and I compared with ReLu on odd MNIST. They both achieved about 94% success rate in 10 epochs. My logic was that usually humans tend to stop feeling more confident once they have learnt something. Similarly, I thought a Convolutional layer neuron should not feel much more confident (higher activation) with growing evidence (higher weighted input). So is there any evidence of perhaps such a rectified tanh being more successful ?
  
  **Tags:** convolutional-neural-networks, activation-functions
  > **Comments:**
  > **username_1**: I don't have any evidence, but I would say probably not. Ultimately you're trying to fit a function, and a ReLU does it so fast because of how easy it is to combine ReLU functions and also how simple it is to compute (in the forward and backward pass). I don't think you'll have much luck getting much more accuracy using a rectified tanh, but if you compare convergence time or computation time I think you'll find that a simple ReLU outperforms by quite a margin.  
  > **username_0**: Yes I agree. I should mention that mathematically, rectified tanh would make it harder for the neuron to change its parameters as they grow larger (if they start out positive). This appears a useful characteristics as long you have small parameter initialization. I think it would allow you to get the extra percent or two in accuracy. For computation, perhaps a rectified arctan would perform better since its derivative is simpler than tanh.  
  
  ---
  
  ## 0 answers
  
  