<p>I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically <code>if z &gt; 0 tanh(z) else 0</code>. I have implemented it and I compared with ReLu on odd MNIST. They both achieved about 94% success rate in 10 epochs. My logic was that usually humans tend to stop feeling more confident once they have learnt something. Similarly, I thought a Convolutional layer neuron should not feel much more confident (higher activation) with growing evidence (higher weighted input). So is there any evidence of perhaps such a rectified tanh being more successful ?</p>

CC BY-SA 4.0
2021-07-16T05:34:12.753Z
I don't have any evidence, but I would say probably not. Ultimately you're trying to fit a function, and a ReLU does it so fast because of how easy it is to combine ReLU functions and also how simple it is to compute (in the forward and backward pass). I don't think you'll have much luck getting much more accuracy using a rectified tanh, but if you compare convergence time or computation time I think you'll find that a simple ReLU outperforms by quite a margin.
Recessive
CC BY-SA 4.0
2021-07-17T03:12:48.867Z
Yes I agree. I should mention that mathematically, rectified tanh would make it harder for the neuron to change its parameters as they grow larger (if they start out positive). This appears a useful characteristics as long you have small parameter initialization. I think it would allow you to get the extra percent or two in accuracy. For computation, perhaps a rectified arctan would perform better since its derivative is simpler than tanh.
Physics
<convolutional-neural-networks><activation-functions>
Using a rectified Tanh to train a CNN?
Physics
# Using a rectified Tanh to train a CNN?

(Asked by: username_0 on 2021-07-16)

I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically `if z > 0 tanh(z) else 0`. I have implemented it and I compared with ReLu on odd MNIST. They both achieved about 94% success rate in 10 epochs. My logic was that usually humans tend to stop feeling more confident once they have learnt something. Similarly, I thought a Convolutional layer neuron should not feel much more confident (higher activation) with growing evidence (higher weighted input). So is there any evidence of perhaps such a rectified tanh being more successful ?

**Tags:** convolutional-neural-networks, activation-functions

> **Comments:**
> **username_1**: I don't have any evidence, but I would say probably not. Ultimately you're trying to fit a function, and a ReLU does it so fast because of how easy it is to combine ReLU functions and also how simple it is to compute (in the forward and backward pass). I don't think you'll have much luck getting much more accuracy using a rectified tanh, but if you compare convergence time or computation time I think you'll find that a simple ReLU outperforms by quite a margin.  
> **username_0**: Yes I agree. I should mention that mathematically, rectified tanh would make it harder for the neuron to change its parameters as they grow larger (if they start out positive). This appears a useful characteristics as long you have small parameter initialization. I think it would allow you to get the extra percent or two in accuracy. For computation, perhaps a rectified arctan would perform better since its derivative is simpler than tanh.  

---

## 0 answers

