Answers.Body: <p>Using tanh in hidden layers require careful initialization of network weights and works best with the input features normalized within the same range as output (i.e. -1 to 1). It have expensive functions to evaluate and suffers with vanishing gradient. But it converges faster because due to the zero centered output.</p>
  <p>But tanh works good with RNNs, as RNNs suffers from vanishing gradients, and they need an activation function whose second derivative can sustain for long range before going to zero.</p>
  <p>If you want any of these properties, use tanh.</p>
  <p>Without any gist of architecture, it's hard to tell why your model train faster than ReLU. But generally speaking, ReLU trains faster than Tanh due to lack of expensive functions to evaluate (such as exponents)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-13T14:17:56.013Z
Answers.Text: "trains faster" is a bit ambiguous in this respect. It could mean "in less steps" or "less wallclock time". Each ReLu step is slightly faster, but you probably need more steps.
Answers.UserDisplayName: MSalters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-14T11:17:15.597Z
Answers.Text: Tanh isn't necessarily slow, the exponential function can be approximated very efficiently using this trick https://www.schraudolph.org/pubs/Schraudolph99.pdf .  Also the cost of the activation function is usually fairly small compared to the cost of the matrix operations.  I would standardise the input features rather than to -1/+1, especially if there is any chance of outlier values.
Answers.UserDisplayName: Dikran Marsupial
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-13T06:55:05.980Z
Answers.LastActivityDate: 2024-05-13T06:55:05.980Z
Answers.UserDisplayName: Leo
Body: <p>When and why would you not use Tanh?</p>
  <p>I just replaced ReLU with Tanh and my model trains about 2x faster, reaching 90% acc within 500 steps.</p>
  <p>While using ReLU it reached 90% acc in &gt;1000 training steps.</p>
  <p>I believe the reason it trained faster was due to a steeper gradient (correct me if I'm wrong).</p>
  <p>This led me to wonder, why would we not use Tanh over ReLU for most contexts?
  Is it the computational complexity? Or something else?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-13T20:54:53.857Z
Comments.Text: What size and type of model is it? How long did training take (in terms of real time, i.e. hours/minutes)?
Comments.UserDisplayName: The Guy with The Hat
Tags: <neural-networks><machine-learning><deep-learning><activation-functions><tanh>
Title: When to use Tanh?
UserDisplayName: vxnuaj
ThreadText: # When to use Tanh?
  
  (Asked by: username_0 on 2024-05-12)
  
  When and why would you not use Tanh?
  
  I just replaced ReLU with Tanh and my model trains about 2x faster, reaching 90% acc within 500 steps.
  
  While using ReLU it reached 90% acc in >1000 training steps.
  
  I believe the reason it trained faster was due to a steeper gradient (correct me if I'm wrong).
  
  This led me to wonder, why would we not use Tanh over ReLU for most contexts?
  Is it the computational complexity? Or something else?
  
  **Tags:** neural-networks, machine-learning, deep-learning, activation-functions, tanh
  
  > **Comments:**
  > **username_1**: What size and type of model is it? How long did training take (in terms of real time, i.e. hours/minutes)?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-05-13. Score: 10 | Accepted answer
  
  Using tanh in hidden layers require careful initialization of network weights and works best with the input features normalized within the same range as output (i.e. -1 to 1). It have expensive functions to evaluate and suffers with vanishing gradient. But it converges faster because due to the zero centered output.
  
  But tanh works good with RNNs, as RNNs suffers from vanishing gradients, and they need an activation function whose second derivative can sustain for long range before going to zero.
  
  If you want any of these properties, use tanh.
  
  Without any gist of architecture, it's hard to tell why your model train faster than ReLU. But generally speaking, ReLU trains faster than Tanh due to lack of expensive functions to evaluate (such as exponents)
  
  > **Comments:**
  > **username_3**: "trains faster" is a bit ambiguous in this respect. It could mean "in less steps" or "less wallclock time". Each ReLu step is slightly faster, but you probably need more steps.  
  > **username_4**: Tanh isn't necessarily slow, the exponential function can be approximated very efficiently using this trick https://www.schraudolph.org/pubs/Schraudolph99.pdf .  Also the cost of the activation function is usually fairly small compared to the cost of the matrix operations.  I would standardise the input features rather than to -1/+1, especially if there is any chance of outlier values.  
  
  ---
  
  