Answers.Body: <p>I don't think it makes sense to decide activation functions based on desired properties of the output; you can easily insert a calibration step that maps the 'neural network score' to whatever units you actually want to use (dollars, probability, etc.).</p>
  
  <p>So I think preference between different activation functions mostly boils down to the <a href="https://en.wikipedia.org/wiki/Activation_function#Comparison_of_activation_functions" rel="nofollow">different properties</a> of those activation functions (like whether or not they're continuously differentiable). Because there's just a linear transformation between the two, I think that means there isn't a meaningful difference between them.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T19:28:00.640Z
Answers.LastActivityDate: 2016-08-02T19:28:00.640Z
Answers.UserDisplayName: Matthew Gray
Answers.Body: <p><strong>Sigmoid > Hyperbolic tangent:</strong></p>
  
  <p>As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-graves says, we can fix this with a simple mapping/calibration step). In other layers, this makes no sense.</p>
  
  <p><strong>Hyperbolic tangent > Sigmoid:</strong></p>
  
  <p>Hyperbolic tangent has a property called "approximates identity near the origin" which means <span class="math-container">$\tanh(0) = 0$</span>, <span class="math-container">$\tanh'(0) = 1$</span>, and <span class="math-container">$\tanh'(z)$</span> is continuous around <span class="math-container">$z=0$</span> (as opposed to  <span class="math-container">$\sigma(0)=0.5$</span> and <span class="math-container">$\sigma'(0)=0.25$</span>). This feature (which also exists in many other activation functions such as <em>identity</em>, <em>arctan</em>, and <em>sinusoid</em>) lets the network learn efficiently even when its weights are initialized with small values. In other cases (eg <em>Sigmoid</em> and <em>ReLU</em>) these small initial values <em>can</em> be problematic.</p>
  
  <p><strong>Further Reading:</strong></p>
  
  <p><a href="https://arxiv.org/abs/1412.6558" rel="nofollow noreferrer">Random Walk Initialization for Training Very Deep Feedforward Networks</a> </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-08T07:14:59.977Z
Answers.LastActivityDate: 2019-04-02T18:31:34.850Z
Answers.LastEditDate: 2019-04-02T18:31:34.850Z
Answers.UserDisplayName: Borhan Kazimipour
Body: <p>Two common activation functions used in deep learning are the hyperbolic tangent function and the sigmoid activation function. I understand that the hyperbolic tangent is just a rescaling and translation of the sigmoid function: </p>
  
  <p><span class="math-container">$\tanh(z) = 2\sigma(z) - 1$</span>. </p>
  
  <p>Is there a significant difference between these two activation functions, and in particular, <strong>when is one preferable to the other</strong>?</p>
  
  <p>I realize that in some cases (like when estimating probabilities) outputs in the range of <span class="math-container">$[0,1]$</span> are more convenient than outputs that range from <span class="math-container">$[-1,1]$</span>. I want to know if there are differences <strong>other than convenience</strong> which distinguish the two activation functions.</p>
  
Tags: <neural-networks><machine-learning><deep-neural-networks><comparison><hidden-layers>
Title: What's the difference between hyperbolic tangent and sigmoid neurons?
UserDisplayName: bpachev
ThreadText: # What's the difference between hyperbolic tangent and sigmoid neurons?
  
  (Asked by: username_0 on 2016-08-02)
  
  Two common activation functions used in deep learning are the hyperbolic tangent function and the sigmoid activation function. I understand that the hyperbolic tangent is just a rescaling and translation of the sigmoid function:
  
  $\tanh(z) = 2\sigma(z) - 1$.
  
  Is there a significant difference between these two activation functions, and in particular, **when is one preferable to the other**?
  
  I realize that in some cases (like when estimating probabilities) outputs in the range of $[0,1]$ are more convenient than outputs that range from $[-1,1]$. I want to know if there are differences **other than convenience** which distinguish the two activation functions.
  
  **Tags:** neural-networks, machine-learning, deep-neural-networks, comparison, hidden-layers
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-03-08. Score: 5 | Accepted answer
  
  **Sigmoid > Hyperbolic tangent:**
  
  As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @username_1-graves says, we can fix this with a simple mapping/calibration step). In other layers, this makes no sense.
  
  **Hyperbolic tangent > Sigmoid:**
  
  Hyperbolic tangent has a property called "approximates identity near the origin" which means $\tanh(0) = 0$, $\tanh'(0) = 1$, and $\tanh'(z)$ is continuous around $z=0$ (as opposed to $\sigma(0)=0.5$ and $\sigma'(0)=0.25$). This feature (which also exists in many other activation functions such as *identity*, *arctan*, and *sinusoid*) lets the network learn efficiently even when its weights are initialized with small values. In other cases (eg *Sigmoid* and *ReLU*) these small initial values *can* be problematic.
  
  **Further Reading:**
  
  [Random Walk Initialization for Training Very Deep Feedforward Networks](https://arxiv.org/abs/1412.6558)
  
  
  ---
  
  ### Answer by username_1 on 2016-08-02. Score: 3
  
  I don't think it makes sense to decide activation functions based on desired properties of the output; you can easily insert a calibration step that maps the 'neural network score' to whatever units you actually want to use (dollars, probability, etc.).
  
  So I think preference between different activation functions mostly boils down to the [different properties](https://en.wikipedia.org/wiki/Activation_function#Comparison_of_activation_functions) of those activation functions (like whether or not they're continuously differentiable). Because there's just a linear transformation between the two, I think that means there isn't a meaningful difference between them.
  
  
  ---
  
  