Answers.Body: <p>Consider a dataset <span class="math-container">$\mathcal{D}=\{x^{(i)},y^{(i)}:i=1,2,\ldots,N\}$</span> where <span class="math-container">$x^{(i)}\in\mathbb{R}^3$</span> and <span class="math-container">$y^{(i)}\in\mathbb{R}$</span> <span class="math-container">$\forall i$</span></p>
  
  <p>The goal is to fit a function that best explains our dataset.We can fit a simple function, as we do in <strong>linear regression</strong>. But that's different about neural networks, where we fit a complex function, say:</p>
  
  <p><span class="math-container">$\begin{align}h(x) &amp; = h(x_1,x_2,x_3)\\
  &amp; =\sigma(w_{46}\times\sigma(w_{14}x_1+w_{24}x_2+w_{34}x_3+b_4)+w_{56}\times\sigma(w_{15}x_1+w_{25}x_2+w_{35}x_3+b_5)+b_6)\end{align}$</span></p>
  
  <p>where, <span class="math-container">$\theta = \{w_{14},w_{24},w_{34},b_4,w_{15},w_{25},w_{35},b_5,w_{46},w_{56},b_6\}$</span> is the set of the respective coefficients we have to determine such that we minimize:
  <span class="math-container">$$J(\theta) = \frac{1}{2}\sum_{i=1}^N (y^{(i)}-h(x^{(i)}))^2$$</span>
  The above optimization problem can be easily solved with <strong><em>gradient descent</em></strong>. Just initiate <span class="math-container">$\theta$</span> with random values and with proper learning parameter <span class="math-container">$\eta$</span>, update as follows till convergence:
  <span class="math-container">$$\theta:=\theta-\eta\frac{\partial J}{\partial \theta}$$</span></p>
  
  <p>In order to get the gradients, we express the above function as a neural network as follows:
  <a href="https://i.sstatic.net/rf7db.png" rel="noreferrer"><img src="https://i.sstatic.net/rf7db.png" alt="enter image description here"></a></p>
  
  <p>Let's calculate the gradient, say w.r.t. <span class="math-container">$w_{14}$</span>.<br>
  <span class="math-container">$$\frac{\partial J}{\partial w_{14}} = \sum_{i=1}^N \Big[\big(h(x^{(i)})-y^{(i)}\big)\frac{\partial h(x^{(i)})}{\partial w_{14}}\Big]$$</span>
  Let <span class="math-container">$p(x) = w_{14}x_1+w_{24}x_2+w_{34}x_3+b_4$</span> , and<br>
  Let <span class="math-container">$q(x) = w_{46}\times\sigma(p(x))+w_{56}\times\sigma(w_{15}x_1+w_{25}x_2+w_{35}x_3+b_5)+b_6)$</span></p>
  
  <p><span class="math-container">$\therefore \frac{\partial h(x)}{\partial w_{14}} = \frac{\partial h(x)}{\partial q(x)}\times\frac{\partial q(x)}{\partial p(x)}\times\frac{\partial p(x)}{\partial w_{14}} = \frac{\partial\sigma(q(x))}{\partial q(x)}\times\frac{\partial\sigma(p(x))}{\partial p(x)}\times\frac{\partial p(x)}{\partial w_{14}}$</span></p>
  
  <p>We see that the derivative of the activation function is important for getting the gradients and so for the learning of the neural network. A constant derivative will not help in the gradient descent and we won't be able to learn the optimal parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-23T19:56:13.083Z
Answers.LastActivityDate: 2019-08-24T05:08:08.457Z
Answers.LastEditDate: 2019-08-24T05:08:08.457Z
Answers.UserDisplayName: babkr
Answers.Body: <p>If what you are asking is what is the <em>intuition</em> for using the <a href="https://en.wikipedia.org/wiki/Derivative" rel="noreferrer">derivative</a> in backpropagation learning, instead of an in-depth mathematical explanation:</p>
  
  <p>Recall that the derivative tells you a function's sensitivity to change with respect to a change in its input. A high (absolute) value for the derivative at a certain point means that the function is very <em>steep</em>, and a small change in input may result in a drastic change in its output; conversely, a low absolute value means little change, so not <em>steep</em> at all, with the extreme case that the function is constant when the derivative is zero.</p>
  
  <p>Training a neural network essentially amounts to an optimization problem where one wants to minimize a certain value, in this case the error produced by the network on the given training examples. Backpropagation learning can be viewed as a case of <a href="https://en.wikipedia.org/wiki/Gradient_descent" rel="noreferrer">gradient descent</a> (the inverse of hill climbing).</p>
  
  <p>If for a moment we assume that your input is only 2-dimensional (just for illustration, the mathematics of course also work for higher dimensions), you could imagine the error function as a landscape with hills, mountains, valleys, ridges etc. You are standing at a high point and want to get down as far as possible. Gradient descent means that, in discrete steps, you always walk down in the direction that has the <em>steepest</em> slope downwards from where you are currently standing, until you eventually reach a (local) minimum.</p>
  
  <p>In order to determine where that steepest slope is, you need the derivative of the activation function. Basically, you want to sort out how much each unit in your network contributes to an error, and adjust in the direction that contributes the most.</p>
  
  <p><strong>Edit</strong>: Regarding constant values for a derivative, in the landscape metaphor it would mean that the gradient is the same no matter where you are, so you'll always go in the same direction and never reach an optimum. However, multi-layer networks with linear activation function are kind of besides the point anyhow when you consider that each cell computes a linear combination of its inputs, which then is again a linear function, so the output of the last layer will ultimately be a linear function of the inputs at the first layer. That is to say, anything you can do with a multi-layer net with linear activation functions, you could also achieve with just a single layer. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-15T01:43:00.913Z
Answers.Text: Thanks. It was a good starter explanation. I understand we want to minimize the whole loss function. But why we need a local minimum at each function?!
Answers.UserDisplayName: Mary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-15T02:09:28.407Z
Answers.Text: @Tina J: I am not sure what you are asking. You are correct that we try to find a _single_ minimum for the error of the _entire_ network. What backpropagation does is to split the observed error up into the parts contributed by each single unit and connection. So we don't minimize at each single unit, but for each training example, we (potentially) adjust every edge's weight, depending on how much it affected the outcome to be wrong. Each weight is a dimension of the "landscape", and one traversal of the net is a single step in gradient descent, which is repeated until reaching convergence.
Answers.UserDisplayName: Jens Classen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-15T00:39:17.153Z
Answers.LastActivityDate: 2019-08-15T01:01:59.207Z
Answers.LastEditDate: 2019-08-15T01:01:59.207Z
Answers.UserDisplayName: Jens Classen
Answers.Body: <p>The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, <a href="https://en.wikipedia.org/wiki/Numerical_analysis" rel="nofollow noreferrer">numerical</a> and <a href="https://en.wikipedia.org/wiki/Mathematical_optimization" rel="nofollow noreferrer">optimization algorithm</a>, called <a href="https://en.wikipedia.org/wiki/Gradient_descent" rel="nofollow noreferrer"><em>gradient descent</em></a>, which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is <a href="https://en.wikipedia.org/wiki/Multivariable_calculus" rel="nofollow noreferrer">multivariable</a>, then, rather than the derivative, gradient descent requires the <a href="https://en.wikipedia.org/wiki/Gradient" rel="nofollow noreferrer">gradient</a>, which is a vector where the <span class="math-container">$i$</span>th element contains the <em>partial</em> derivative of the function with respect to the <span class="math-container">$i$</span>th variable. Hence the name <em>gradient descent</em>, where the derivative of a function of one variable can be considered the gradient of the function. </p>
  
  <p>In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the <a href="https://en.wikipedia.org/wiki/Chain_rule" rel="nofollow noreferrer">chain rule</a>) is often used, given that the MSE is a function of the ANN, which is a composite function of multiple non-linear functions, the activation functions, whose main purpose is thus to introduce non-linearity, or, in other words, it makes the ANN powerful. Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights: <em>to fully see this, you will need to learn the details of back-propagation!</em> Hence the importance of the derivatives of the activation functions.</p>
  
  <p>A constant derivative would always give the same <em>learning signal</em>, independently of the error, but this is not desirable. </p>
  
  <p><em>To fully understand all these statements, I recommend you learn about back-propagation and gradient descent in detail, which requires a little bit of effort!</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-15T19:54:15.023Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/97477/discussion-on-answer-by-nbro-why-is-the-derivative-of-the-activation-functions-i).
Answers.UserDisplayName: Ben N
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-15T00:43:38.783Z
Answers.LastActivityDate: 2019-08-15T01:29:03.253Z
Answers.LastEditDate: 2019-08-15T01:29:03.253Z
Answers.UserDisplayName: nbro
Body: <p>I'm new to NN. I am trying to understand some of its foundations. One question that I have is: <em>why the <strong>derivative</strong> of an activation function is important (not the function itself), and why it's the derivative which is tied to how the network performs learning</em>? </p>
  
  <p>For instance, when we say a constant derivative isn't good for learning, what is the intuition behind that? Is the activation function somehow like a <strong>hash function</strong> that needs to well differentiate small variance in inputs?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-14T23:27:48.167Z
Comments.Text: https://towardsdatascience.com/comparison-of-activation-functions-for-deep-neural-networks-706ac4284c8a
Comments.UserDisplayName: Mary
Tags: <neural-networks><machine-learning><deep-learning><math><activation-functions>
Title: Why is the derivative of the activation functions in neural networks important?
UserDisplayName: Mary
ThreadText: # Why is the derivative of the activation functions in neural networks important?
  
  (Asked by: username_0 on 2019-08-14)
  
  I'm new to NN. I am trying to understand some of its foundations. One question that I have is: *why the **derivative** of an activation function is important (not the function itself), and why it's the derivative which is tied to how the network performs learning*?
  
  For instance, when we say a constant derivative isn't good for learning, what is the intuition behind that? Is the activation function somehow like a **hash function** that needs to well differentiate small variance in inputs?
  
  **Tags:** neural-networks, machine-learning, deep-learning, math, activation-functions
  > **Comments:**
  > **username_0**: https://towardsdatascience.com/comparison-of-activation-functions-for-deep-neural-networks-706ac4284c8a  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-08-23. Score: 10
  
  Consider a dataset $\mathcal{D}=\{x^{(i)},y^{(i)}:i=1,2,\ldots,N\}$ where $x^{(i)}\in\mathbb{R}^3$ and $y^{(i)}\in\mathbb{R}$ $\forall i$
  
  The goal is to fit a function that best explains our dataset.We can fit a simple function, as we do in **linear regression**. But that's different about neural networks, where we fit a complex function, say:
  
  $\begin{align}h(x) & = h(x\_1,x\_2,x\_3)\\
  & =\sigma(w\_{46}\times\sigma(w\_{14}x\_1+w\_{24}x\_2+w\_{34}x\_3+b\_4)+w\_{56}\times\sigma(w\_{15}x\_1+w\_{25}x\_2+w\_{35}x\_3+b\_5)+b\_6)\end{align}$
  
  where, $\theta = \{w\_{14},w\_{24},w\_{34},b\_4,w\_{15},w\_{25},w\_{35},b\_5,w\_{46},w\_{56},b\_6\}$ is the set of the respective coefficients we have to determine such that we minimize:
  $$J(\theta) = \frac{1}{2}\sum\_{i=1}^N (y^{(i)}-h(x^{(i)}))^2$$
  The above optimization problem can be easily solved with ***gradient descent***. Just initiate $\theta$ with random values and with proper learning parameter $\eta$, update as follows till convergence:
  $$\theta:=\theta-\eta\frac{\partial J}{\partial \theta}$$
  
  In order to get the gradients, we express the above function as a neural network as follows:
  [![enter image description here](https://i.sstatic.net/rf7db.png)](https://i.sstatic.net/rf7db.png)
  
  Let's calculate the gradient, say w.r.t. $w\_{14}$.  
  $$\frac{\partial J}{\partial w\_{14}} = \sum\_{i=1}^N \Big[\big(h(x^{(i)})-y^{(i)}\big)\frac{\partial h(x^{(i)})}{\partial w\_{14}}\Big]$$
  Let $p(x) = w\_{14}x\_1+w\_{24}x\_2+w\_{34}x\_3+b\_4$ , and  
  Let $q(x) = w\_{46}\times\sigma(p(x))+w\_{56}\times\sigma(w\_{15}x\_1+w\_{25}x\_2+w\_{35}x\_3+b\_5)+b\_6)$
  
  $\therefore \frac{\partial h(x)}{\partial w\_{14}} = \frac{\partial h(x)}{\partial q(x)}\times\frac{\partial q(x)}{\partial p(x)}\times\frac{\partial p(x)}{\partial w\_{14}} = \frac{\partial\sigma(q(x))}{\partial q(x)}\times\frac{\partial\sigma(p(x))}{\partial p(x)}\times\frac{\partial p(x)}{\partial w\_{14}}$
  
  We see that the derivative of the activation function is important for getting the gradients and so for the learning of the neural network. A constant derivative will not help in the gradient descent and we won't be able to learn the optimal parameters.
  
  
  ---
  
  ### Answer by username_2 on 2019-08-15. Score: 7
  
  If what you are asking is what is the *intuition* for using the [derivative](https://en.wikipedia.org/wiki/Derivative) in backpropagation learning, instead of an in-depth mathematical explanation:
  
  Recall that the derivative tells you a function's sensitivity to change with respect to a change in its input. A high (absolute) value for the derivative at a certain point means that the function is very *steep*, and a small change in input may result in a drastic change in its output; conversely, a low absolute value means little change, so not *steep* at all, with the extreme case that the function is constant when the derivative is zero.
  
  Training a neural network essentially amounts to an optimization problem where one wants to minimize a certain value, in this case the error produced by the network on the given training examples. Backpropagation learning can be viewed as a case of [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) (the inverse of hill climbing).
  
  If for a moment we assume that your input is only 2-dimensional (just for illustration, the mathematics of course also work for higher dimensions), you could imagine the error function as a landscape with hills, mountains, valleys, ridges etc. You are standing at a high point and want to get down as far as possible. Gradient descent means that, in discrete steps, you always walk down in the direction that has the *steepest* slope downwards from where you are currently standing, until you eventually reach a (local) minimum.
  
  In order to determine where that steepest slope is, you need the derivative of the activation function. Basically, you want to sort out how much each unit in your network contributes to an error, and adjust in the direction that contributes the most.
  
  **Edit**: Regarding constant values for a derivative, in the landscape metaphor it would mean that the gradient is the same no matter where you are, so you'll always go in the same direction and never reach an optimum. However, multi-layer networks with linear activation function are kind of besides the point anyhow when you consider that each cell computes a linear combination of its inputs, which then is again a linear function, so the output of the last layer will ultimately be a linear function of the inputs at the first layer. That is to say, anything you can do with a multi-layer net with linear activation functions, you could also achieve with just a single layer.
  
  > **Comments:**
  > **username_0**: Thanks. It was a good starter explanation. I understand we want to minimize the whole loss function. But why we need a local minimum at each function?!  
  > **username_2**: @Tina J: I am not sure what you are asking. You are correct that we try to find a _single_ minimum for the error of the _entire_ network. What backpropagation does is to split the observed error up into the parts contributed by each single unit and connection. So we don't minimize at each single unit, but for each training example, we (potentially) adjust every edge's weight, depending on how much it affected the outcome to be wrong. Each weight is a dimension of the "landscape", and one traversal of the net is a single step in gradient descent, which is repeated until reaching convergence.  
  
  ---
  
  ### Answer by username_3 on 2019-08-15. Score: 4
  
  The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, [numerical](https://en.wikipedia.org/wiki/Numerical_analysis) and [optimization algorithm](https://en.wikipedia.org/wiki/Mathematical_optimization), called [*gradient descent*](https://en.wikipedia.org/wiki/Gradient_descent), which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is [multivariable](https://en.wikipedia.org/wiki/Multivariable_calculus), then, rather than the derivative, gradient descent requires the [gradient](https://en.wikipedia.org/wiki/Gradient), which is a vector where the $i$th element contains the *partial* derivative of the function with respect to the $i$th variable. Hence the name *gradient descent*, where the derivative of a function of one variable can be considered the gradient of the function.
  
  In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the [chain rule](https://en.wikipedia.org/wiki/Chain_rule)) is often used, given that the MSE is a function of the ANN, which is a composite function of multiple non-linear functions, the activation functions, whose main purpose is thus to introduce non-linearity, or, in other words, it makes the ANN powerful. Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights: *to fully see this, you will need to learn the details of back-propagation!* Hence the importance of the derivatives of the activation functions.
  
  A constant derivative would always give the same *learning signal*, independently of the error, but this is not desirable.
  
  *To fully understand all these statements, I recommend you learn about back-propagation and gradient descent in detail, which requires a little bit of effort!*
  
  > **Comments:**
  > **username_4**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/97477/discussion-on-answer-by-nbro-why-is-the-derivative-of-the-activation-functions-i).  
  
  ---
  
  