Answers.Body: <p>Not completely sure your question. Do you mean</p>
  
  <p>Q. why should we use activation function? </p>
  
  <p>Ans: we need to introduce non-linearity to the network. Otherwise, multiple layers are no difference from single layer network. (It is obvious as we write things in matrix form, and say when we have two layers with weights <span class="math-container">$W_1$</span> and <span class="math-container">$W_2$</span>, the two layer is no difference from a single layer with weight <span class="math-container">$W_2 W_1$</span>.</p>
  
  <p>Q. why they need to be differentiable?</p>
  
  <p>Ans: Just for sake that we can back-propagate gradients back to earlier layers. Note that back-propagation is nothing but the chain rule in calculus. Say <span class="math-container">$f(\cdot)$</span> is an activation function in one layer and the output of that activation function is <span class="math-container">$\bf y$</span> and the input is <span class="math-container">${\bf u}=W \bf x$</span>, where <span class="math-container">$\bf x$</span> is output from the previous layer and mix with weights <span class="math-container">$W$</span> in current layer. Of course, the final loss <span class="math-container">$L$</span> will depend on <span class="math-container">${\bf y} = f({\bf u})= f(W {\bf x})$</span>. Say, loss <span class="math-container">$L=g(\bf y)$</span> somehow. To train the weights <span class="math-container">$W$</span>, we have to find the gradient <span class="math-container">$\frac{\partial L}{\partial W}$</span> so that we can adjust weight <span class="math-container">$W$</span> to minimize <span class="math-container">$L$</span>. But <span class="math-container">$\frac{\partial L}{\partial W}=\frac{\partial g({\bf y})}{\partial W}=\frac{\partial g({\bf y})}{\partial \bf y}\frac{\partial {\bf y}}{\partial {\bf u}}\frac{\partial {\bf u}}{\partial W}$</span>. Each of these product terms can be computed locally and will be accumulatively multiplied as we apply backprop.  And note that the middle term <span class="math-container">$\frac{\partial {\bf y}}{\partial {\bf u}}=\frac{\partial f({\bf u})}{\partial {\bf u}}$</span> is just "derivative" of <span class="math-container">$f(\cdot)$</span>, thus we require the activation function to be differentiable and "informative"/non-zero (at least most of the time). Note that ReLU is not differentiable everywhere and that is why researchers (at least Yoshua Bengio) worried about that when they first tried to adopt ReLU. You may check out the interview of Bengio by Andrew Ng for that. </p>
  
  <p>Q. why step function is a bad activation function?</p>
  
  <p>Ans: Note that step function is differentiable almost everywhere but is not "informative" though. For places (flat region) where it is differentiable, the derivative is simply zero. Consequently, any later layer gradient (information) will get cut off as it passes through a step function activation function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-05T19:29:09.823Z
Answers.Text: I think you should have asked for clarifications before answering. I have edited the question and I tried to clarify it, but I am not sure this is what the OP was asking. He/she will eventually clarify it.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-07T23:31:37.060Z
Answers.Text: Note that ReLU and some other non-continuously differentiable functions *do* possess well-defined [subgradients](https://en.wikipedia.org/wiki/Subderivative) over the full range.  We can simply select one of those to use as the gradient at the non differentiable points and backprop will work fine.
Answers.UserDisplayName: Ray
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-05T19:19:34.603Z
Answers.LastActivityDate: 2019-04-05T19:25:06.237Z
Answers.LastEditDate: 2019-04-05T19:25:06.237Z
Answers.UserDisplayName: user559678
Answers.Body: <p>Differentiability for activation functions is desirable but not necessary since you can reformulate the derivatives at the non-differentiable points, like in the case of ReLU.</p>
  <p>Properties Needed:</p>
  <ul>
  <li>To take advantage of Universal Approximation Theorem, and to take advantage of the modeling capacity it promises, the activation functions need to be continuous or Borel measurable (if this term is confusing just think of this as common functions), discriminatory (similarly discriminatory means the function doesn't just produce 0 all the time when integrating w.r.t. another function) and nonpolynomial (nonlinear).</li>
  <li>Also with the recent research works it is beneficial to have it monotonically increasing.</li>
  </ul>
  <p>Recent theoretical analysis of the activation functions gives the edge to ReLU with some preliminary theoretical guarantees like this &quot;Convergence Analysis of Two-layer Neural Networks with ReLU Activation&quot; <a href="https://arxiv.org/abs/1705.09886" rel="nofollow noreferrer">https://arxiv.org/abs/1705.09886</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T20:22:04.743Z
Answers.LastActivityDate: 2020-06-25T20:22:04.743Z
Answers.UserDisplayName: saiRegrefree
Body: <p>I read that functions are used as activation functions only when they are differentiable. What about the unit step activation function? So, is there any other reason a function can be used as an activation function (apart from being differentiable)?</p>
  
Tags: <neural-networks><machine-learning><activation-functions>
Title: What kind of functions can be used as activation functions?
UserDisplayName: hina munir
ThreadText: # What kind of functions can be used as activation functions?
  
  (Asked by: username_0 on 2019-04-05)
  
  I read that functions are used as activation functions only when they are differentiable. What about the unit step activation function? So, is there any other reason a function can be used as an activation function (apart from being differentiable)?
  
  **Tags:** neural-networks, machine-learning, activation-functions
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-04-05. Score: 2
  
  Not completely sure your question. Do you mean
  
  Q. why should we use activation function?
  
  Ans: we need to introduce non-linearity to the network. Otherwise, multiple layers are no difference from single layer network. (It is obvious as we write things in matrix form, and say when we have two layers with weights $W\_1$ and $W\_2$, the two layer is no difference from a single layer with weight $W\_2 W\_1$.
  
  Q. why they need to be differentiable?
  
  Ans: Just for sake that we can back-propagate gradients back to earlier layers. Note that back-propagation is nothing but the chain rule in calculus. Say $f(\cdot)$ is an activation function in one layer and the output of that activation function is $\bf y$ and the input is ${\bf u}=W \bf x$, where $\bf x$ is output from the previous layer and mix with weights $W$ in current layer. Of course, the final loss $L$ will depend on ${\bf y} = f({\bf u})= f(W {\bf x})$. Say, loss $L=g(\bf y)$ somehow. To train the weights $W$, we have to find the gradient $\frac{\partial L}{\partial W}$ so that we can adjust weight $W$ to minimize $L$. But $\frac{\partial L}{\partial W}=\frac{\partial g({\bf y})}{\partial W}=\frac{\partial g({\bf y})}{\partial \bf y}\frac{\partial {\bf y}}{\partial {\bf u}}\frac{\partial {\bf u}}{\partial W}$. Each of these product terms can be computed locally and will be accumulatively multiplied as we apply backprop. And note that the middle term $\frac{\partial {\bf y}}{\partial {\bf u}}=\frac{\partial f({\bf u})}{\partial {\bf u}}$ is just "derivative" of $f(\cdot)$, thus we require the activation function to be differentiable and "informative"/non-zero (at least most of the time). Note that ReLU is not differentiable everywhere and that is why researchers (at least Yoshua Bengio) worried about that when they first tried to adopt ReLU. You may check out the interview of Bengio by Andrew Ng for that.
  
  Q. why step function is a bad activation function?
  
  Ans: Note that step function is differentiable almost everywhere but is not "informative" though. For places (flat region) where it is differentiable, the derivative is simply zero. Consequently, any later layer gradient (information) will get cut off as it passes through a step function activation function.
  
  > **Comments:**
  > **username_2**: I think you should have asked for clarifications before answering. I have edited the question and I tried to clarify it, but I am not sure this is what the OP was asking. He/she will eventually clarify it.  
  > **username_3**: Note that ReLU and some other non-continuously differentiable functions *do* possess well-defined [subgradients](https://en.wikipedia.org/wiki/Subderivative) over the full range.  We can simply select one of those to use as the gradient at the non differentiable points and backprop will work fine.  
  
  ---
  
  ### Answer by username_4 on 2020-06-25. Score: 0
  
  Differentiability for activation functions is desirable but not necessary since you can reformulate the derivatives at the non-differentiable points, like in the case of ReLU.
  
  Properties Needed:
  
  * To take advantage of Universal Approximation Theorem, and to take advantage of the modeling capacity it promises, the activation functions need to be continuous or Borel measurable (if this term is confusing just think of this as common functions), discriminatory (similarly discriminatory means the function doesn't just produce 0 all the time when integrating w.r.t. another function) and nonpolynomial (nonlinear).
  * Also with the recent research works it is beneficial to have it monotonically increasing.
  
  Recent theoretical analysis of the activation functions gives the edge to ReLU with some preliminary theoretical guarantees like this "Convergence Analysis of Two-layer Neural Networks with ReLU Activation" <https://arxiv.org/abs/1705.09886>.
  
  
  ---
  
  