Answers.Body: <blockquote>
  <p>How is back propagation applied when the activation function is not differentiable?</p>
  </blockquote>
  <p>It isn't. You cannot establish gradient values through non-differentiable functions.</p>
  <p>However, this isn't a common reason to reject an activation function. When an activation function cannot be differentiated at one or two specific values, that isn't a problem. ReLU cannot be differentiated at 0 for example, and is often a good choice. The problem at 0 can be ignored by assigning an arbitrary gradient value (typically 0 or 1).</p>
  <p>If someone tells you the step function cannot be used in backpropagation because it is &quot;non-differentiable&quot;, that is not strictly true. The domain of where it is differentiable is the same as ReLU. The problem with the step function is that it doesn't have a <em>useful</em> gradient. Many activation functions are problematic because the gradients - despite being calculatable - are not well-behaved in other ways. Very large gradients cause instability, or areas of very low/zero gradients mean training cannot progress.</p>
  <p>If you want to train a neural network with non-differentiable activation functions, or functions that would be unstable using gradient approaches, then you cannot use backpropagation. However, you can search for optimal neural network weights in other ways, such as genetic algorithms. These search methods are typically less efficient than gradient-based approaches, but might be worth it if there is some other good reason for picking a &quot;difficult&quot; activation function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-16T08:02:53.430Z
Answers.LastActivityDate: 2023-12-16T08:02:53.430Z
Answers.UserDisplayName: Neil Slater
Body: <p>Back propagation is based on partial derivatives including the activation functions.</p>
  <p>How is back propagation applied when the activation function is not differentiable?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-17T17:28:59.610Z
Comments.Text: It would be great if you can explain the rationale behind your question because it seems you know when BP is applied, but you still ask about the impossibility.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><backpropagation>
Title: How is back propagation applied in case the activation function is not differentiable?
UserDisplayName: DSPinfinity
ThreadText: # How is back propagation applied in case the activation function is not differentiable?
  
  (Asked by: username_0 on 2023-12-16)
  
  Back propagation is based on partial derivatives including the activation functions.
  
  How is back propagation applied when the activation function is not differentiable?
  
  **Tags:** neural-networks, deep-learning, backpropagation
  
  > **Comments:**
  > **username_1**: It would be great if you can explain the rationale behind your question because it seems you know when BP is applied, but you still ask about the impossibility.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-12-16. Score: 1 | Accepted answer
  
  > How is back propagation applied when the activation function is not differentiable?
  
  It isn't. You cannot establish gradient values through non-differentiable functions.
  
  However, this isn't a common reason to reject an activation function. When an activation function cannot be differentiated at one or two specific values, that isn't a problem. ReLU cannot be differentiated at 0 for example, and is often a good choice. The problem at 0 can be ignored by assigning an arbitrary gradient value (typically 0 or 1).
  
  If someone tells you the step function cannot be used in backpropagation because it is "non-differentiable", that is not strictly true. The domain of where it is differentiable is the same as ReLU. The problem with the step function is that it doesn't have a *useful* gradient. Many activation functions are problematic because the gradients - despite being calculatable - are not well-behaved in other ways. Very large gradients cause instability, or areas of very low/zero gradients mean training cannot progress.
  
  If you want to train a neural network with non-differentiable activation functions, or functions that would be unstable using gradient approaches, then you cannot use backpropagation. However, you can search for optimal neural network weights in other ways, such as genetic algorithms. These search methods are typically less efficient than gradient-based approaches, but might be worth it if there is some other good reason for picking a "difficult" activation function.
  
  
  ---
  
  