Answers.Body: <p>Even the first artificial neural network - Rosenblatt's perceptron [<a href="https://psycnet.apa.org/doiLanding?doi=10.1037%2Fh0042519" rel="noreferrer">1</a>] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, <em>Michael Negnevitsky. Artificial intelligence: a guide to intelligent systems. Second Edition</em> shows how to train such networks on pages 170-174.</p>
  
  <p>Error backpropagation algorithm can be modified to accommodate discontinuous activation functions. The details are in paper [<a href="https://ieeexplore.ieee.org/document/51993" rel="noreferrer">2</a>]. That paper points out a possible application: training a neural network on micro-controllers. As the multiplication of the output of the previous layer <span class="math-container">$x_j$</span> by the weigth <span class="math-container">$w_{ij}$</span> is exspensive, the author suggested to approximate it with a left shift by <span class="math-container">$n$</span> bits (multiplication by <span class="math-container">$2^n$</span>) for the corresponding <span class="math-container">$n$</span> in which case the activation function is discontinuous (a staircase). </p>
  
  <p>An example of a neural network with discontinuous activation functions applied to a restoration of degraded images is in Ref. [<a href="https://ieeexplore.ieee.org/document/4634149" rel="noreferrer">3</a>]. Applications of recurrent neural networks with discontinuous activation functions to convex optimization problems are in Ref. [<a href="https://www.worldscientific.com/doi/abs/10.1142/9789814280150_0004" rel="noreferrer">4</a>]. Probably more examples can be found in the literature.</p>
  
  <p>References</p>
  
  <ol>
  <li>Rosenblatt, F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev. 1958 Nov; 65(6):386-408. PMID: 13602029 DOI: 10.1037/h0042519 </li>
  <li>Findlay, D.A. Training networks with discontinuous activation functions. 1989 First IEE International Conference on Artificial Neural Networks, (Conf. Publ. No. 313), London, UK, 1989, pp. 361-363.</li>
  <li>Ferreira, L. V.; Kaszkurewicz, E.; Bhaya, A. Image restoration using L1-norm regularization and a gradient-based neural network with discontinuous activation functions. 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, 2008, pp. 2512-2519.  DOI: 10.1109/IJCNN.2008.4634149 </li>
  <li>Liu, Q.; Wang, J. Recurrent Neural Networks with Discontinuous Activation Functions for Convex Optimization. Integration of Swarm Intelligence and Artificial Neural Network, pp. 95-119 (2011) DOI: 10.1142/9789814280150_0004</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-06T06:06:50.983Z
Answers.Text: The problem is how do you find which way to adjust your function without some type of gradient information?
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-07T02:45:06.683Z
Answers.Text: @FourierFlux This "some type of gradient" does not have to be continuous. Just estimate where the error is lower and modify the weights to move to that region. The gradient is only continuous in analytic formulas, anyway. It is discrete in a computer. Besides that, following the exact gradient is expensive, so approximations are used such as the stochastic gradient descent. This means that you don't have to know the exact gradient. Discrete approximations to derivatives will do.
Answers.UserDisplayName: Vladislav Gladkikh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-09T05:26:40.953Z
Answers.LastActivityDate: 2020-04-09T05:26:40.953Z
Answers.UserDisplayName: Vladislav Gladkikh
Answers.Body: <p>I would say that it is possible, but probably not a very good idea.?Like you say, the hard requirement is that the network (and thus its components, including the activation functions) must be differentiable.  ReLU isn't, but you can cheat by defining f'(0) to be 0 (or 1).</p>
  
  <p>A continuous function means that gradient descent leads to some local minimum¹, for piecewise continuous functions, it may not converge (i.e. the breakpoints themselves may not be part of the segment you descend, so you will never get get to an actual minimum).  This is not likely to be a problem in practice, though.</p>
  
  <p>¹ At least for functions that are bounded from below, like cost functions are.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-22T23:04:51.127Z
Answers.Text: I think that this answer could be improved by citing any paper or, in general, a reliable source that discusses the topic (i.e. the usage of discontinuous activation functions in neural networks).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-23T11:53:13.657Z
Answers.Text: I'm not aware of any, off hand. Feel free to suggest edits!
Answers.UserDisplayName: Ketil Malde
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-22T20:41:37.620Z
Answers.LastActivityDate: 2020-01-22T20:41:37.620Z
Answers.UserDisplayName: Ketil Malde
Body: <p>In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)?</p>
  <p>(My guess: for example, ReLU is non-differentiable at a single point, but it still has a well-defined derivative. If an activation function has a jump discontinuity, then its derivative is supposed to have a delta function at that point. However, the backpropagation process is incapable of considering that delta function, so the convex optimization process will have some problem?)</p>
  
Tags: <deep-learning><backpropagation><optimization><activation-functions><relu>
Title: In deep learning, is it possible to use discontinuous activation functions?
UserDisplayName: Gyeonghoon Ko
ThreadText: # In deep learning, is it possible to use discontinuous activation functions?
  
  (Asked by: username_0 on 2020-01-22)
  
  In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)?
  
  (My guess: for example, ReLU is non-differentiable at a single point, but it still has a well-defined derivative. If an activation function has a jump discontinuity, then its derivative is supposed to have a delta function at that point. However, the backpropagation process is incapable of considering that delta function, so the convex optimization process will have some problem?)
  
  **Tags:** deep-learning, backpropagation, optimization, activation-functions, relu
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-04-09. Score: 5
  
  Even the first artificial neural network - Rosenblatt's perceptron [[1](https://psycnet.apa.org/doiLanding?doi=10.1037%2Fh0042519)] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, *Michael Negnevitsky. Artificial intelligence: a guide to intelligent systems. Second Edition* shows how to train such networks on pages 170-174.
  
  Error backpropagation algorithm can be modified to accommodate discontinuous activation functions. The details are in paper [[2](https://ieeexplore.ieee.org/document/51993)]. That paper points out a possible application: training a neural network on micro-controllers. As the multiplication of the output of the previous layer $x\_j$ by the weigth $w\_{ij}$ is exspensive, the author suggested to approximate it with a left shift by $n$ bits (multiplication by $2^n$) for the corresponding $n$ in which case the activation function is discontinuous (a staircase).
  
  An example of a neural network with discontinuous activation functions applied to a restoration of degraded images is in Ref. [[3](https://ieeexplore.ieee.org/document/4634149)]. Applications of recurrent neural networks with discontinuous activation functions to convex optimization problems are in Ref. [[4](https://www.worldscientific.com/doi/abs/10.1142/9789814280150_0004)]. Probably more examples can be found in the literature.
  
  References
  
  1. Rosenblatt, F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev. 1958 Nov; 65(6):386-408. PMID: 13602029 DOI: 10.1037/h0042519
  2. Findlay, D.A. Training networks with discontinuous activation functions. 1989 First IEE International Conference on Artificial Neural Networks, (Conf. Publ. No. 313), London, UK, 1989, pp. 361-363.
  3. Ferreira, L. V.; Kaszkurewicz, E.; Bhaya, A. Image restoration using L1-norm regularization and a gradient-based neural network with discontinuous activation functions. 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, 2008, pp. 2512-2519. DOI: 10.1109/IJCNN.2008.4634149
  4. Liu, Q.; Wang, J. Recurrent Neural Networks with Discontinuous Activation Functions for Convex Optimization. Integration of Swarm Intelligence and Artificial Neural Network, pp. 95-119 (2011) DOI: 10.1142/9789814280150\_0004
  
  > **Comments:**
  > **username_2**: The problem is how do you find which way to adjust your function without some type of gradient information?  
  > **username_1**: @username_2 This "some type of gradient" does not have to be continuous. Just estimate where the error is lower and modify the weights to move to that region. The gradient is only continuous in analytic formulas, anyway. It is discrete in a computer. Besides that, following the exact gradient is expensive, so approximations are used such as the stochastic gradient descent. This means that you don't have to know the exact gradient. Discrete approximations to derivatives will do.  
  
  ---
  
  ### Answer by username_3 on 2020-01-22. Score: 1
  
  I would say that it is possible, but probably not a very good idea.?Like you say, the hard requirement is that the network (and thus its components, including the activation functions) must be differentiable. ReLU isn't, but you can cheat by defining f'(0) to be 0 (or 1).
  
  A continuous function means that gradient descent leads to some local minimum¹, for piecewise continuous functions, it may not converge (i.e. the breakpoints themselves may not be part of the segment you descend, so you will never get get to an actual minimum). This is not likely to be a problem in practice, though.
  
  ¹ At least for functions that are bounded from below, like cost functions are.
  
  > **Comments:**
  > **username_4**: I think that this answer could be improved by citing any paper or, in general, a reliable source that discusses the topic (i.e. the usage of discontinuous activation functions in neural networks).  
  > **username_3**: I'm not aware of any, off hand. Feel free to suggest edits!  
  
  ---
  
  