I’m not sure if this is the right place to ask this question but I’m going to ask this anyways because people on this forum are really helpful and intelligent!
I was reading about non-linearity functions and given that ReLU or its family (leaky ReLU, PReLU, etc) all are discontinuous functions, yet they work really really well with gradient based optimization algorithms.
How does this work? Shouldn’t the non-linearity create a problem while calculating gradients?