Calculus, Backpropagation, Neural Networks, Mathematics, AI.
but the derivation of ReLU is a bit annoying to deal with so I’ll just be wishy washy and not derive it.* * if you’ve ever seen the graph of ReLU you can tell that it isn’t differentiable in the way we are familiar with. It’s technically not continuous — and it’s differentiation is described