Machine Learning, Neural Networks, Backpropagation, Gradient Descent.
non-linearity. They are used at every layer in a Neural Network. Now, in neural networks, we stack such layers one over the others. Finally, we obtain complex functions using cascaded functions like f(f(x)). The common types of activation function are: ReLU: f(x)= max(0,x) Sigmoid: Sigmoid 3. Tanh