Machine Learning, Neural Networks, Artificial Intelligence, Google Brain, Math.
more activities within the network when the input value was multiplied with low alpha value. (such as 0.02). Finally, it is already well known fact that saturation of gradient prevents the model from converging to good quality minima, and if that happens the network’s performance is hindered