Optimization, Deep Learning, Convex Optimization, Optimal Performance, Approximation.
we can use adaptive step size or more intuitively limit the size of the gradient step. Finally, if we end up having a wrong setting of hyperparameters, we can go for hyperparameter optimization methods. Finally, it seems that the NCO in DL will remain an art unless someone finds a generic framework