If you have the compute/patience is there a reason not to train DNN slowly? When fitting a deep neural network it is not uncommon for the learning rate to be too high, and then the fit goes horribly wrong.  On the other hand as far as I can tell, the only problem with setting the learning rate low is that it takes a long time.
Is there any other downside to a low learning rate?  Or if you are patient enough is it acceptable to just train for say a million epochs?  In other words, is it better to err on the side of learning slowly and using as much time as you can afford?

In general when optimizing a complex cost function that has many local minima, you have to worry not just about converging to one of them, but also about avoiding ending up in a shallow local minimum. Using a very small step size (learning rate) roughly means that the optimization will converge to the nearest  local minimum to the starting point. Imagine for example what will happen if you cost function looks something like this and your starting point is at the red circle :

A larger step size can help you skip over many local minima and might give a better chance of reaching the global minimum at the bottom. (Of course this is a bit simplistic, it's hard to make definite statements about very complex function such as those of a deep networks, so there are lot's of heuristics involved)
