Hi @shirui-japina,
There is actually a guy called Leslie N. Smith 25 who created this paper 78.
Based on this paper, some other guy created the learning rate finder 515.
What, it does, it measures the loss for the different learning rates and plots the diagram as this one:

It shows up (empirically) that the best learning rate is a value that is approximately in the middle of the sharpest downward slope.
However, the modern practice is to alter the learning rate while training described in here 246.
At the end you would probable do learning rate annealing.

image.png730×264 16.1 KB

[first image is the learning rate second is the momentum in time]
It really depends on optimization algorithm but the techniques from fastai work well on Adam and SGD.
Lastly some newer optimizer algorithms don’t even care about the learning rate ;).