Solved by crowsonkb in post #2 
              
              
                I don’t know—I think the optimal learning rate has to do with other aspects of the problem besides gradient scale. That said, SGD and other optimizers like Adam use the learning rate in different ways. Adam automatically adapts the step size to the gradient scale on a per-parameter basis; its ‘learn…