Solved by mrshenli in post #6 
              
              
                I agree with all your analysis on the magnitude of the gradients, and I agree that it depends on the loss function. But even with MSE loss fn, it can lead to different conclusions: 

If the fw-bw has processed 8X data, we should set lr to 8X, meaning that the model should take a larger step if it ha…