ekurtic:

However, multiplying after should be fine too but you have to be careful and multiply before the optimizer.step() is called. I guess this second approach can then be achieved in two different ways: either by multiplying gradients (which could be slow) or by multiplying your learning_rate so that the final weight update has a proper scale.


Good call, I will have to remember that! Thanks for your quick answer.