I believe you have to re-instantiate a new optimizer every time you change your lr if you want to achieve this. This also means you’ll have to store the optimizer.state_dict and then load it into the new optimizer on each iteration.