It’s likely that the lr was lowered by the optimizer and the new one doesn’t know about it, so it applies too large updates for a few iterations. You can always safely modify the parameter_groups dict, but I guess we’ll need to figure out a better way.