I think you can try with optimizer.add_param_group() https://pytorch.org/docs/stable/optim.html#torch.optim.Optimizer.add_param_group 1
If you look at the source code of this function, there is more going on under the hood .