Ok so the issue is I have some list of parameters which have non-zero .grad values (backward from one loss function). Now I use those .grad values to calculate another grad via torch.autograd.grad which gives another set of .grad values for only some of the parameters. Now I want to use different optimizers for updating those .grad values. Do you have any insight on how to do this, other than making a copy of all the .grad tensors from one loss and doing the step() update separately?