I have not solved it yet. There is an open issue for it on github https://github.com/pytorch/pytorch/issues/28478 36 maybe you can give a thumbs up on it to give it more attention.
I had an idea which was manually call the set parameters function after the gradient step. I haven’t tried it yet.