retain_graph wouldn’t change the issue, since the calculation itself is failing, because needed values are gone.
If you know how the gradient should be calculated after the manipulation, you could try to implement a custom autograd.Function as described here 2.
In any case, I’m still unsure why you want to manipulate the values.