Would your workflow work with hooks (via tensor.register_hook()) to manipulate the gradients?
If not, could you try to wrap the gradient manipulations into a with torch.no_grad() block, if that’s not already the case?