Calling torch.autograd.grad will not accumulate gradients in your tensors, but will free the computation graph.
Can your problem be solved by using a strategically-placed .detach somewhere in your loss computation?