torch.autograd.grad does not manipulate the .grad attributes, but returns the desired gradients as the return value.
Here is a small example for resnet18, where I only calculate the gradients for the weight parameter of the first convolution layer.
Note that even though I get the gradients as the return value, the .grad attribute is still None:
model = models.resnet18()

out = model(torch.randn(1, 3, 224, 224))
grad = torch.autograd.grad(out.mean(), model.conv1.weight)
print(grad[0].shape == model.conv1.weight.shape)
> True
for name, param in model.named_parameters():
    print(name, param.grad)

So you would have to apply the gradients manually or let Autograd calculate and assign them to the .grad attributes via loss.backward().
I’m not sure if this might be related, but you are also detaching the returns tensor.
I’m not familiar with your use case, so it might work as expected.