Could you post the code you are using to select the neuron?
This dummy example seems to work:
lin = nn.Linear(2, 2, bias=False)
x = torch.randn(1, 2)
out = lin(x)
loss = out.mean()
loss.backward()
print(lin.weight.grad)

lin.zero_grad()
out = lin(x)
loss = out.mean() + lin.weight[0, 0]
loss.backward()
print(lin.weight.grad)

I haven’t tested it on more complicated models, so let me know, what the difference between your code and the example is.