I was trying to verify what you just said that I came to something unexpected.
I have a dummy example of a 1*2 network without the inner layer. I compute the loss based on the first node of the output. As a result, the gradient on the second node is zero as the code also shows correctly. However, when I check the gradients on the weights, I can see non-zero values for the second row, which I expected to be zeros. How does this happen? if the gradient on a node is zero, shouldn’t the weights coming to that node also have zero gradients?
import torch 
from torch import optim
#### network ####
class Net(torch.nn.Module):
  def __init__(self, N, M):
      super(Net, self).__init__()
      self.fc1 = torch.nn.Linear(N, M)

  def forward(self, x):
      x = torch.softmax(self.fc1(x),1)
      return x 

nn = Net(1,2)
optimizer = optim.Adam(nn.parameters(), lr=0.01)

xx = torch.ones((1,1),requires_grad = True)

ys =  nn.forward(xx)
ys.register_hook(print)
loss = abs(ys[0][0]-0.5)  
optimizer.zero_grad()
loss.backward()
print(nn.fc1.weight.grad)
optimizer.step()



output:
Gradients on the output nodes:
tensor([[-1.,  0.]])
Gradients on the weights:
tensor([[-0.1939],
        [ 0.1939]])