I think there is a misunderstanding how gradients are calculated.



 hyunwookim:

Shouldn’t gradients be tracked throughout each all forward-passing functions until it reaches the loss function so it can apply partial derivatives to calculate the weight?


No, forward activations are stored if they are needed for the gradient calculation, but you need to finish the entire forward pass before calculating the gradients in the backward pass.
In your code snippets you are checking the .grad of activations in the forward method, which won’t work because:


backward() wasn’t called yet, so the gradients are not calculated
you are accessing the .grad attribute of non-leaf tensors, which are not stored by default as explained in the warning you should see:

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.

This code should work using your simple model:
model = nn.Sequential(nn.Linear(10,2), nn.Linear(2,1))

x = torch.randn(1, 10)
x1 = model(x)
x1.retain_grad()
x2 = x1**2
x2.retain_grad()
x3 = x2/ 10
x3.retain_grad()

loss = torch.norm(x3)
loss.backward()

print(x1.grad)
print(x2.grad)
print(x3.grad)