No, the intermediate activations won’t retain the gradient unless you explicitly ask for it:
lin1 = nn.Linear(1, 1)
lin2 = nn.Linear(1, 1)

# fails
x = torch.randn(1, 1)
y1 = lin1(x)
y2 = lin2(y1)

y2.backward()

print(y1.grad)
> UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.

# works
x = torch.randn(1, 1)
y1 = lin1(x)
y1.retain_grad()
y2 = lin2(y1)

y2.backward()
print(y1.grad)




 hyunwookim:

Do you notice anything in the above model that would cause such issues?


Yes, don’t use the .data attribute and don’t call to() on the nn.Parameter as it won’t be a leaf node anymore, but use it on the tensor instead:
self.linear2.bias = nn.Parameter(init_grid(grid_size).view(-1).to(DEVICE))

The better way would be to remove all to() operations from the __init__ and call model.to() once it’s initialized.