Solved by ptrblck in post #2 
              
              
                The parameters might still get updated even with a zero gradient, if you are using an optimizer with momentum or other running estimates. 
Here is a small code example: 
model = nn.Linear(1, 1, bias=False)

optimizer = torch.optim.SGD(model.parameters(), lr=1., momentum=0.) # same results for w1 and…