Solved by ptrblck in post #4 
              
              
                momentum would be such an internal state, which uses the previous gradients to update the parameters even with a zero gradients. 
Here is a small example showing this behavior: 
# Setup
model = nn.Conv2d(1, 10, 3, 1, 1)
weight_reference = model.weight.clone()
optimizer = torch.optim.SGD(model.parame…