Solved by ptrblck in post #2 
              
              
                Zeroing out the gradients should work for optimizers without running estimates. 
If you are using an optimizer with internal states (e.g. Adam), this approach will still work, if you zero out the gradients from the beginning: 
conv = nn.Conv2d(1, 1 ,3)
weight_reference = conv.weight.clone()
optimize…