Solved by ptrblck in post #4 
              
              
                They might yield the same result, but in specific edge cases you might get different and unexpected results: 

An optimizer, which doesn’t hold certain parameters, won’t update them. However other optimizers could still update these parameters, if they are passed.
If you are using weight decay, have…