Solved by tom in post #2 
              
              
                As you note, most of the time it should make no difference. 

You might have more than one model in the same optimizer, GAN’s typically use two separate optimizers, but you might have a feature extractor and two heads that are separate and you train them together. Here the optimizer contains more th…