Does zero_grad clear all state information in the optimizer or can the optimizer maintain some information from the previous minibatch?
For example 1: if I change batch size as a function of epoch do I need to recreate the optimizer to zero out its internal state?
For example 2: if I have a common model (cnn layers), and 2 different heads (FC layers) with a small amount of data for head 1 and head 2, and a lot of data for head 1. I can train head 1, then train head 2.  Can I ping pong between training head 1 and head 2 or with the optimizer get all confused because of an internal state or discontinuous loss from the previous minibatch?
Note: the above examples are only to understand the optimizer and if it has any state information.