Yes, the first explanation of “stateful” modules makes sense.
I’m not sure how TorchScript is related to this. Note that you surely can re-initialize modules in the forward pass, if you explicitly don’t want to train these layers and want to create new random parameters.
A scripted model should respect this workflow (even if it’s wrong from the point of view of training the model).