Modules are computation blocks with state, can we put it this way? But simple ops like add do not hold states. And what you explained makes sense because during training we need to save states for every iteration.
But Torchscript is often used for inference. I don’t see why we cannot script or trace a forward function with a submodule defined in it. From my understanding, scripting or tracing a Function(or a Module) does not save states like weight or bias right? It will just record a pure function path.