albanD:

They are supposed to be learnable components only


It would work just like Parameter has a requires_grad True but still provides an option to change it. This way same network class(architecture) can be reused as required in the above scenario.
Or is there any other way to create non learnable modules without having to rewrite their forward passes.