Solved by driazati in post #2 
              
              
                You should move the initialization of submodules out into __init__ rather than in your model’s forward. This will also likely improve performance since costly parameter allocations only need to happen once instead of on each run of your model’s forward. 
class M(nn.Module):
    def __init__(self, nu…