Solved by ptrblck in post #4 
              
              
                Rewriting the model definition would of course work. 
However, using getattr and setattr might be the hacky but faster way: 
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 3, 3, 1, 1)
        self.bn1 = nn.BatchNorm2d(3)
  …