Solved by ptrblck in post #2 
              
              
                Yes, you can use a condition to add a new layer to a list first and then create the nn.Sequential model: 
layers = [nn.Conv2d(3, 5, kernel_size=3, stride=1,padding=1)]
cond = True
if cond:
    layers.append(nn.LeakyReLU(0.2,True))
model = nn.Sequential(*layers)