Follow up question, do I have to make a new parameter list  to feed it into my optimizer every time I increase my network’s complexity? Does Pytorch keep a track of that?
Eg:
# Before growth

Encoder = [ conv2d, relu ]
Decoder = [ contranspose2d, sigmoid]
parameter = list(Encoder.parameters() ) + list(Decoder.parameters() )
optimizer = optim.Adam(parameter, lr)

# After growth

Encoder = [ conv2d, relu, conv2d, relu ]
Decoder = [ constranspose2d, relu, contranspose2d, sigmoid]

parameter = list(Encoder.parameters() ) + list(Decoder.parameters() ) ?

# Should I grab the new parameters again?