Yes, I was passing net.parameters() to the optimizer, not net. That was a typo when I was writing the post here in the forum.
Anyway, I was not aware of the existence of nn.ModuleList and it actually solves my problem! Great, thank you!