I ran into a problem that I think is related.
I have a CNN which I created using nn.ModuleList:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.modules = list()
        self.modules.append(nn.Conv2d(1, 32, (3, 3), (1, 1), (1, 1)))
        self.modules.append(nn.PReLU(32))
        # etc.: keep adding some modules, the last one being No.14
        self.model = nn.ModuleList(self.modules)

Then, in my main file, when I create the optimizer, I write the following:
model = Net()
model = model.cuda()
optimizer = optim.Adam([{'params': model.modules[14].parameters(), 'lr': 0.1*opt.lr}], lr=opt.lr, weight_decay=1e-5)

So by this line I thought that I would have the learning rate equal to opt.lr 6 for all the modules, EXCEPT the module #14, for which I want a 10 times lower learning rate. However, I get totally weird convergence. To check if that has something to do with the initialization, I removed the “0.1” in the optimizer initialization, making it basically having the same “lr” as every other module in my Net. BUT! The same weird convergence holds, much slower vs. if I just do a regular optimizer initialization (optimizer = optim.Adam(model.parameters(), lr=opt.lr 4, weight_decay=1e-5)) What might be the issue? Thanks!