Soumith,
If I do
optimizer = optim.Adam([{'params': model.modules[14].parameters(), 'lr': 0.1*opt.lr}, {'params': list(module.parameters()) for index, module in enumerate(model.modules) if index != 14}], lr=opt.lr, weight_decay=1e-5)
then it seems again that the convergence is very slow and I’m not passing all my params to the optimizer. Or did I put the list() in the wrong place?
So far I tried two ways:
optimizer = optim.Adam([{'params': model.modules[14].parameters(), 'lr': 0.1*opt.lr}, {'params': model.modules[0].parameters(), 'lr':opt.lr}, {'params': model.modules[1].parameters(), 'lr':opt.lr}, {'params': model.modules[2].parameters(), 'lr':opt.lr}, {'params': model.modules[3].parameters(), 'lr':opt.lr}, {'params': model.modules[4].parameters(), 'lr':opt.lr}, {'params': model.modules[5].parameters(), 'lr':opt.lr}, {'params': model.modules[6].parameters(), 'lr':opt.lr}, {'params': model.modules[7].parameters(), 'lr':opt.lr}, {'params': model.modules[8].parameters(), 'lr':opt.lr}, {'params': model.modules[9].parameters(), 'lr':opt.lr}, {'params': model.modules[10].parameters(), 'lr':opt.lr}, {'params': model.modules[11].parameters(), 'lr':opt.lr}, {'params': model.modules[12].parameters(), 'lr':opt.lr}, {'params': model.modules[13].parameters(), 'lr':opt.lr}], lr=opt.lr, eps=1e-8, weight_decay=1e-5)
and this works very well. But, obviouslty, it’s not satisfactory w.r.t. readability.
And I tried the second way:
optimizer = optim.Adam([{'params': model.modules[14].parameters(), 'lr': 0.1*opt.lr}, {'params': module.parameters() for index, module in enumerate(model.modules) if index != 14}], lr=opt.lr, eps=1e-8, weight_decay=1e-5)
This is much clearer and Pythonic, but it just doesn’t work.
UPD:
I changed to a more Pythonic way, and this time it works:
ml = list()
ml.append({'params': model.modules[14].parameters(), 'lr': 0.1*opt.lr})
for index, module in enumerate(model.modules):
    if (index != 14):
        ml.append({'params': module.parameters()})
optimizer = optim.Adam(ml, lr=opt.lr, eps=1e-8, weight_decay=1e-5)

Thanks Soumith and Andy!