Looks like you’re not passing the optimizer the rest of the model’s parameters, only the parameters of model.module[14]. Instead try:
optimizer = optim.Adam([{'params': model.modules[14].parameters(), 'lr': 0.1*opt.lr}, {'params': [module.parameters() for index, module in enumerate(model.modules) if index != 14]}], lr=opt.lr, weight_decay=1e-5)