In addition, how to set learning rates for different layers?
I think use
for param_group in optimizer.state_dict()['param_groups']: param_group['lr'] = lr
can only set the learning rate for hole model.