@crcrpar
I encounter another issue in relation with the previous.
Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)
works fine.
However, the nets coming from pretrained_model turns out to be requires_grad = True  in the new_net although I already set those nets to be  requires_grad = True explicitly. It is because copying pretrained_net state_dict to a new_net state_dict with others remaining same in new_net state_dict looks not preserving grad status and initializes the nest to default, True. What is the right way to set the  only nets coming from the pretrained_model to be requires_grad = False after copying their state_dict to a new_net state_dict?