@justusschock Thanks for the quick reply!
I see, so the underlying mechanics are different. So what if I do the following:
I freeze and unfreeze layers by using the with requires_grad depending on the particular graph topology before each forward-backward pass during the training loop. But when I add new layers back into the graph, I would need to add their parameters back into the optimizer with add_param_group.
Would that work? Also, do I need to take out parameter group from the optimizer when I freeze layers at some particular instance in-between the training loop too? (Opposite of add_param_group) Or is that already tracked and taken care of when I freeze it with requires_grad?