Hello, I have a question about methods like detach() (and tweaking requires_grad to False), and their relation to add_param_group. Does these 2 functions serve the exact opposite purpose, as one undo the other?
For example, if I build a net that may either freeze or unfreeze different layers for each backward pass during training, would it be suitable to use detach() (or tweak requires_grad) to omit them for frozen layers, and vice versa, using add_param_group to add them back into the optimizer’s param groups?
Thank you!