I did not know that thank you for the tip.
Edited: For add_param_group I assume you need to loop through the entire network and grab all newly added weights and then feed it into add_param_group is this thinking correct? The documentation says param_group is a dict so I should feed the state_dict() correct?