Solved by SimonW in post #5 
              
              
                When you add new parameter: 

assign the parameter to attribute of the module
add it to the optimizer via optim.add_param_group({"params": my_new_param})