Up to this moment I haven’t code it up as I wanted to see first which is the best way to do it. I was thinking in using a similar approach as the one you propose which is create a parameter group per new parameter and then loop over them if I want to change the learning rates.
I think is the cleanest and bug safe. I guess the only problem is that we end up having a bigger loop over self.params_groups .
From my doc check there is no cleaner way to handle what I propose in this post so yes I think the safest is to have several groups in the parameter. Will come back If I see a very drop in performance or any strange behavior.