I’m not sure, if weight decay on frozen parameters is beneficial or not and you would have to run some experiments on it.
Just for the sake of completeness, weight decay won’t be applied if the .grad attribute is None as seen here 4. If you set the requires_grad attribute to False from the beginning, these parameters won’t be updated, as their .grad will never be set to a valid value.
However, if you’ve applied some training to these parameters before setting requires_grad=False and only zero out their gradients, weight decay will manipulate these in the next optimizer steps.