I optimize some parts of a model, while other parts are set fixed with requires_grad_(False).
Doesn’t weight decay change the weights of the layers, whose requires_grad is set to False, even if inputs are forwarded through these layers?
What about the case of a layer, whose requires_grad is set to True, but inputs are not forwarded through it during this optimization step?
Thanks