On a second thinking, in this case using weight decay + requires_grad = False and still having the parameters in the optimizer ( what they did in the fine tuning case ) will imply that the of the “frozen” part of the network are going to get reduced at every epoch even if they are not changed by backprop, would it be not good for the network ?