I am actually freezing them from the beginning and I do use weight decay.
I believe I am already passing only the parameters that require grads to the optimizer. See below:
self.optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, self.model.parameters()), lr=self.learning_rate, weight_decay=self.penalty)