Hm okay for retraining Im already doing this in the following way:
    if params is None:
        params = filter(lambda p: p.requires_grad, self.model.parameters())

    self.optimizer = torch.optim.Adam(
        params, lr=lr, eps=eps, weight_decay=self.weight_decay
    )

But then I still have the problem that the whole first Layer is unfreezed and not only the weights for the conditions…