Solved by ptrblck in post #2 
              
              
                You can set the .requires_grad attribute of the parameters of the first layers to False to freeze them. 

You don’t necessarily need to filter out frozen parameters, as they won’t be updated, but could do it via e.g torch.optim.SG(filter(lambda p: p.requires_grad, model.parameters()), lr=...).