How are you freezing the layers? Freezing is the only way in which you can exclude parameters during training. In your example I see that you have defined your optimizer as checking out all params. While freezing, this is the way to set up your optimizer:
 optim = torch.optim.SGD(filter(lambda p: p.requires_grad, net.parameters()), lr, momentum=momentum, weight_decay=decay, nesterov=True) 

The filter doesn’t offer so much of a change in a simple optimizer with a learning rate but since you are using momentum and weight decay, the params whose requires_grad is set to False will be updating on the basis of these.