Solved by ptrblck in post #2 
              
              
                You could filter out the batch norm layers and set the requires_grad attribute of all other parameters to False: 
model = models.resnet18()

def freeze_all_but_bn(m):
    if not isinstance(m, torch.nn.modules.batchnorm._BatchNorm):
        if hasattr(m, 'weight') and m.weight is not None:
          …