Hi all,
When I’m loading my pretrained model without freezing or removing any layers and using an lr of 0.001 I’m getting quite good results. Ex:
models.resnet18(pretrained=True)

def get_trainable(model_params):
    return (p for p in model_params if p.requires_grad)

  
optimizer = torch.optim.Adam(
    get_trainable(model.parameters()),
    lr=0.001,
)

However, when I do freeze layers for feature extraction my doesn’t learn an all. Ex:
def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False

n_classes = 5
set_parameter_requires_grad(model, True)
num_ftrs = model.classifier.in_features
model.classifier = nn.Linear(num_ftrs, n_classes)

There’s a third scenario I just ran into where I don’t freeze anything but specify the number of classes. In this case, my model overfits.
model.classifier = nn.Linear(num_ftrs, n_classes)

Any idea why this happens?