I’m curious what happens in this scenario. If we set requires_grad = False to some parameters, but accidentally pass these to the optimizer, are they skipped over or still optimized? As in the following:
model_conv = torchvision.models.resnet18(pretrained=True)
for param in model_conv.parameters():
    param.requires_grad = False

# Parameters of newly constructed modules have requires_grad=True by default
num_ftrs = model_conv.fc.in_features
model_conv.fc = nn.Linear(num_ftrs, 2)

model_conv = model_conv.to(device)

criterion = nn.CrossEntropyLoss()

optimizer_conv = optim.SGD(model_conv.parameters(), lr=0.001, momentum=0.9)

Note that model_conv.parameters() should actually be model_conv.fc.parameters().
Are all of model_conv parameters optimized or only model_conv.fc?