The difference between 2nd and 3rd is that for the 2nd scenario I’m using the set_parameter_requires_grad function.
Regarding optimization, I’m using this for both:
optimizer = torch.optim.Adam(model.parameters(),lr=0.001)