Hi Mrityunjay!



 mrityu:

we are only updating the weights of the ‘FC’ layer(or any other layer for that matter) and not the rest of the model.


Yes, you are correct.  When you call optimizer_ft.step(), the
optimizer will only update the weights that were specified in that
optimizer.
Even, if other weights – not in the optimizer – have had their
gradients computed, those other weights will not be updated.


But yes, I agree it will be computationally expensive.


Yes.  Just to confirm, computing gradients for other weights not
in the optimizer will add (unnecessary) computational cost.  You
will pay the cost – mostly in memory – of building the “computation
graph” for those weights during the forward pass, and pay the
cost – mostly computational – of computing their gradients during
the backward pass.  But then you throw those gradients away,
wasting that effort.  (But doing so won’t break anything, except
maybe cause you to run out of memory.)



 OrielBanne:

yes - freezing any part of the model requires freezing the grads. otherwise, every optimizer stepyou make will change all the weights


Hi Oriel!
No, if by “freezing the grads” you mean setting requires_grad = False
for them, this is not true.  Even if you compute the gradients for various
weights (for which requires_grad = True), they will not be updated if
they are not “part of” the optimizer on which you call optimizer.step().
For example, it is perfectly possible to use one optimizer, optA, to
optimize one part, “part A,” of a model and use another, optB to
optimize another part, “part B.”  You could compute the gradients
for all of your weights, both those in “part A” and “part B” of your
model, with a single backward pass.  But when you call optA.step()
you will only update the weights in “part A,” leaving “part B” unchanged,
and only update the weights in “part B” when you call optB.step().
Best.
K. Frank