I don’t think there is a simple answer to this question.



 bing:

I have seen multiple tutorial where few people adds single fc layers after removing the initial fc layer
Also, it seems like a rabbit hole where we can freeze-unfreeze the different number of sub-modules. It seems not very efficient to me.


I think that’s unfortunately still often the case and you will read in a lot of papers that the training hyperparameters were “tuned” or “heuristically determined”.
You could have a look at e.g. FastAI 4 which explains the best practices for some use cases.