Freezing part of the network might help in some situations (like computer vision) but it doesn’t really help for Transformer models, usually it gives worse results.

Hi @sgugger, I’ve heard this mentioned a few times on here. Just wondering if this is just “common knowledge” of the community (perhaps from the experience of training many models), or it’s from a source that investigates this question. I remember trying to find a definitive answer to this in the literature a while ago and not being able to find much, so any pointers would be appreciated!