Fine-tuning approach for Bert and GPT2 classifiers
## ❓ Questions & Help

Hey folks, when we are fine-tuning BERT or GPT2 model for a classification task via classes like GPT2DoubleHeadsModel or BertForSequenceClassification, what is the recommended fine-tuning strategy? I assume all transformer layers of the base model are unfrozen for fine-tuning. Does this result in catastrophic forgetting in practice? Do people use gradual unfreezing (as in ULMFiT)?
AFAIK, BERT doesn't make use of gradual unfreezing. Instead, during fine-tuning all model parameters are trainable. It can result in catastrophic forgetting, if you train it for long enough/ large enough learning rate, which is why we usually fine tune for 1-2 epochs at a low learning rate.

When it comes to doing it yourself, you'll should be able to just tweak the number of epochs/train steps and then find which number gives you the best results. IMO anymore than a couple epochs will result in overfitting/forgetting.

Hope that helps.

https://arxiv.org/pdf/1905.05583.pdf<|||||>This issue can be closed.<|||||>Thanks @andrewpatterson2018.