Gradual unfreezing and discriminative fine-tuning for BERT
Three of the tips for fine-tuning proposed in ULMFIT are slanted triangular learning rates, gradual unfreezing, and discriminative fine-tuning.

I understand that BERT's default learning rate scheduler does something similar to STLR, but I was wondering if gradual unfreezing and discriminative fine-tuning are considered in BERT's fine-tuning implementations. Has anyone had experience implementing these two features in BERT fine-tuning? I'd like to hear your thoughts on it. Thanks!
I've tried a bit to play with these training schemes on a deep transformer for our [tutorial on Transfer Learning in Natural Language Processing](https://naacl2019.org/program/tutorials/#t4-transfer-learning-in-natural-language-processing) held at NAACL last week but I couldn't get gradual unfreezing and discriminative fine-tuning to out-perform a standard fine-tuning procedure (multi-tasking did help, however).

You can have a look at the results by reading the "Hands-on" parts of the tutorial here: https://tinyurl.com/NAACLTransfer.

You can give it a try your-self with the associated Colab notebook which is here: https://tinyurl.com/NAACLTransferColab (and a full stand-alone codebase is here: https://tinyurl.com/NAACLTransferCode).

It's possible that I just didn't spend enough time scanning the hyper-parameters or maybe these two training variants are better suited to LSTM than Transformers.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> I've tried a bit to play with these training schemes on a deep transformer for our [tutorial on Transfer Learning in Natural Language Processing](https://naacl2019.org/program/tutorials/#t4-transfer-learning-in-natural-language-processing) held at NAACL last week but I couldn't get gradual unfreezing and discriminative fine-tuning to out-perform a standard fine-tuning procedure (multi-tasking did help, however).
> 
> You can have a look at the results by reading the "Hands-on" parts of the tutorial here: https://tinyurl.com/NAACLTransfer.
> 
> You can give it a try your-self with the associated Colab notebook which is here: https://tinyurl.com/NAACLTransferColab (and a full stand-alone codebase is here: https://tinyurl.com/NAACLTransferCode).
> 
> It's possible that I just didn't spend enough time scanning the hyper-parameters or maybe these two training variants are better suited to LSTM than Transformers.

I find the standard fine-tuning procedure having a unstable issue, that different shuffle order affects a lot. I wander if unfreezing help the bert finetune get a relative stable result?
This is also mentioned in bert paper, they just use different random seeds to get a best result.