prajjwal1:

Btw why are cyclical schedulers not used with transformers ?


They are not SOTA (or take a longer time to be) since 2 years ago. It’s just that fastai and transformers models are the only wide place that knows it  In the case of transformers, I know the GPT models just don’t train as well with another schedule. I don’t know if all the BERT models are just following that trend but I’d guess they also need that kind of schedule to train properly.