Has anyone had any issue with mobileBERT? I use mobileBERT and DistillBERT on Hugging face with the same code for text classification. DistillBert quickly converges and gives me a good result! But mobileBERT is so slow and doesn’t converge at all. I couldn’t find any good tip in mobileBERT paper … Is there anything else which I miss?
@ptrblck : Do you have any suggestions for that? Thanks.