I have exactly the same issue. Too slow. And in my case, it doesn’t converge at all. I increased the learning rate as was advised in the paper, but it didn’t help. Any idea?
My code is working perfectly with normal bert and distill_bert! But very bad performance with mobilebert.