No experience with MobileBert here, but is there any reason why you’re posting this here, instead of https://discuss.huggingface.co 3 or Issues · huggingface/transformers · GitHub?
This is very HF-specific question and you will have a much better turnaround for getting answers by asking these to the right audiences 
When you make a post there, it’d help for you to provide some details for others to know what you’re doing specifically. i.e. how you train, which versions of main components you use, etc. Think what the other person needs to know in order to understand what you’re trying to do.