I’m not deeply familiar with MobileBERT but by skimming through some blog posts it seems that the MobileBERT model needs a teacher model for the training? Could it be the case that your training script is indeed training two models (a larger teacher and the smaller MobileBERT model)?
I’m also sure, that @stas might be more familiar with these implementations and could help out.